Week 2
This week I work on the conversion from .ipynb notebook to a json file with python ast module
I started with an example benchmark that was created manually, and try to achieve the same output with python code.
This is my approach:
1.First, create a json object for each line of code
{ “operation_type”: “”, “new_df”: null, “prev_df”: null, “code”: “df.head(3)”, “required_variables”: {}, “imports”: {}, “snippet_cell_number”: 11 }
- Start from import statements, check which module is imported and what its “nickname” is
- look for when assign happens, any method call, etc
- check plotting, which would be a read operation
- output all the lines that do modify the data frame (read, read/write)
- view the output of step 5 as input and execute line by line, record the dfs
- update the df name after each (read, read/write)
- variables?
During this process, some questions emerge: In step 3, I am now hardcoding all the possible methods:
if node.value.func.attr in [‘to_datetime’, ‘apply’, ‘drop’, ‘merge’, ‘update’]:
and how to handle edge case like “df[‘zip’].value_counts().head(5)\n”
In step 4, I am looking for countplot in sns, which is hard coding, and if they use other plotting method like matplotlib, the code would not work
In step 8, how to identify parameters and record them
And also: Why do we want read operation? Scope of data cleaning? Dropna, groupby, pivot, merge…
This week in the weekly study group I led discussion on a paper: Solo: Data Discovery Using Natural Language Questions Via A Self-Supervised Approach
Last updated on June 20th
