<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://stawberryslayer.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://stawberryslayer.github.io/" rel="alternate" type="text/html" /><updated>2024-09-23T04:52:33+00:00</updated><id>https://stawberryslayer.github.io/feed.xml</id><title type="html">Yuqing Wang</title><subtitle>Summer 2024 DREU Project</subtitle><entry><title type="html">Week 10</title><link href="https://stawberryslayer.github.io/week10/" rel="alternate" type="text/html" title="Week 10" /><published>2024-07-22T00:00:00+00:00</published><updated>2024-07-22T00:00:00+00:00</updated><id>https://stawberryslayer.github.io/week10</id><content type="html" xml:base="https://stawberryslayer.github.io/week10/"><![CDATA[<p>This is the final week of my summer program.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[This is the final week of my summer program.]]></summary></entry><entry><title type="html">Week 9</title><link href="https://stawberryslayer.github.io/week9/" rel="alternate" type="text/html" title="Week 9" /><published>2024-07-15T00:00:00+00:00</published><updated>2024-07-15T00:00:00+00:00</updated><id>https://stawberryslayer.github.io/week9</id><content type="html" xml:base="https://stawberryslayer.github.io/week9/"><![CDATA[<p>These was issue last week that there are duplicate operations from different notebooks, which might happen if there are several notebooks in the same subfolder. To resolve this issue, I cleared the new notebook directory before download notebook</p>

<p>Resolve issue and add ther parameters in the dataset</p>]]></content><author><name></name></author><summary type="html"><![CDATA[These was issue last week that there are duplicate operations from different notebooks, which might happen if there are several notebooks in the same subfolder. To resolve this issue, I cleared the new notebook directory before download notebook]]></summary></entry><entry><title type="html">Week 8</title><link href="https://stawberryslayer.github.io/week8/" rel="alternate" type="text/html" title="Week 8" /><published>2024-07-08T00:00:00+00:00</published><updated>2024-07-08T00:00:00+00:00</updated><id>https://stawberryslayer.github.io/week8</id><content type="html" xml:base="https://stawberryslayer.github.io/week8/"><![CDATA[<p>Resolve the issue about rate limit: Implement API test to check the capability of the kaggle. When I there are 105 pages, which displays 20 notebooks each page, and we are filtering the notebooks with min_upvotes of 5, that explains why api stopped 
Encoding vs bracketing</p>

<p>manually restart after crawling 10 pages of datasets, estimat to get 1000 notebooks each time
Checked other benchmark Dataset:</p>

<p>Add apply operations</p>

<p>The notebook param</p>]]></content><author><name></name></author><summary type="html"><![CDATA[Resolve the issue about rate limit: Implement API test to check the capability of the kaggle. When I there are 105 pages, which displays 20 notebooks each page, and we are filtering the notebooks with min_upvotes of 5, that explains why api stopped Encoding vs bracketing]]></summary></entry><entry><title type="html">Week 7</title><link href="https://stawberryslayer.github.io/week7/" rel="alternate" type="text/html" title="Week 7" /><published>2024-07-01T00:00:00+00:00</published><updated>2024-07-01T00:00:00+00:00</updated><id>https://stawberryslayer.github.io/week7</id><content type="html" xml:base="https://stawberryslayer.github.io/week7/"><![CDATA[<h3 id="debug-the-crawl-to-see-reasons-of-rate-limit">Debug the crawl to see reasons of rate limit</h3>

<p>Add dataframe name and dataset name to the summary excel sheet</p>

<p>Implement retry mechanism: sleep for a while and restart, prevent rate limit, and handling connection error</p>

<h3 id="cluster-the-apply-operations">Cluster the apply operations:</h3>

<p>Do the preprocess on excel sheet
Implent extract-var to get the variable name inside apply/mapping operations</p>

<p>We want to make sense of what the variable represents
<a href="../images/apply-operation-example.png">apply operation example</a>
Some appraches are knowing how many if and return statements it have, lambda operations, map with dictionaries</p>

<h3 id="explore-alternative-approach">Explore alternative approach</h3>
<p>Another way is to start with the dataset, find the dataset with low usability score, then find and download its notebooks</p>]]></content><author><name></name></author><summary type="html"><![CDATA[Debug the crawl to see reasons of rate limit]]></summary></entry><entry><title type="html">Week 6</title><link href="https://stawberryslayer.github.io/week6/" rel="alternate" type="text/html" title="Week 6" /><published>2024-06-24T00:00:00+00:00</published><updated>2024-06-24T00:00:00+00:00</updated><id>https://stawberryslayer.github.io/week6</id><content type="html" xml:base="https://stawberryslayer.github.io/week6/"><![CDATA[<p>Next step is to analyze code and get an idea of common data wrangling operations, the target is 10k notebooks</p>

<p>Implement page tracker to keep track of the current page in Kaggle, ensure that no pages are revisited when the program rerun.
Create an output summary in excel spreadsheet, which is a summary of all the operation in the crawled notebooks
We encouter an issue where the notebook stopped after getting about 600 notebooks</p>

<p>Last update on July 16th</p>]]></content><author><name></name></author><summary type="html"><![CDATA[Next step is to analyze code and get an idea of common data wrangling operations, the target is 10k notebooks]]></summary></entry><entry><title type="html">Week 5</title><link href="https://stawberryslayer.github.io/week5/" rel="alternate" type="text/html" title="Week 5" /><published>2024-06-17T00:00:00+00:00</published><updated>2024-06-17T00:00:00+00:00</updated><id>https://stawberryslayer.github.io/week5</id><content type="html" xml:base="https://stawberryslayer.github.io/week5/"><![CDATA[<p>1, Update the format of code for other team members to use it.
Organize the directory so that it has a clear hirearchy, the user can navigate to the src folder and run the program like a python module, with no hardcoded parameters.
Move all the program parameters in cocnfig.py</p>

<p>2, Keep track of current notebook pages, don’t crawl repeat notebooks</p>

<p>3, We decide not to process files with several notebooks because these notebooks in kaggle often do machine learning tasks with train and test csvs, and they have different type of data wranging operations as our project.</p>

<p>Last update on July 16th</p>]]></content><author><name></name></author><summary type="html"><![CDATA[1, Update the format of code for other team members to use it. Organize the directory so that it has a clear hirearchy, the user can navigate to the src folder and run the program like a python module, with no hardcoded parameters. Move all the program parameters in cocnfig.py]]></summary></entry><entry><title type="html">Week 4</title><link href="https://stawberryslayer.github.io/week4/" rel="alternate" type="text/html" title="Week 4" /><published>2024-06-10T00:00:00+00:00</published><updated>2024-06-10T00:00:00+00:00</updated><id>https://stawberryslayer.github.io/week4</id><content type="html" xml:base="https://stawberryslayer.github.io/week4/"><![CDATA[<p>1, Managed to download the datasets associated with the notebooks, I extracted the source data info from notebook metadatas</p>

<p>2, Resolved issue about kaggle API premission, shifted to openddatasets API, but eventually go back to kaggle API</p>

<p>3, An issue sets in:</p>

<p><img src="../images/ForLoopExample.png" alt="screenshot" /></p>

<p>The “:” after the for loop is causing trouble:</p>

<p>Final solution: If a line is not recoginzed, then collect this line and output it</p>

<p>4, Implement quality control for the notebooks
The code would need parameters: Minimum valid operations, upvotes, number of dataset.</p>

<p>Minimum valid operations: parse the notebook to json and see how many wrangling, we only want the note book with certain minimum number of operations, they are viewed as good quality.</p>

<p>Number of dataset: we want to ensure that the notebooks are from at least n datasets</p>

<p>upvotes: notebooks with higher upvotes have better quality</p>

<p>Last update on June 27th</p>]]></content><author><name></name></author><summary type="html"><![CDATA[1, Managed to download the datasets associated with the notebooks, I extracted the source data info from notebook metadatas]]></summary></entry><entry><title type="html">Week 3</title><link href="https://stawberryslayer.github.io/week3/" rel="alternate" type="text/html" title="Week 3" /><published>2024-06-03T00:00:00+00:00</published><updated>2024-06-03T00:00:00+00:00</updated><id>https://stawberryslayer.github.io/week3</id><content type="html" xml:base="https://stawberryslayer.github.io/week3/"><![CDATA[<p>This week my work shifted from conversion to crawling, my task is to crawl 100 notebooks and datasets and inspect them.
We want to find a way to identify some datasets, then download notebooks to process</p>

<p>I finished some remaining work for week two, I developed a way to capture the paremeter in the function calls, so that we could have a reference to them in JSON format benchmark
Kaggle API was recommended to me as an effective tool to crawl data from kaggle, professor Fariha commented that notebooks with more upvotes may have better quality.</p>

<p>After crawling several notebooks, I find out that many of them are used for machine learning purpose, they are not really helpful for our benchmark.</p>

<p>I decide to start with the notebooks that have “pd.readcsv()” syntax, and exclude the files that is reading multiple csvs</p>

<p>This week I also did a presentation on Splitting Tuples of Mismatched Entities  https://dl.acm.org/doi/10.1145/3626763</p>

<p>Last update on June 20th</p>]]></content><author><name></name></author><summary type="html"><![CDATA[This week my work shifted from conversion to crawling, my task is to crawl 100 notebooks and datasets and inspect them. We want to find a way to identify some datasets, then download notebooks to process]]></summary></entry><entry><title type="html">Week 2</title><link href="https://stawberryslayer.github.io/week2/" rel="alternate" type="text/html" title="Week 2" /><published>2024-05-27T00:00:00+00:00</published><updated>2024-05-27T00:00:00+00:00</updated><id>https://stawberryslayer.github.io/week2</id><content type="html" xml:base="https://stawberryslayer.github.io/week2/"><![CDATA[<p>This week I work on the conversion from .ipynb notebook to a json file with python ast module</p>

<p>I started with an example benchmark that was created manually, and try to achieve the same output with python code.</p>

<p>This is my approach:</p>

<p>1.First, create a json object for each line of code</p>

<p>{
           “operation_type”: “”,
           “new_df”: null,
           “prev_df”: null,
           “code”: “df.head(3)”,
           “required_variables”: {},
           “imports”: {},
           “snippet_cell_number”: 11
       }</p>

<ol>
  <li>Start from import statements, check which module is imported and what its “nickname” is</li>
  <li>look for when assign happens, any method call, etc</li>
  <li>check plotting, which would be a read operation</li>
  <li>output all the lines that do modify the data frame (read, read/write)</li>
  <li>view the output of step 5 as input and execute line by line, record the dfs</li>
  <li>update the df name after each (read, read/write)</li>
  <li>variables?</li>
</ol>

<p>During this process, some questions emerge:
In step 3, I am now hardcoding all the possible methods:</p>

<p>if node.value.func.attr in [‘to_datetime’, ‘apply’, ‘drop’, ‘merge’, ‘update’]:</p>

<p>and how to handle edge case like “df[‘zip’].value_counts().head(5)\n”</p>

<p>In step 4, 
I am looking for countplot in sns, which is hard coding, and if they use other plotting method like matplotlib, the code would not work</p>

<p>In step 8, how to identify parameters and record them</p>

<p>And also:
Why do we want read operation?
Scope of data cleaning? Dropna, groupby, pivot, merge…</p>

<p>This week in the weekly study group I led discussion on a paper: Solo: Data Discovery Using Natural Language Questions Via A Self-Supervised Approach</p>

<p>Last updated on June 20th</p>]]></content><author><name></name></author><summary type="html"><![CDATA[This week I work on the conversion from .ipynb notebook to a json file with python ast module]]></summary></entry><entry><title type="html">Week 1</title><link href="https://stawberryslayer.github.io/week1/" rel="alternate" type="text/html" title="Week 1" /><published>2024-05-20T00:00:00+00:00</published><updated>2024-05-20T00:00:00+00:00</updated><id>https://stawberryslayer.github.io/week1</id><content type="html" xml:base="https://stawberryslayer.github.io/week1/"><![CDATA[<p>I started my research at May 20th and this is a 10-week research program</p>

<p>I read five papers about data cleaning/wrangling syetem</p>

<ol>
  <li>C Yan and Y He. 2020. Auto-Suggest: Learning-to-Recommend Data Preparation Steps Using Data Science Notebooks. In SIGMOD. 1539–1554. https://github.com/congy/AutoSuggest</li>
  <li>R Bavishi, C Lemieux, R Fox, K Sen, and I Stoica. 2019. AutoPandas: Neural-Backed Generators for Program Synthesis. Proc. ACM Program. Lang. OOPSLA, Article 168 (oct 2019), 27 pages.</li>
  <li>P J Guo, S Kandel, J Hellerstein, and J Heer. 2011. Proactive Wrangling: MixedInitiative End-User Programming of Data Transformation Scripts. In UIST. 65–74.</li>
  <li>S Kandel, A Paepcke, J Hellerstein, and J Heer. 2011. Wrangler: Interactive Visual Specification of Data Transformation Scripts. In CHI. 3363–3372.</li>
  <li>M Raza and S Gulwani. 2017. Automated data extraction using predictive program synthesis. In AAAI, Vol. 31.</li>
</ol>

<p>They provide rich insight to me about how algorithms plays an cricial role in helping users process data and how researchers develop such algorithms
I learnt from the first paper the graph-like representation and min-cut clustering for choosing the next recommendation.
Some things to consider in this project that some system would have both input and output of notebook as the input for the system, but in this project we only have the input data.</p>

<p>Semantic Understanding should be helpful for recommendation
Benchmark dataset: I am planning to lookup github on my own, I will also include some edge cases</p>

<p>Last update on Jun 20th</p>]]></content><author><name></name></author><summary type="html"><![CDATA[I started my research at May 20th and this is a 10-week research program]]></summary></entry></feed>