Week 3
This week my work shifted from conversion to crawling, my task is to crawl 100 notebooks and datasets and inspect them. We want to find a way to identify some datasets, then download notebooks to process
I finished some remaining work for week two, I developed a way to capture the paremeter in the function calls, so that we could have a reference to them in JSON format benchmark Kaggle API was recommended to me as an effective tool to crawl data from kaggle, professor Fariha commented that notebooks with more upvotes may have better quality.
After crawling several notebooks, I find out that many of them are used for machine learning purpose, they are not really helpful for our benchmark.
I decide to start with the notebooks that have “pd.readcsv()” syntax, and exclude the files that is reading multiple csvs
This week I also did a presentation on Splitting Tuples of Mismatched Entities https://dl.acm.org/doi/10.1145/3626763
Last update on June 20th
