Week 9

These was issue last week that there are duplicate operations from different notebooks, which might happen if there are several notebooks in the same subfolder. To resolve this issue, I cleared the new notebook directory before download notebook

Read More

Week 8

Resolve the issue about rate limit: Implement API test to check the capability of the kaggle. When I there are 105 pages, which displays 20 notebooks each page, and we are filtering the notebooks with min_upvotes of 5, that explains why api stopped Encoding vs bracketing

Read More

Week 6

Next step is to analyze code and get an idea of common data wrangling operations, the target is 10k notebooks

Read More

Week 5

1, Update the format of code for other team members to use it. Organize the directory so that it has a clear hirearchy, the user can navigate to the src folder and run the program like a python module, with no hardcoded parameters. Move all the program parameters in cocnfig.py

Read More

Week 4

1, Managed to download the datasets associated with the notebooks, I extracted the source data info from notebook metadatas

Read More

Week 3

This week my work shifted from conversion to crawling, my task is to crawl 100 notebooks and datasets and inspect them. We want to find a way to identify some datasets, then download notebooks to process

Read More

Week 2

This week I work on the conversion from .ipynb notebook to a json file with python ast module

Read More

Week 1

I started my research at May 20th and this is a 10-week research program

Read More