Week 10
This is the final week of my summer program.
Summer 2024 DREU Project
This is the final week of my summer program.
These was issue last week that there are duplicate operations from different notebooks, which might happen if there are several notebooks in the same subfolder. To resolve this issue, I cleared the new notebook directory before download notebook
Resolve the issue about rate limit: Implement API test to check the capability of the kaggle. When I there are 105 pages, which displays 20 notebooks each page, and we are filtering the notebooks with min_upvotes of 5, that explains why api stopped Encoding vs bracketing
Next step is to analyze code and get an idea of common data wrangling operations, the target is 10k notebooks
1, Update the format of code for other team members to use it. Organize the directory so that it has a clear hirearchy, the user can navigate to the src folder and run the program like a python module, with no hardcoded parameters. Move all the program parameters in cocnfig.py
1, Managed to download the datasets associated with the notebooks, I extracted the source data info from notebook metadatas
This week my work shifted from conversion to crawling, my task is to crawl 100 notebooks and datasets and inspect them. We want to find a way to identify some datasets, then download notebooks to process
This week I work on the conversion from .ipynb notebook to a json file with python ast module
I started my research at May 20th and this is a 10-week research program