Week 7
Debug the crawl to see reasons of rate limit
Add dataframe name and dataset name to the summary excel sheet
Implement retry mechanism: sleep for a while and restart, prevent rate limit, and handling connection error
Cluster the apply operations:
Do the preprocess on excel sheet Implent extract-var to get the variable name inside apply/mapping operations
We want to make sense of what the variable represents apply operation example Some appraches are knowing how many if and return statements it have, lambda operations, map with dictionaries
Explore alternative approach
Another way is to start with the dataset, find the dataset with low usability score, then find and download its notebooks
Written on July 1, 2024
