Week 7

Debug the crawl to see reasons of rate limit

Add dataframe name and dataset name to the summary excel sheet

Implement retry mechanism: sleep for a while and restart, prevent rate limit, and handling connection error

Cluster the apply operations:

Do the preprocess on excel sheet Implent extract-var to get the variable name inside apply/mapping operations

We want to make sense of what the variable represents apply operation example Some appraches are knowing how many if and return statements it have, lambda operations, map with dictionaries

Explore alternative approach

Another way is to start with the dataset, find the dataset with low usability score, then find and download its notebooks

Written on July 1, 2024