Scraping 107M rows of data to build this

The author used AI agents to scrape and process 107 million rows of UK council spending data, creating a web application to visualize these expenditures. The process involved data collection, cleaning, prototyping different visualizations, and optimizing for large datasets.

• AI agents were employed to scrape 107 million rows of UK council spending data, focusing on payments over £500.
• The project involved cataloging official data sources, downloading data from multiple councils, and consolidating it into a single format.
• Various prototypes were built to visualize the data, including maps, ledgers, and comparison cards, with a focus on an ‘Apple Maps’ style interface.
• Technical challenges included handling large datasets, leading to the adoption of Parquet and DuckDB for efficient data management.
• The final stages involved data cleaning, vendor classification, duplicate merging, and refining the user experience and data validity.
• The resulting web application aims to make public council spending data accessible and understandable to the public.
  https://bender.layer3.press/articles/eccf713d-d351-45b7-bc20-7d6f0cb335ac