What it was

iScraper, built under ProAPIs, Inc., provided high-throughput, globally distributed data APIs for harvesting and structuring public web data at scale. Customers used it to power AI training datasets, market analytics, and recruitment intelligence.

The engineering

Running a data API at that scale meant solving hard infrastructure problems: globally distributed scraping fleets, resilient retry and queueing systems, real-time parsing pipelines, and an API layer that stayed fast under heavy concurrent load. Much of what I know about designing resilient data pipelines — retry budgets, dead-letter queues, per-source observability — was learned building and operating iScraper in production.

Where it stands

The product has been discontinued. The lessons from building it now inform everything I ship, and I write about many of them on the blog.