Distributed Web Scraper

The work

Overview

Scraping at any real scale runs into the same three walls: throughput, resilience, and politeness. This project tackles all three with a cleanly separated architecture. A stateless Flask/Gunicorn web tier accepts crawl jobs and a Redis/RQ worker tier executes them, with every piece of shared state — the frontier, dedup set, and results — living in PostgreSQL so either tier can scale horizontally without coordination. The fetch layer is built to survive the open web: exponential-backoff retries, rotating user-agents, and a headless-Selenium fallback for JavaScript-heavy pages, all fronting a breadth-first crawler that deduplicates URLs and respects robots.txt. The whole thing runs on Kubernetes, where a CPU-based HorizontalPodAutoscaler grows the worker tier on demand.

Architecture

How it's built

Fig. 004.1 · A stateless web tier enqueues jobs to Redis/RQ workers; all state lives in shared PostgreSQL, and a CPU-based HPA scales the worker tier on Kubernetes.

Detail

Key features

• Stateless Flask/Gunicorn web tier and a separate Redis/RQ worker tier, with all shared state (frontier, dedup set, results) in PostgreSQL — so each tier scales horizontally and independently.
• Resilient fetch layer: exponential-backoff retries, rotating user-agents, and a headless-Selenium fallback for JavaScript-rendered pages that a plain HTTP fetch can't read.
• Breadth-first crawler with URL deduplication and robots.txt compliance — polite and predictable by default.
• Deployed to Kubernetes via Deployments, an Ingress, a Postgres StatefulSet, and a CPU-based HorizontalPodAutoscaler that adds workers under load.

Imagery

In context

Results

Impact

• Horizontally scalable by design — web and worker tiers scale independently behind a CPU-based autoscaler
• Resilient against flaky targets: retries, user-agent rotation, and a Selenium fallback keep the crawl alive through failures
• Polite by construction — URL deduplication and robots.txt compliance are built into the crawler, not bolted on
• Stateless tiers over shared PostgreSQL make the system restart-safe and trivially replicable

Open source

How to contribute

A systems playground built to scale. The cleanest ways to extend it:

• Add pluggable site parsers behind a common extraction interface
• Swap Redis/RQ for Celery or Arq to compare worker ergonomics
• Add a shared, Redis-backed distributed rate-limiter across workers
• Ship a Helm chart so the Kubernetes deploy is a single command