Queues and workers: when one script stops being enough
The point where a single script breaks down
Most scrapers start the same way: a script that loops over a list of URLs, fetches each one, parses it, writes the result somewhere, and moves to the next. That works fine for a few hundred pages. It falls apart somewhere between a few thousand and a few hundred thousand, and the failure mode is always the same. One request hangs on a slow response and the whole loop stalls behind it. One proxy goes bad and every request after it inherits the same connection until you notice. The script dies at row 40,000 of 100,000 because of an unhandled exception, and now you’re guessing which rows already completed.
None of these are proxy problems or target-site problems. They’re architecture problems. A single-threaded loop has no concept of “this one task failed, retry it later while the others keep moving.” It has no memory of what’s done if the process dies. It treats every request as sequential and blocking, which means your total runtime is the sum of every request’s latency, including the slow ones and the ones that time out.
The fix isn’t a smarter script. It’s a different shape: a queue that holds the work, and a pool of workers that pull from it independently.
What a job queue actually does
A scraping job queue is a list of tasks, each one usually just a URL plus whatever metadata the task needs, sitting in something that can be read and written safely by multiple processes at once. That can be as simple as a Redis list, a Postgres table with a status column, or a proper message broker like RabbitMQ or SQS. The mechanism doesn’t matter much at small scale. What matters is the separation it creates between “what needs to be done” and “who is doing it right now.”
Workers are separate processes (or threads, or async tasks, depending on your stack) that each do the same loop: pull a task off the queue, run it, report the result, ask for the next one. Because they’re independent, one worker hanging on a slow response doesn’t block the other nine. Because the task list lives outside any single worker’s memory, killing a worker mid-run doesn’t lose the queue state, only whatever that one worker was holding at the moment.
This is the core value: failure becomes local instead of global. A bad proxy, a timeout, a 500 response, a parsing exception, all of these now affect one task instead of stopping the whole job.
Retries stop being an afterthought
In a single script, retry logic tends to get bolted on as a try/except around the request call, maybe with a sleep and a counter. It works until the exception happens somewhere the try/except doesn’t cover, or the retry sleep blocks the next 500 URLs behind it.
A queue-based system makes retries a property of the task itself, not the code path. A task that fails goes back onto the queue, or into a delayed queue that becomes visible again after a backoff period, with its attempt count incremented. A worker that picks it up later doesn’t need to know why it failed before, it just tries again. If a task fails past a max attempt count, it moves to a dead-letter queue where you can inspect it separately instead of it silently vanishing or crashing the run.
This also changes how you think about failure types. A connection timeout probably deserves a short backoff and a retry. A 403 from a site that’s actively pattern-matching bot traffic deserves a longer backoff, a different proxy on the next attempt, and honestly a second look at whether your request pattern is triggering it in the first place, not just brute-force retrying into the same block.
Rate limits become a queue property, not a script property
When you’re respecting a site’s published rate limits or just trying to keep your request pattern from looking like a flood, that limit belongs to the target, not to any one worker. If you have ten workers each independently sleeping 2 seconds between requests, you don’t have a 2-second gap between requests to that site, you have up to ten requests landing in the same 2-second window.
Rate limiting has to sit between the queue and the workers, not inside each worker. A token bucket or a shared rate limiter that all workers check against before pulling a task keeps the aggregate request rate under control regardless of how many workers you’re running. Some queue systems (Celery with rate_limit, Redis-based limiters) support this natively. If yours doesn’t, it’s worth building before you scale worker count, because worker count multiplying your effective rate is exactly the thing that turns a polite scraper into one that gets flagged.
Where proxies fit into this shape
Proxies solve a different problem than queues do, but they interact directly. If every worker uses the same proxy, you’ve just centralized your outbound traffic through one IP that now sees the combined request rate of every worker, which is a fast way to get that IP rate-limited or blocked regardless of how well-behaved any individual worker is.
The common pattern is to assign proxies per-worker or per-task, so the request pattern a target site sees is spread across a rotating pool rather than concentrated. Datacenter proxies are cheap and fast, which suits high-volume workers hitting sites with light protection. Residential and mobile proxies route through real ISP or carrier IPs and are better suited to targets that actively fingerprint traffic and treat datacenter ASNs as a signal, but they cost more per request and are usually slower, so it’s worth using them where they’re actually needed rather than by default across every worker.
Rotation strategy also has to match your queue’s retry behavior. If a task fails because a proxy got blocked, retrying with the same proxy just fails again. The retry logic needs to know to grab a different proxy from the pool on the next attempt, which means proxy selection has to be a decision made per-attempt, not something baked into the worker at startup.
Monitoring the queue, not just the output
With a single script, you know it’s working because you can watch it print progress to a terminal. With a queue and a worker pool, that visibility disappears unless you build it back in. The things worth tracking: queue depth (is work piling up faster than workers can clear it), task age (is anything sitting unprocessed for an unusually long time), retry counts (is one particular target generating way more retries than others, which usually means something changed on their end), and dead-letter volume (how much work is failing outright).
A queue depth that climbs steadily means you’re underprovisioned on workers or your workers are running slower than the intake rate. A retry count that spikes for one domain specifically, while others stay flat, is a signal that something about your traffic pattern to that domain is being flagged, and the fix is upstream of the queue: request headers, timing, session handling, not just throwing more workers at it.
Where this is overkill
None of this is worth building for a one-off job of a few hundred pages. A sequential script with basic retry logic will finish that in minutes and debugging it is trivial because everything happens in one place, in order, visible in your terminal. The queue and worker pattern earns its complexity when you’re running scrapers regularly, at volumes where a single failed request shouldn’t cost you the whole batch, or when you need the job to survive a process restart without starting over.
The honest signal that you’ve crossed that line is usually not the volume of URLs, it’s how often you find yourself re-running the whole script because of one bad row, or manually tracking which URLs already succeeded in a spreadsheet. That’s the point where the loop stops being the right tool and a queue starts paying for itself.
If you’re working out the proxy side of this, whether that’s residential versus datacenter for a specific target or how to structure rotation across a worker pool, that’s the kind of thing worth getting concrete about before you scale up. Take a look at what we’ve written on Proxy Scraping.
Get new guides and videos first — join the Telegram channel.