← all guides

What to check before you scale a scraper ten times

A scraper that runs clean at low volume tells you almost nothing about how it’ll behave at ten times that volume. I’ve watched this happen more times than I can count on our own farm: a job that pulls a few thousand pages a day without a hiccup gets pointed at ten times the targets, and within an hour half the requests are timing out, block rates are climbing, and nobody can tell if it’s the proxies, the target site, or the code. Scaling a scraper isn’t “buy more proxies and turn up the concurrency.” It’s a set of assumptions your small-scale run got away with, and scale is what exposes them.

This isn’t a guide to getting past anything. It’s a checklist for what to look at before you 10x a job that already works, written from the operator side of proxy infrastructure.

Your proxy math almost certainly doesn’t scale linearly

At small volume, a shared pool of a few hundred proxies covers you because your request rate per proxy stays low enough that nothing looks unusual. Multiply your request volume by ten without multiplying your pool, and the request rate per IP goes up by the same factor. That’s the single most common cause of a scraper that “used to work fine” suddenly getting flagged constantly.

Before you scale, work out your actual requests-per-proxy-per-minute at the new volume, not the old one. If you’re on a rotating residential or mobile pool, check whether the provider’s pool size and rotation frequency can actually absorb the new request rate, or whether you’re about to hammer the same subset of IPs harder than before. Datacenter IPs are cheaper and more predictable in supply, but they’re also the easiest for a target site to fingerprint as a block by ASN or IP range, so a bigger datacenter pool from the same subnet doesn’t buy you as much headroom as it looks like on paper. None of this is about hiding traffic. It’s about not accidentally concentrating ten times the load onto the same handful of exit points.

The target site’s own systems scale with you, not against you

Every site running anti-bot infrastructure sees your traffic in aggregate: request rate from a given IP or ASN, timing patterns, header consistency, session behavior, JS challenge completion rates. A detection system doesn’t care that your scraper “worked before.” It cares whether the traffic pattern it’s seeing right now looks like a human browsing or a script hammering endpoints. Scaling a job means that pattern gets ten times more visible, ten times faster, which is exactly when previously-invisible tells (identical timing between requests, missing referer headers, a user agent that never changes) start tripping thresholds they never reached before.

The honest takeaway here isn’t “here’s how to look more human.” It’s that a compliant, respectful scraper (reasonable rate limits, honoring robots.txt where it applies, not hammering endpoints that clearly aren’t meant for bulk access) stays further from those thresholds at any scale, and a scraper that was already borderline at low volume will get caught immediately at high volume. Scale doesn’t create new problems so much as it makes existing ones impossible to ignore.

Concurrency isn’t just a proxy problem

A lot of people assume that if the proxies can handle it, the scraper can handle it. In practice, the first thing that falls over when you scale a job is usually your own concurrency handling, not the proxy layer. Connection pools that were sized for 20 simultaneous requests choke at 200. Retry logic that quietly retried three times at small scale now triples your effective request volume against the target, which compounds the rate problem above. DNS resolution, TLS handshakes, and session/cookie management all have their own limits that don’t show up until you’re running enough workers at once to hit them.

Before scaling, check what your scraper actually does under concurrency: does it queue requests sensibly, or does it fire everything at once and let the proxy layer sort it out? Does each worker keep its own session state, or are sessions getting crossed between proxies mid-flow, which looks exactly like account or session hijacking to a detection system watching for it? A scraper that was “clean” at low concurrency because it happened to run mostly sequential can become messy fast once you parallelize it for scale.

Your parsing and storage layer has limits you haven’t hit yet

This one gets overlooked because it has nothing to do with proxies or blocking. If your scraper writes to a database, a CSV, or a queue, that write path has throughput limits too. At low volume, a slow database write or a single-threaded parser doesn’t matter because the requests aren’t coming in fast enough to expose it. At ten times the volume, that bottleneck becomes the actual ceiling on your job, and requests start backing up behind it, which can cause its own timing problems on the scraping side (connections held open longer than they should be, retries firing because a response wasn’t consumed in time).

Before scaling, load test the non-scraping parts of the pipeline separately: parsing, deduplication, writes to storage. It’s a common mistake to spend a week tuning proxy rotation and rate limits only to find the actual bottleneck was a single-threaded insert into a database with no batching.

Cost math changes in ways that aren’t obvious upfront

Scaling ten times doesn’t mean ten times the proxy bill in every case, but it also doesn’t mean the bill stays flat. Residential and mobile proxy pricing is usually metered by bandwidth or by IP allocation, and both usage patterns shift at scale: more retries from a higher block rate mean more wasted bandwidth on failed requests, and a bigger pool requirement to keep per-IP request rates sane means a bigger allocation cost. Run the math on what a realistic failure and retry rate does to your effective cost per successful page, not just the sticker price per GB or per IP. A job that looks profitable at small volume based on optimistic assumptions about block rate can look very different once failures scale along with successes.

What to actually check before you flip the switch

A short, practical list, in the order I’d work through it:

  • Recalculate requests-per-proxy-per-minute at the new volume and confirm your pool size and rotation actually support it.
  • Separate your ASN and IP diversity from your raw proxy count. Ten times the IPs from the same subnet isn’t the same as ten times the diversity.
  • Load test your parsing, dedup, and storage layers independently of the scraping traffic.
  • Check your retry logic for amplification: does a failure trigger one retry or an unbounded loop that multiplies your effective request rate?
  • Confirm session and cookie handling stays consistent per worker under concurrency, not crossed between proxies.
  • Re-check the target site’s terms and any robots.txt or rate guidance at the volume you’re about to run, not the volume you tested at.
  • Model your cost per successful page at a realistic failure rate, not a best case one.

None of this guarantees a scaled job runs cleanly. No proxy type, provider, or configuration is undetectable or risk free, and anyone telling you otherwise is selling something. What actually holds up at scale is a scraper built with margin: proxy capacity ahead of your request rate, a pipeline that doesn’t choke on its own throughput, and request behavior that stays reasonable because it was designed to, not because it happened to fly under the radar at a smaller size.

If you’re building out proxy-based scraping infrastructure and want more from us on picking the right proxy type for the job or reading provider comparisons honestly, you can find the rest of our writing on the home page.

Get new guides and videos first — join the Telegram channel.

proxies
Need proxies that survive the block wall?

Singapore Mobile Proxy runs real 4G/5G mobile IPs on rotating SIMs — the carrier-grade addresses most of these targets still trust.

see plans →
read on
More scraping guides

The rest of the field manual: target-site playbooks, library walkthroughs, provider reviews, and anti-bot troubleshooting.

browse all guides →