Setting a block rate you are willing to live with
The zero-block myth
Every new scraping project I’ve watched start the same way: someone sets a target of zero blocks. No captchas, no 403s, no soft challenges, nothing. It sounds like a reasonable bar. It isn’t one, and chasing it wastes more time than it saves.
Anti-bot systems aren’t pass/fail gates. They’re scoring systems. A site’s defense stack looks at IP reputation, request cadence, header and TLS fingerprint consistency, behavioral signals like mouse movement or scroll timing on JS-heavy pages, and session history, then assigns a risk score. Some traffic clears instantly. Some gets a challenge page. Some gets silently degraded (wrong prices, stale data, a shadow-banned version of the page) instead of an outright block. Some gets cut off cold. A zero-block target assumes you can sit permanently in the “clears instantly” bucket. On any site that takes bot traffic seriously, you can’t, not indefinitely, because the scoring thresholds move as the site tunes its own false-positive rate against real users.
So the useful question isn’t “how do I never get blocked.” It’s “what block rate can this project absorb without the data going bad or the cost going up.” That’s a number you can plan around. Zero isn’t.
What block rate actually measures
Before you can set a target, you need to define what you’re counting, because “blocked” hides at least three different things:
Hard blocks. The request fails outright: a 403, a connection reset, an IP added to a deny list. Easy to log, easy to count.
Soft challenges. A captcha, a JS challenge page, a “verify you’re human” interstitial. These aren’t failures in the HTTP sense, they return a 200, but they’re not the data you wanted either. If you’re not specifically parsing for challenge markup, these get miscounted as successes in your logs, which quietly inflates your apparent success rate.
Degraded responses. The request succeeds and returns a 200 with a page, but the content is wrong: geo-shifted pricing, a stripped-down layout, missing fields, deliberately stale cache. This is the hardest category to catch because nothing in the transport layer flags it. You need a content-level sanity check (expected field present, price within a plausible range, page hash different from the known “blocked” template) to catch it at all.
A block rate that only counts hard blocks looks a lot better than reality. If you’re reporting a number to justify a proxy spend or a scraping cadence, decide up front which of these three you’re counting, and count all three if you can.
Setting a number before you start
An acceptable block rate isn’t a universal constant. It depends on three things specific to your project:
How time-sensitive the data is. If you’re scraping something that changes hourly and you re-check every 30 minutes, a 15% block rate on any given pass is a non-event, the next pass fills the gap. If you’re pulling something you can only realistically check once a day, that same 15% is a real hole in your dataset.
What a miss costs. A missed price point on a monitoring job that has 200 other data points that day is noise. A missed record on a one-shot compliance pull might mean redoing the whole run.
What retrying costs. Retrying through the same proxy type after a block usually just teaches the target site’s system that this identity is worth blocking harder next time. Retrying through a different pool costs more per request. If your budget assumes datacenter IP pricing and your actual block rate forces you onto residential or mobile IPs to hit your data-quality bar, your acceptable block rate and your acceptable cost per record are the same conversation, not two separate ones.
Write the number down before the run starts, not after you see the logs. “We can tolerate up to 10% hard blocks and 5% soft challenges on this target, checked daily” is a real operating parameter. “We’ll see how it goes” is how projects end up either overpaying for proxy tiers they didn’t need or shipping data nobody trusts.
What moves the needle
None of these are secrets, they’re just the mechanical levers that sit behind whatever number you land on:
Proxy type. Datacenter IPs are cheap and fast, and they’re also the easiest category for a defended site to flag, because datacenter IP ranges are known and published. Residential IPs route through real ISP-assigned addresses, which cost more per GB and generally see a lower block rate on sites that specifically screen for datacenter ASNs. Mobile IPs sit behind carrier-grade NAT, so many users share the same address, which changes the risk calculus again, usually in the scraper’s favor on strict targets, at the highest cost per GB of the three. None of these guarantee an outcome. They’re a starting risk profile, not a result.
Concurrency and cadence. A burst of requests from one IP in a short window is one of the more reliable bot signals a site can use, because it’s cheap to compute and rarely produces false positives against real users. Slower, more evenly spaced requests look more like normal traffic, at the direct cost of throughput.
Fingerprint consistency. Sites also flag mismatches: a User-Agent claiming a browser and OS combination that doesn’t match the actual TLS handshake, or a session that claims to be the same browser tab across requests but resets its cookies and headers each time. A scraper that behaves like one consistent client for the life of a session produces fewer of these mismatch signals than one that changes its declared identity mid-session.
Rotation strategy. Rotating IPs on a fixed clock instead of per-request can lower block rate on sites that specifically watch for identical-request-different-IP patterns, but it raises the cost of any one IP getting flagged, since more requests ride on it before the switch. There’s no rotation cadence that’s correct for every target; it’s a tradeoff between blast radius and detectability that you tune against your own logs.
Each of these is a dial, not a switch. Turning any of them toward “lower block rate” generally costs you something else: money, speed, or complexity. That’s the actual tradeoff surface, and it’s worth treating it as one instead of hunting for a single setting that removes the tradeoff.
When you blow past your own number
If your measured block rate comes in above what you set, the honest options are limited: slow down, reduce concurrency, move up a proxy tier, shrink the scope of what you’re pulling, or accept the gap in the data. What doesn’t work, long-term, is treating a rising block rate as an obstacle to push through with more aggressive automation. A site raising its block rate against a given traffic pattern is telling you something about where its threshold sits right now. Pushing harder against that threshold usually just moves your traffic into a worse-scored bucket, which tends to cost you retries, IP burn, and eventually the target entirely.
A note on scope, because none of this matters otherwise
A block rate target only makes sense inside a scope you’re actually allowed to be operating in. Check the site’s robots.txt and terms of service for the specific endpoints you’re hitting, stick to data that’s public and not paywalled or gated behind a login, and don’t pull personal data. None of the proxy or rotation choices above change what’s in bounds. They only change how reliably you collect data that was in bounds to begin with.
Testing your own number
You don’t need to guess at any of this. Pick a small, representative sample of the pages you actually intend to scrape, run a fixed batch through each proxy type or configuration you’re considering, and log hard blocks, soft challenges, and content-level mismatches separately for each. Run it long enough to see more than one pass, since a single burst tells you almost nothing about sustained behavior. That log, not a vendor’s marketing page, is what should set your number, because it’s the one dataset that reflects your actual target, your actual cadence, and your actual proxy pool rather than someone else’s.
Want more breakdowns like this on picking proxies and running scrapers that hold up under real conditions? Head back to the homepage for the rest of the Proxy Scraping library.
Get new guides and videos first — join the Telegram channel.