Retry logic that does not make things worse

Most scrapers don’t die from the first failure. They die from what the code does after the first failure. A request times out, the script retries immediately, the retry fails too, and now you’ve got a loop firing dozens of requests a second at a target that was already struggling to respond. That’s not resilience. That’s a self-inflicted denial of service, and it usually ends with your whole IP range in someone’s blocklist.

A good scraper retry strategy isn’t about trying harder. It’s about knowing when trying again helps, when it doesn’t, and when it actively makes your situation worse. This is the part of scraping infrastructure that rarely gets attention until it breaks something.

What a retry actually signals to the other side

Every request you send is a data point in someone else’s monitoring system. A single failed request is noise. A burst of identical requests, from the same IP, seconds apart, after a non-200 response, is a pattern. Rate limiters, WAFs, and behavioral detection systems are built specifically to catch that pattern, because it’s also what a broken client, a bug, or an actual attack looks like from the outside.

This matters because the goal of defensive infrastructure on the other end isn’t to catch “scrapers” as a category. It’s to catch abnormal load and abnormal request shapes. A retry loop with no backoff is one of the most abnormal shapes there is. It doesn’t matter how well your headers are set or how clean your proxy is if the timing pattern alone is enough to flag the session.

The practical takeaway: your retry logic is part of your fingerprint, not separate from it.

Not all failures are the same

The first mistake in most retry code is treating every non-success response the same way. They’re not the same, and they don’t call for the same response.

A connection timeout or a DNS failure is usually a network-layer problem, on your end, the proxy’s end, or somewhere in between. Retrying with a different proxy or a fresh connection is often reasonable, because the failure has nothing to do with the target’s willingness to serve you.

A 5xx response means the target’s own infrastructure is struggling. Hammering it with retries adds load to a system that’s already in trouble. This is where backing off matters most, both for your own success rate and because piling on a degraded service is a bad way to operate.

A 429 is the target telling you, explicitly, that you’re over its rate limit. Many well-behaved APIs send a Retry-After header with that response. Ignoring it and retrying on your own schedule is not a resilience technique, it’s just not reading the instructions you were given.

A 403, a CAPTCHA page, or a 200 response with an empty or malformed body (a soft block dressed up as success) is a different category entirely. This is the target’s detection layer making a decision about the request, not a transient failure. Retrying the same request immediately, from the same proxy, with the same fingerprint, doesn’t fix anything. It just repeats the exact signal that got you blocked in the first place, and it can escalate a soft block into a longer or harder one. The right response here is to stop, log it, and route future requests through a different session identity rather than loop.

Treating a 429 like a timeout, or a soft block like a timeout, is the single most common source of retry storms.

Exponential backoff and why jitter matters

Once you’ve decided a retry is warranted, the interval matters as much as the decision. Fixed-interval retries (retry every 2 seconds, always) create synchronized load: if a target has a brief hiccup and a thousand clients all retry on the same clock, the retry wave itself can look like the problem.

Exponential backoff, doubling the wait after each failed attempt, spreads that load out over time. Adding jitter, a small random offset on top of the backoff interval, spreads it out across clients too, so you’re not producing a metronomic request pattern that’s trivially easy to fingerprint. A request that lands at 2.0s, then 4.0s, then 8.0s after failure looks like a script. A request that lands at 2.1s, then 4.6s, then 7.8s looks like normal variance.

Backoff also needs a ceiling and a maximum retry count. Doubling forever isn’t a strategy, it’s just delaying the moment you admit the request isn’t going to succeed. Two or three retries with real backoff between them, then give up and move on, is usually more productive than five or six attempts stacked on a target that’s already told you no.

The proxy rotation trap

Rotating proxies are often treated as a retry mechanism by default: request fails, grab a new IP, try again. This works for genuine network-layer failures, but it can quietly turn into its own problem.

If every failure triggers a proxy swap, a scraper under sustained pressure from a target can burn through a proxy pool fast, and it does so in a way that’s visible from the target’s side as a wide spread of source IPs all requesting the same resource in a short window with the same failure-then-retry pattern. That’s a distinctive shape too, arguably more visible than a single IP retrying, because now the target’s detection system sees a whole IP range behaving in lockstep.

There’s also a cost you pay even if nothing gets flagged: burning through residential or mobile IPs on retries against a target that’s already rejecting you wastes pool capacity you’ll want later, for requests that were actually going to succeed. Rotation should be a deliberate choice tied to the failure type, not an automatic reflex on every non-200.

Circuit breakers: knowing when to stop

A circuit breaker is a simple idea borrowed from general software reliability practice: after a certain number of failures against a given target in a given window, stop sending requests to it entirely for a cooldown period, instead of continuing to retry one at a time. This protects two things at once. It protects the target from a client that keeps hammering a resource that’s clearly unavailable or actively blocking. And it protects your own operation from spending compute, proxy bandwidth, and time on a lane that isn’t going to open back up in the next thirty seconds.

The threshold and cooldown period depend on what you’re scraping and how much you know about the target’s own rate windows, but the principle holds everywhere: if the last five requests to a domain all failed the same way, request six should not fire on the same schedule as request one. Something has to change, either the timing, the target, or the decision to stop for now.

Idempotency: only retry what’s safe to repeat

Retry logic assumes that repeating a request is safe. That’s true for reads, most GET requests fetching a page or an API resource. It’s not automatically true for anything that has a side effect, submitting a form, triggering an action, or calling an endpoint that isn’t guaranteed to be idempotent. Retrying those blindly on a timeout can double up the action if the first attempt actually succeeded and only the response was lost. Before wiring retries into any scraper, it’s worth confirming which requests are pure reads and which aren’t, and keeping retry logic scoped to the former.

What good retry telemetry looks like

None of this works without visibility. A scraper that logs “request failed, retrying” with no detail can’t tell you whether it’s dealing with network flakiness, a target’s rate limiting, or a block. At minimum, retry logs should capture the response code or error type, the target, the proxy or session used, and the backoff interval applied. Over time this data tells you which targets need slower pacing, which proxy types are producing more transient failures, and where your retry budget is actually going. Without it, tuning retry behavior is guesswork, and guesswork under production load tends to guess wrong in the expensive direction.

Putting it together

Retry logic that doesn’t make things worse comes down to a few habits: tell failure types apart before deciding to retry at all, back off with real jitter instead of a fixed clock, treat proxy rotation as a deliberate response to a specific failure rather than a default action, stop entirely with a circuit breaker once a target is clearly not going to respond, and keep retries scoped to requests that are actually safe to repeat. None of this guarantees a scraper won’t get blocked. It just means that when something does fail, the response doesn’t turn a small problem into a bigger one.

If you want more of this kind of practical, no-hype breakdown of how scraping infrastructure actually behaves under load, head back to the Proxy Scraping home page for the rest of our writeups.

Get new guides and videos first — join the Telegram channel.