Rate limiting and retry strategies that avoid block escalation

Most scraping teams treat a 429 or a 403 as an inconvenience to code around. Catch the error, sleep a bit, try again. That instinct is reasonable for a flaky internal API. It is close to the worst thing you can do against a target running a modern bot management stack, because the retry itself is often the signal that turns a soft throttle into a permanent block.

I run scraping infrastructure out of Singapore across a mix of residential, mobile, and datacenter proxy pools, and the single most expensive mistake I’ve watched teams make, including my own crews in earlier years, is writing retry logic that was designed for AWS or Stripe rate limits and pointing it at a target defended by Akamai, Cloudflare, or PerimeterX. Those systems don’t just count requests per minute. They score the shape of your retries: the timing, the concurrency pattern, whether you back off at all, whether every retry comes from the same IP with the same TLS fingerprint. Get that wrong and you don’t get a slower response, you get the whole subnet quarantined.

The stakes are proxy spend and time, not abstractions. A blown IP range on a residential plan is gigabytes you paid for and can’t use. A hard ban on a /24 you’re routing through costs you the rest of that pool until it’s rotated out. This piece is about the mechanics of doing retries and rate limiting in a way that respects the target’s actual capacity and doesn’t hand its bot detection a free confirmation that you’re automated. I’ll assume you already know what a 429 is and have seen Retry-After in a response header before.

background and prior art

The vocabulary here comes from two different worlds that got smashed together. Rate limiting as a concept is old, token bucket and leaky bucket algorithms show up in networking QoS literature going back to the 1980s, and HTTP formalized the polite version of it with RFC 6585, which defined status code 429 Too Many Requests in 2012. The Retry-After header itself predates that, it’s been part of HTTP since the original HTTP/1.1 spec and was meant for cooperative systems: a client and server owned by different parties who both want the exchange to succeed, just not too fast.

Exponential backoff comes from the same cooperative lineage, mostly popularized for engineers by cloud vendors solving their own API throttling problems. The AWS Architecture Blog post on exponential backoff and jitter from 2015 is still the clearest writeup of why naive exponential backoff without randomization causes synchronized retry storms, and it’s the source most retry libraries (tenacity in Python, polly in .NET) cite for their jitter implementations. GitHub’s own REST API rate limit documentation is a good real-world reference for what a cooperative, well-documented limit looks like: fixed budget, a reset timestamp in the response headers, no ambiguity.

Scraping targets are not cooperative in that sense. A retailer’s storefront or a professional network’s search endpoint isn’t publishing a rate limit spec, it’s running a WAF that treats retry patterns as a classification feature alongside TLS fingerprint, header ordering, and mouse movement on the JS challenge page. That’s the gap this article sits in: the mechanics of rate limiting and retries are the same math, but the target on the other end is actively trying to distinguish your retry logic from a browser’s, and getting that distinction wrong is what causes escalation from a rate limit to an IP ban to a subnet ban.

the core mechanism

Three ideas do almost all the work: local rate limiting before you get throttled, backoff with real jitter when you do, and a notion of block escalation levels so your response matches the severity of what actually happened.

Local rate limiting. Don’t wait to get a 429 to learn a target’s capacity, budget for it up front with a token bucket per identity (per proxy IP, per session, whichever the target keys its limits on). If you don’t know the ceiling, start conservative and raise it based on clean response streaks, not based on how fast you want the job to finish.

import time
import threading

class TokenBucket:
    def __init__(self, rate_per_sec, capacity):
        self.rate = rate_per_sec
        self.capacity = capacity
        self.tokens = capacity
        self.updated = time.monotonic()
        self.lock = threading.Lock()

    def acquire(self):
        with self.lock:
            now = time.monotonic()
            self.tokens = min(self.capacity, self.tokens + (now - self.updated) * self.rate)
            self.updated = now
            if self.tokens < 1:
                wait = (1 - self.tokens) / self.rate
                time.sleep(wait)
                self.tokens = 0
            else:
                self.tokens -= 1

Keep one bucket instance per proxy identity, not one global bucket for the whole job. A global bucket hides the fact that individual IPs are each getting hammered above the target’s real per-IP ceiling.

Backoff with full jitter. When a limit is hit anyway, exponential backoff without randomization causes every worker that got throttled at the same moment to retry at the same moment, which just recreates the spike. AWS’s “full jitter” formula, sleep = random(0, min(cap, base * 2^attempt)), spreads retries out instead of bunching them:

import random

def backoff_sleep(attempt, base=1.0, cap=60.0):
    ceiling = min(cap, base * (2 ** attempt))
    return random.uniform(0, ceiling)

Always check for Retry-After before falling back to computed backoff. Parse it correctly, it can be an integer number of seconds or an HTTP-date string per the spec, and a lot of retry code silently breaks on targets that send the date format.

Block escalation levels. This is the piece most retry code skips entirely, and it’s the one that actually prevents damage. Treat these as distinct states with distinct responses, not one undifferentiated “error, retry” bucket:

• Soft throttle: slower responses, occasional captcha, intermittent 429s that clear on their own. Response: reduce concurrency on that identity, apply the token bucket more conservatively, keep the same IP.
• Hard block: sustained 403/429 on every request from one identity. Response: stop retrying on that IP immediately, rotate to a fresh identity, put the burned one in a cooldown queue.
• Subnet or ASN block: multiple IPs from the same pool segment start failing simultaneously. Response: don’t retry within that segment at all. Retrying against an already-flagged range just confirms automated traffic to the WAF and, on some anti-bot vendors, extends the blocklist duration rather than testing it. Pull the whole segment and move to a different proxy pool or geography.

The mistake I see constantly is code that only implements the first two states, or that implements all three but keeps retrying inside a blocked subnet because “the retry logic is generic.” It isn’t generic once you’re past soft throttle. More detail on telling these apart is in debugging 429 errors, rate limits, and proxy quality, which walks through separating a proxy-quality problem from an actual rate limit.

worked examples

Example 1: G2.com review pages via Decodo residential. Running a review-monitoring job through Decodo’s rotating residential gateway on sticky sessions, we were hitting 429s at around 55 requests inside a 60-second window per exit IP, at a request rate of roughly 1/sec. That told us the real bucket was close to 60 tokens with a 1/sec refill. Setting the client-side token bucket to 50 requests per minute per IP with 200 to 800ms of jitter between requests took the 429 rate on that job from about 6% of requests down to effectively zero over a 48-hour run. The fix wasn’t smarter retries, it was staying under the ceiling in the first place.

Example 2: Booking.com search endpoint behind Akamai. This is the case that taught me the escalation-level lesson the hard way. A search-scraping job hit a page-level throttle (a handful of 403s), and the retry logic fired three back-to-back retries within about 2 seconds because the base backoff was too short and had no jitter. Those rapid, uniform retries from the same IP within a 2-second window were enough to convert a transient throttle into a 15-minute full IP block on Akamai’s side. After switching to a 4-second base with doubling capped at 60 seconds, full jitter, and dropping concurrency on that domain from 20 threads to 6, the 403 rate across a comparable run dropped from around 9% of requests to under 0.4%. Same target, same content, the only change was the shape of the retries.

Example 3: LinkedIn people-search via session cookies. Session-bound targets like this rate-limit the session, not just the IP, so an identity swap on the proxy alone doesn’t reset anything. We measured a soft throttle appearing consistently around 80 requests within a rolling hour on one authenticated session. Instead of retrying past that and eating a temporary lock, we force-rotated the session (fresh cookie jar, fresh login state) at 65 requests, roughly 80% of the observed ceiling, and never triggered the hard lock across several weeks of runs. Session and cookie mechanics for this pattern are covered in more depth in cookie and session handling at scale across rotating proxies.

edge cases and failure modes

Soft blocks disguised as HTTP 200. Some anti-bot systems don’t bother with 429 or 403 once they’re confident you’re a bot, they just serve a 200 with an empty results array, a honeypot page, or stale cached content. Retry logic keyed only on status code sails right past this because there’s nothing to retry, the request “succeeded.” Counter: validate the response shape, not just the status, hash or fingerprint the expected payload structure and treat a mismatch as a block signal that feeds the same escalation logic as a 429.

Synchronized retries across a worker fleet. If every worker computes backoff off the same shared clock or the same seed, you get a thundering herd where 40 workers throttled at once all retry at once, recreating the spike that got them throttled. This is exactly what AWS’s jitter article addresses, and the fix is full jitter per worker plus staggering the initial start times of your workers by a few hundred milliseconds so they don’t even start in lockstep.

Retry-After parsed wrong. I’ve seen production code that assumes Retry-After is always an integer number of seconds, then throws or silently no-ops when a target sends the HTTP-date form instead. Test both formats. If the header is missing or malformed, fall back to your computed backoff rather than crashing the retry path or, worse, retrying immediately.

Retrying inside a proxy that’s already flagged. Retry code that doesn’t track block state per-identity will happily keep hammering an IP that just got hard-blocked, because from the retry logic’s point of view it’s just another 403 to back off from. On WAFs that track repeat offenses, retrying against an already-blocked IP can extend the block window rather than test it. Track block state explicitly, quarantine a burned IP for a defined cooldown (we use 30 to 60 minutes minimum before it’s eligible for reuse), and don’t let generic retry code touch it during that window.

No circuit breaker at the job level. Per-request retry logic with no aggregate check will burn through an entire proxy plan’s monthly allotment chasing a target that’s simply closed for the day, maintenance window, geo-block, whatever. Add a circuit breaker that watches the error rate across the whole pool, if more than roughly 30% of requests in a 5-minute rolling window are 429 or 403, stop the job and alert rather than let individual request-level retries keep spending IPs one at a time. This is the difference between losing a few requests and losing the whole pool. For distinguishing whether that error spike is actually the proxy’s fault versus the target’s fingerprinting catching you, see diagnosing IP bans: when it’s the proxy vs when it’s your fingerprint.

what we learned in production

The single change that cut our block rate the most wasn’t a smarter backoff formula, it was moving the unit of accounting from “request” to “session budget.” Once every scraper had a hard ceiling on requests-per-identity before a forced rotation, most of the escalation problems upstream of that just stopped happening, because we were rotating out before the target’s own counters got anywhere near their limit. Backoff and jitter still matter for the requests that do get throttled, but they’re a second line of defense, not the primary control.

The other lesson, and this one cost us a chunk of a monthly Decodo allotment before we internalized it, is that circuit breakers are not optional once you’re running more than a handful of concurrent workers. A retry-per-request design has no mechanism to notice “this whole target is hostile right now” until every single worker has independently discovered that the hard way, by which point you’ve burned proxies you didn’t need to burn. We now bake a pool-level circuit breaker into every scraper by default, even the small ones, because the cost of adding it up front is minutes and the cost of not having it during an unexpected WAF update is real money. None of this is about beating a target permanently, targets update their bot detection and you adjust again, it’s about not paying for the same mistake twice.

Worth noting since this touches account and session behavior: if your scraping work also involves managing many logged-in accounts against a target rather than anonymous requests, the retry and rate-limit discipline here overlaps with account isolation practices covered on multiaccountops.com, that’s a related but distinct discipline worth reading if sessions, not just IPs, are what’s getting flagged. And always check a target’s terms of service and robots.txt before scraping it at scale, this article covers the technical mechanics, not the legal ones, and it isn’t legal advice.

references and further reading

• RFC 6585 - the IETF spec defining HTTP 429 Too Many Requests
• MDN: Retry-After header - format reference for both the seconds and HTTP-date forms
• AWS Architecture Blog: Exponential Backoff and Jitter - the source most retry libraries cite for full jitter
• GitHub REST API rate limit docs - a clean example of a cooperative, well-documented rate limit for comparison

More troubleshooting deep-dives like this one are in the blog index, including the companion piece on bypassing Cloudflare 403s with Playwright and residential proxies if your escalation problem is starting at the TLS/JS challenge layer rather than the rate limiter.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-22.