Diagnosing IP bans: when it is the proxy vs when it is your fingerprint

The support message I get most often from people running scrapers out of Singapore and elsewhere is some version of “the proxies stopped working.” Nine times out of ten, what actually broke is not the proxy pool. It is the fingerprint sitting on top of it: the TLS handshake shape, the header order, the canvas hash, the cookie that survived three IP rotations it should not have survived. Swapping proxies at that point does nothing except burn IPs and money.

This matters because the two failure modes look identical from the outside. A 403, a CAPTCHA wall, a soft redirect to a login checkpoint, a silently empty response body. If it is genuinely the proxy (bad ASN reputation, an IP that is already blacklisted, a subnet Cloudflare has already fingerprinted as a known hosting range) then rewriting your HTTP client is a waste of an afternoon. If it is your fingerprint, buying a bigger proxy pool just multiplies the number of IPs you burn per hour without fixing anything, and you will be back in the same ticket queue next week.

This piece assumes you already know what a proxy is and have read the residential-vs-datacenter 101 content. I want to walk through how I actually separate the two causes on a live target, the tools I use to do it, three worked examples with real numbers from our own testing, and the failure modes that still catch us after doing this for years.

background and prior art

IP-based blocking used to be almost the whole game. Rate limit by IP, blacklist by IP, geofence by IP. That model broke down once residential proxy pools became commercially available at scale (Luminati, now Bright Data, opened this market around 2014), because an attacker with a large enough residential pool could look, IP-for-IP, indistinguishable from real traffic. Sites had to move detection up the stack, from “who is this IP” to “what is this client.”

The two research threads that matter most here both predate the current anti-bot industry by years. The Electronic Frontier Foundation’s Panopticlick project, now continued as Cover Your Tracks, demonstrated as far back as 2010 that browser configuration alone, canvas rendering, installed fonts, screen size, plugin list, was often unique enough to identify a single machine without touching its IP at all. Separately, in 2017, engineers at Salesforce published JA3, a method for hashing the shape of a TLS ClientHello message (cipher order, extension order, elliptic curve list) into a short fingerprint that reliably identifies which TLS library and version produced a given connection, independent of anything in the HTTP layer above it. Both ideas were originally framed as privacy and threat-hunting research. Commercial bot management vendors, Akamai, Cloudflare, PerimeterX (now HUMAN Security), DataDome, and Kasada among them, absorbed variants of both into their detection stacks over the following few years, which is why a client with a “clean” residential IP can still get blocked in ten requests today.

the core mechanism

I think about detection as five layers, and I test each one independently before touching the others.

Network layer. ASN reputation, geolocation consistency, and whether the IP has already been fingerprinted as a proxy, VPN, hosting range, or Tor exit. Databases like IPQualityScore and ipinfo.io’s privacy detection endpoint exist specifically to sell this signal to the sites you are scraping. A residential IP from a real ISP block starts clean here. A datacenter IP from a cheap, well known hosting ASN is often flagged before a single byte of your request is inspected.

Transport layer. The TLS ClientHello. Every TLS client library has a default ordering for its supported cipher suites and extensions, and that ordering is deterministic per library and version, which is exactly what makes JA3 hashing work. RFC 8446, the TLS 1.3 spec, defines the ClientHello structure that gets fingerprinted, but it says nothing about ordering, that part is left entirely up to the implementation. Python’s requests, built on OpenSSL, produces a ClientHello shape that is trivially distinct from Chrome’s, which is built on BoringSSL. You can set a Chrome User-Agent header, and the connection is still transparently a Python process the moment the TLS handshake completes, before your HTTP headers are even read.

HTTP layer. Header presence, casing, and order. Real Chrome sends headers in a specific order and with specific casing that differs from what most HTTP libraries send by default. It also matters whether your Accept-Language header agrees with the geography your IP claims to be in, and whether you are sending the User-Agent Client Hints headers modern Chrome uses instead of relying purely on the User-Agent string (see Chrome’s own notes on this transition in its User-Agent Reduction documentation). Beyond TLS, HTTP/2 has its own fingerprint surface, frame order and pseudo-header order, that a lot of “stealth” HTTP clients patch TLS for and then forget entirely.

Application layer. Canvas rendering hash, WebGL renderer string, font enumeration, audio context fingerprint, the navigator.webdriver flag, and whether your claimed timezone matches your IP’s geolocation. This only applies once you are running an actual browser engine (Playwright, Puppeteer, Selenium), not a raw HTTP client, but it is where most stealth failures happen because it is the layer people patch last.

Behavioral and session layer. Request timing that is too regular, absence of any mouse or scroll events on a real browser session, and critically, session cookie continuity across IP changes. If a site issues you a session cookie and then sees that cookie attached to five different IPs in six minutes, that is a stronger signal than any individual IP’s reputation.

The diagnostic method is simple once you separate the layers: change one variable at a time. Hit the target with no proxy at all from a clean connection, to get a baseline for whether your fingerprint alone passes. Then hit it through the proxy with the crudest possible client (bare curl, no header spoofing) to isolate whether the IP itself is the problem. Then add your real client stack back in, one layer at a time. Tools worth keeping bookmarked for this: browserleaks.com and tlsfingerprint.io for viewing your own JA3/JA4 hash live, and Cover Your Tracks for a quick read on how unique your browser-layer configuration is against the current EFF dataset.

A quick illustration of isolating the transport layer, using curl-impersonate, which ships prebuilt curl binaries that reproduce a real browser’s TLS ClientHello instead of curl’s own default:

# default curl: OpenSSL ClientHello shape, easy to fingerprint as "not a browser"
curl -s -o /dev/null -w "%{http_code}\n" https://target-site.example

# curl-impersonate: ClientHello shaped like real Chrome 116
curl_chrome116 -s -o /dev/null -w "%{http_code}\n" https://target-site.example

Same IP, same headers if you set them identically, different result, because the difference lives entirely in the handshake, before your HTTP request line is even sent.

worked examples

Example 1: LinkedIn checkpoint wall, fingerprint problem. We were pulling public profile pages through a residential rotating pool from Decodo (reviewed in detail on this site) using plain Python requests with a spoofed Chrome User-Agent header and nothing else changed. Across a pool of roughly 500 rotating residential IPs, we hit LinkedIn’s identity checkpoint within the first 20 to 30 profile views, consistently, regardless of which IP served the request. Rotating harder did not move the number. Switching only the HTTP client, to Python’s tls_client library, which ships a ClientHello shaped like real Chrome instead of OpenSSL’s default, on the exact same Decodo pool, same request pattern, same headers, pushed us past 2,000 profile views before the first checkpoint appeared. Nothing about the IPs changed. The fingerprint was the entire problem.

Example 2: real estate listing site, proxy problem. For a listings-monitoring job (the kind covered in our realtor.com scraping guide), the client stack was already solid: Playwright with stealth patches applied, a current Chrome-shaped JA3, correct header order, randomized timing between page loads. Running it through a cheap datacenter proxy block from a low-cost provider, we got 0 successful page loads out of 50 attempts, instant 403s. Switching only the proxy, same script, same fingerprint stack untouched, to a mobile 4G proxy pool similar to what GeoSurf offers, got us 48 successful loads out of the next 50 attempts. The datacenter subnet was sitting on a known-hosting-ASN block list before our request ever reached the fingerprint-checking layer. No amount of client-side polish fixes a block that happens at the network layer.

Example 3: e-commerce price monitor, session-continuity problem. This one is the hybrid case people miss most. Monitoring pricing on a site behind a PerimeterX-style bot manager, we were rotating IP on every single request (standard practice for scale) but reusing one long-lived session cookie across all of them, because the code had been written for a target that did not care about session binding. We got flagged after roughly 6 to 8 IP changes within a single cookie’s active lifetime, well before any individual IP looked suspicious on its own. The fix was not a better proxy or a better fingerprint, it was binding one IP to one cookie for its full lifetime and only rotating both together, never independently. This is the same discipline covered in more depth in our piece on cookie and session handling at scale across rotating proxies.

edge cases and failure modes

Sticky-session TTL shorter than your app session. Most residential proxy providers sell “sticky sessions” that hold one IP for a fixed window, often 10 minutes. If your application session outlives that window, the proxy silently swaps your IP mid-session and you get an involuntary version of the Example 3 problem. Counter-strategy: match your sticky window to your actual session length, or force a full session re-auth every time the proxy is due to rotate, never let it happen underneath you.

Over-fitting to a stale fingerprint. Chrome ships new stable releases roughly every four weeks, and its TLS extension ordering and Client Hints headers shift with some of those releases. A fingerprint profile that was indistinguishable from real Chrome in January can be a stale, easily clustered outlier by March if you never refresh it. Counter-strategy: check your active fingerprint profile against current Chrome stable on browserleaks.com monthly, and version your client’s fingerprint templates the same way you’d version anything else that decays.

IPv6 leaking your real origin. A lot of proxy setups tunnel IPv4 only. If the host machine still has a working IPv6 route, some HTTP clients and browsers will use it directly for IPv6-capable targets, leaking your real ASN straight past the proxy you just paid for. Counter-strategy: check with an IPv6 leak test through your actual client stack, not just a browser tab, and either disable the IPv6 route on the box or move to a proxy provider that tunnels both stacks.

TLS-perfect, HTTP/2-wrong. Several popular “stealth” patches fix the JA3 hash and stop there. HTTP/2 frame order and pseudo-header order are a separate fingerprint surface that Akamai in particular has published research on using for bot detection, and a client that nails TLS but sends HTTP/2 frames in library-default order is still distinguishable. Counter-strategy: verify with an impersonation client that patches both layers together, not a TLS-only patch bolted onto a stock HTTP/2 stack.

One canvas hash across an entire fleet. If every worker in your scraping fleet runs the identical headless browser image with no per-instance randomization, every one of your “different visitors” served from hundreds of different IPs shares one canvas and WebGL hash. That is a trivial server-side clustering signal, hundreds of unique IPs collapsing into one machine identity. Counter-strategy: randomize canvas noise and WebGL parameters per worker, not per fleet. This is the entire product category antidetectreview.org covers in depth if you want vendor-by-vendor comparisons of how well different anti-detect browsers actually do this.

Treating a soft block as a hard ban. Some targets do not 403 you, they serve a technically-valid but degraded response: an empty listing array, a paywall snippet instead of the article body, a page missing the data block your scraper actually wants. If your monitoring only checks HTTP status codes, this failure mode is invisible for weeks. Counter-strategy: assert on response shape and a minimum expected data size, not just status code 200.

One general caveat worth stating plainly: whether a given scraping target’s terms of service allow this kind of access at all is a separate question from the technical one this article covers, and it varies by site and by jurisdiction. This is not legal advice, and if a target’s ToS or a jurisdiction’s computer-access law is a live concern for what you are building, that is a question for a lawyer, not a blog post.

what we learned in production

The single highest-leverage habit we adopted was refusing to change more than one variable per test. Early on, when something broke, the instinct was to swap the proxy provider and the client library and the headers all at once, because we wanted the fastest possible fix. That approach reliably produced a fixed symptom and zero understanding of why, which meant the same failure would reappear a month later against a slightly different target with no diagnostic history to fall back on. Now every target gets a small regression check, a scheduled request that logs status code and response byte size daily, so a fingerprint drift or a proxy subnet going bad shows up as a trend line instead of an angry client message.

The second habit is keeping a plain changelog per target: which fingerprint profile last worked, which proxy type (residential, mobile, datacenter) was in use, and the date it was last verified. It sounds unglamorous next to talking about JA4 hashes, but it is the difference between a five-minute fix (“their Chrome version moved, bump the template”) and a half-day debugging session repeating tests you already ran three weeks ago. Related reading on our site if you are building this discipline out further: bypassing PerimeterX shields goes deeper on the specific detection signals that vendor watches for.

references and further reading

• EFF Cover Your Tracks — live test of how unique your browser fingerprint is, successor to the original Panopticlick research.
• RFC 8446, TLS 1.3 — the protocol spec defining the ClientHello structure that JA3/JA4 fingerprinting hashes.
• Salesforce Engineering: TLS Fingerprinting with JA3 and JA3S — the original method paper behind the fingerprinting technique most bot managers use today.
• Chrome Developers: User-Agent Reduction — Google’s own documentation on why User-Agent strings are being replaced with Client Hints, and what that changes for header-based detection.
• MDN Web Docs: HTTP headers — reference for header semantics and ordering conventions worth knowing before you try to spoof them.

For more on the site: the blog index has the full library of scraping and proxy guides, including the vendor reviews and how-to-scrape series referenced above.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-22.