How to bypass Cloudflare 403s with Playwright plus residential proxies

If you run scrapers against anything mid-sized to enterprise, you’ve hit a Cloudflare 403 that made no sense. Same script, same target, worked yesterday, blocked today. No captcha, no redirect, just a flat 403 and a “sorry, you have been blocked” page with a Ray ID at the bottom. That Ray ID is Cloudflare telling you the edge network made a decision about your request before your target’s origin server ever saw it.

This matters because Cloudflare fronts a huge share of the web you probably want to scrape: SaaS pricing pages, e-commerce catalogs, ticketing platforms, job boards. When Cloudflare’s bot management decides your traffic looks automated, it doesn’t just rate-limit you, it kills the request at the edge. Datacenter IPs get flagged fast. Plain HTTP clients get flagged faster. And a naive headless browser gets flagged fastest of all, because it hands over a browser fingerprint that’s easier to profile than a bare TCP connection.

I’ve spent the last few years running scraping infrastructure for lead gen and price monitoring work, and Cloudflare 403s are the single most common wall I run into, more than PerimeterX, more than Akamai, more than DataDome. This piece is about what actually works: Playwright configured correctly, paired with residential proxies used correctly, because either one alone gets you partway and both together get you most of the way. I’ll walk through the mechanism, three worked examples with real numbers from my own runs, the failure modes that waste the most time, and what changed in how I run this stuff after two years of doing it wrong first.

background and prior art

Cloudflare’s bot detection isn’t one check, it’s a composite score. The company’s own documentation on bot scores describes a 1-99 scale where lower means “more likely automated,” built from a mix of signals: IP reputation and ASN classification, TLS handshake characteristics, HTTP/2 frame behavior, and for JavaScript-capable clients, a managed challenge that profiles the browser environment itself. A plain requests or curl call fails on the TLS and HTTP fingerprint layer before it ever gets to see a challenge. A vanilla Playwright browser can pass the network layer but fail the browser-environment layer, because stock Chromium under CDP control exposes navigator.webdriver and other automation tells that Cloudflare’s managed challenge script checks for.

The scraping community’s response to this has evolved in stages. First came user-agent spoofing, which never really worked against Cloudflare because the UA string isn’t the primary signal. Then came headless-detection patches like puppeteer-extra-plugin-stealth, which patched the obvious JS-visible tells (navigator.webdriver, missing chrome object, broken permissions API). Then came the realization that IP reputation matters as much as browser fingerprint, since Cloudflare weighs datacenter ASNs (AWS, GCP, OVH, Hetzner) heavily against you regardless of how clean your browser looks. That’s where residential proxies entered the stack, not as a silver bullet but as the fix for the half of the score that browser tuning can’t touch. Most of what I cover here is old ground independently rediscovered by a lot of operators; my contribution is what actually held up in production versus what sounded right in a GitHub issue thread.

the core mechanism

Break the block into the two layers Cloudflare actually checks, because you fix them with different tools.

Layer one: connection reputation. Before any HTTP request is parsed, Cloudflare’s edge already knows your IP’s ASN, whether that ASN is a known hosting provider or a residential ISP, whether that IP has abuse history, and roughly what country and network type it belongs to. This is why a scraper running from an AWS us-east-1 box gets challenged constantly even with a flawless browser: AWS IP ranges are heavily used for scraping and abuse, so they carry a bad prior. A residential proxy routes your traffic through a real ISP-assigned IP, typically an actual consumer connection, which starts from a much better prior. This is the single highest-leverage fix for a large share of Cloudflare 403s, and it’s why residential proxy vendors like Decodo, Bright Data, Oxylabs, SOAX, and IPRoyal exist as a category. None of them are magic, they’re all reselling access to real consumer IP pools (often via SDKs bundled into other apps), and the score you get still depends on how clean that specific IP’s history is.

Layer two: browser environment fingerprint. If Cloudflare’s edge doesn’t resolve you as clean or dirty outright, it serves a managed challenge, JavaScript that runs in your browser and checks dozens of signals: canvas rendering output, WebGL parameters, font enumeration, navigator.webdriver, timing entropy in event dispatch, and specifically for CDP-driven browsers (Playwright, Puppeteer, Selenium with the DevTools protocol), whether the Runtime.enable CDP method has been called. That last one is subtle and worth knowing: several anti-bot vendors, and researchers have documented Cloudflare doing similar probing, detect the side effects of Runtime.enable being active on the page’s JS context, because that’s a CDP artifact that a real user’s browser never produces. Stock Playwright calls Runtime.enable internally by default. Projects like rebrowser-patches exist specifically to patch this leak out of Playwright and Puppeteer.

The two layers compound. A clean residential IP with a leaky browser fingerprint still gets challenged. A stealth-patched browser on a flagged datacenter IP still gets a 403 before the page even loads. You need both pieces working together, and critically, you need the two signals to agree with each other: an IP that geolocates to Germany paired with a browser reporting en-US locale and US timezone is its own kind of tell.

worked examples

Example one: a Shopify-hosted DTC storefront, catalog monitoring. Target was a mid-size apparel storefront sitting behind Cloudflare’s default “I’m Under Attack” tier disabled but managed challenge enabled on non-checkout traffic. Running plain httpx with rotating datacenter proxies (a $50/mo pool of ~200 IPs from a generic datacenter provider) got a 403 rate of roughly 90% within the first 20 requests per IP. Switching to Playwright (Chromium, channel: 'chrome' to use the real Chrome binary instead of the bundled Chromium build) with no proxy change still sat around 55-60% blocked, because the datacenter ASN penalty dominated. Adding a residential proxy pool (Decodo, sticky session per catalog crawl, roughly 8-12 minutes per session) with Playwright dropped the block rate to about 5%, measured over 3,000 product page requests across four days. The remaining 5% resolved on retry with a fresh sticky session, so effective throughput after retries was north of 99%.

Example two: a B2B SaaS pricing and signup flow, competitive monitoring. This one had a stricter challenge because the target treats its pricing page as sensitive (common for SaaS trying to stop competitors from scraping tier changes). Plain residential proxy plus stealth-patched Playwright still got challenged with an interactive Turnstile widget roughly 1 in 4 loads, not a flat 403 but a checkbox challenge that a headless run can’t click through on its own. I paired this with CapSolver’s Turnstile-solving endpoint (a paid captcha-solving API that returns a token you inject into the page rather than trying to defeat the widget client-side) and saw the completion rate on that flow go from about 74% to 97% across a 600-request sample over two weeks. Cost was real: at CapSolver’s per-solve pricing, that added roughly $0.001-0.003 per solved challenge on top of proxy bandwidth, trivial at this volume but worth budgeting if you’re running thousands of sessions a day against a target with Turnstile enabled everywhere.

Example three: a European classifieds marketplace, listing scrape at higher concurrency. This is where sticky session length mattered more than anything else. Running 20 concurrent Playwright contexts against the target, each on a rotating (non-sticky) residential IP that changed every request, produced a 403 rate around 40% because Cloudflare’s cf_clearance cookie, issued after a successful challenge, is bound to the IP and user-agent pair that solved it. Rotate the IP mid-session and the cookie stops validating, forcing a fresh challenge on the very next request. Switching to session-sticky proxies (IPRoyal, which supports sticky sessions up to 24 hours) held one IP per browser context for the full 15-20 minute scrape run, and the block rate fell to roughly 3%, with most of that residual coming from a handful of consistently bad IPs in the pool that I started blocklisting after the second week.

edge cases and failure modes

Sticky session shorter than the challenge cookie’s validity. If your proxy’s sticky session expires before cf_clearance does (or vice versa), you’ll re-trigger challenges mid-crawl for no visible reason. Match your provider’s sticky window to your actual per-session crawl duration, and if a provider only offers short sticky windows (some cap at 10 minutes), keep your Playwright session length under that ceiling rather than fighting it.

TLS/JA3 fingerprint drift from stale browser builds. Cloudflare’s models are trained against what real, current browser releases look like at the TLS layer. An old Playwright/Chromium bundle produces a TLS ClientHello that’s subtly different from this month’s actual Chrome stable release, and that gap widens every release cycle. Keep Playwright current (npm update playwright on a real schedule, not “whenever it breaks”), and prefer channel: 'chrome' or channel: 'msedge' over the bundled Chromium when fingerprint parity matters more than binary size.

WebRTC leaking your real IP behind a proxy. Even with proxy config set correctly on the browser context, WebRTC can enumerate local and STUN-reflexive IPs that bypass the HTTP proxy tunnel entirely, exposing your actual origin IP to any page that runs a WebRTC leak check. Disable it explicitly:

const context = await browser.newContext({
  proxy: { server: 'http://gate.decodo.com:10001', username: 'user-session-abc123', password: '***' },
  extraHTTPHeaders: { 'Accept-Language': 'en-US,en;q=0.9' },
});
await context.addInitScript(() => {
  Object.defineProperty(navigator, 'webdriver', { get: () => undefined });
});

Pair this with --disable-webrtc style launch flags or a context-level policy where possible, since a leaked real IP undoes everything the residential proxy was supposed to buy you.

CDP artifacts surviving stealth patches. Stealth plugins patch the obvious JS-visible properties but don’t always cover CDP-level leaks like the Runtime.enable issue mentioned above. If you’re getting flagged despite a clean IP and a patched fingerprint, check whether your automation framework is calling CDP methods you don’t need. rebrowser-patches is the most actively maintained fix for this specific leak in both Playwright and Puppeteer as of this writing.

Geo and locale mismatch between IP and browser context. A residential IP that geolocates to, say, Osaka paired with a browser context reporting en-US locale, US timezone, and an Accept-Language header of en-US is inconsistent in a way that’s trivial for a fingerprinting script to flag, separate from anything Cloudflare-specific. Set locale, timezoneId, and Accept-Language to match the proxy exit country. If you’re running geo-targeted residential proxies (most vendors let you select country, some let you select city), this is a five-minute fix that a lot of scrapers skip. It’s worth reading the deeper breakdown on diagnosing IP bans versus fingerprint bans if you’re not sure which side of that line your blocks are coming from.

what we learned in production

The biggest mistake I made early on was treating residential proxies as the whole fix and layering more IPs on top of a bad browser setup whenever block rates crept up, instead of asking whether the browser fingerprint was the actual problem. Buying more residential bandwidth to brute-force through a fingerprint-layer block is expensive and doesn’t work, because Cloudflare’s managed challenge doesn’t care how clean your IP’s reputation is once the JS-level checks flag the session. The fix was always cheaper than more proxy spend: patch the CDP leak, match locale to geo, keep the browser binary current, and use sticky sessions long enough to hold a cf_clearance cookie for the crawl’s actual duration.

The second thing that changed how I operate: I stopped treating a 403 and a Turnstile challenge as the same failure. A 403 usually means the connection-layer score failed before any JS ran, so the fix is proxy quality and TLS fingerprint. A Turnstile widget means you passed the connection layer and Cloudflare wants interactive proof, which needs either a solving service or a genuinely human-driven session, not more IP rotation. Diagnosing which one you’re actually hitting before you start changing config saves a lot of wasted iteration, and it pairs well with disciplined cookie and session handling across rotating proxies, since a cf_clearance cookie that isn’t persisted correctly across your session store will make a fixed problem look unfixed.

One honest caveat: scraping behind an anti-bot system, even a technically successful bypass, can put you in breach of a target site’s terms of service, and in some jurisdictions there’s real legal exposure around unauthorized access depending on what you’re scraping and how. This isn’t legal advice, and if you’re operating at any real scale or against a target with sensitive data, get an actual read from counsel on your specific situation rather than assuming a working scraper means a compliant one.

If you’re dealing with other bot-detection vendors on the same target stack, the mechanics differ enough that it’s worth reading the PerimeterX bypass writeup separately rather than assuming Cloudflare tactics transfer directly, and if reCAPTCHA v3 shows up alongside Cloudflare on the same flow, handling reCAPTCHA v3 without dropping IP reputation covers the interaction between the two. For a deeper comparison of antidetect browser tooling as an alternative to hand-rolling Playwright stealth patches, antidetectreview.org/blog covers that ground in more depth than fits here.

references and further reading

• Cloudflare: Bot Scores — Cloudflare’s own explanation of the 1-99 bot score and what feeds into it.
• Playwright documentation — official docs for browser contexts, proxy configuration, and the channel option used to run real Chrome/Edge builds instead of bundled Chromium.
• rebrowser-patches — open-source patches addressing the CDP Runtime.enable detection leak in Playwright and Puppeteer.
• MDN: 403 Forbidden — the HTTP status code reference, useful baseline for distinguishing origin-level 403s from edge-level ones.
• RFC 9110, Section 15.5.4 — the IETF’s formal definition of the 403 status in current HTTP semantics.

For more troubleshooting deep-dives like this one, see the full blog index, and for a residential proxy vendor breakdown if you’re choosing a pool for this kind of work, the Decodo review covers pricing and session behavior in more detail than fits here.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-22.