What a JavaScript challenge page is checking
The page that says it’s checking your browser
A person visiting a site sometimes sees a plain page for a second or two, often with a spinner and a line like “checking your browser” or “verifying you are human”. Then the real page loads and they forget about it. Many people never see one at all, because it clears too fast to notice, or never appears for them.
A scraper hitting the same site can get a very different experience. The response is that same interstitial page, every time. A plain HTTP client receives some HTML and a script and stops there, because it doesn’t run scripts. A headless browser may run the script and then land on the same page again, or on a harder challenge. The logs fill with a status code the team doesn’t expect, and the extracted data is empty.
That page is a JavaScript challenge, served by the CDN or web application firewall in front of the site, before the request ever reaches the site’s own servers. This piece explains, at a high level, what those challenges evaluate, why they show up for some clients and not others, and what a scraper should take from one.
It’s deliberately different from the companion pieces. The piece on what a CAPTCHA actually measures covers visible puzzles and why proxies alone don’t fix them. The piece on what changes when a target adds a CDN covers the whole transition a site goes through when it puts a new edge in front. This one is about the invisible step in between: the automatic check that runs before any puzzle, and often instead of one.
The stance throughout is defensive. A challenge page is a site telling a client that it wants more evidence before serving content. The useful response is to understand why and adjust, and this piece doesn’t describe how to get past one.
What the interstitial actually is
Mechanically, a JavaScript challenge is an HTML page returned in place of the content you asked for. It usually carries very little visible content, a small script or a reference to one, and some parameters unique to the request. It’s often served with a status code that signals the content isn’t being delivered, such as 403 or 503, though some systems use 200 or other codes, so status alone isn’t a reliable way to spot one.
When a browser receives it, the script runs. It gathers information about the environment it’s running in, may perform some computation, and sends the results back to the edge, usually to a special endpoint on the same domain. The edge evaluates the results. If they pass, it issues a clearance token, typically as a cookie, and the page redirects or reloads to the original URL. That second request carries the token, the edge recognises it, and the real content is served.
The clearance token is what makes the process tolerable for people. It lasts for some period, from minutes to hours or longer depending on the site’s configuration, so a visitor doesn’t face the check on every page. During that period, requests carrying the token go straight through.
The token is usually bound to things about the client. Commonly that includes the address it was issued to or something derived from it, and properties of the client that were observed during the challenge. If those change, the token may stop being accepted, and the challenge appears again.
From the site’s point of view, this happens entirely at the edge. The origin servers never see the challenged request. That’s part of the point: the site gets a filter in front of its infrastructure without writing any code for it.
Why sites put one in front
JavaScript challenges exist because they’re a cheap filter that catches a lot of unwanted traffic with little effect on real visitors.
The first thing they filter out is any client that can’t or won’t run JavaScript. That covers most simple automated traffic: plain HTTP libraries, basic scripts, many vulnerability scanners and a good share of attack tools. Real browsers run JavaScript by default, so nearly all human visitors pass this part without doing anything.
The second thing they do is buy time and evidence. Running a script in the client lets the edge collect information that isn’t available from the request alone, and combine it with what it already sees at the network level. That makes the next decision better informed.
The third is cost shifting. Some challenges ask the client to do a small amount of computation. For one visitor on one page, it’s negligible. For a client making huge numbers of requests, it adds up, which makes mass traffic more expensive for the sender without costing the site anything.
Sites use challenges for different reasons. Some turn them on during an attack, when a flood of traffic threatens to overwhelm the origin. Some apply them permanently to sensitive paths like logins, search or checkout. Some apply them to traffic that scores as risky, and let everything else through. And some use them as a general setting across the whole site because it came with their CDN plan and seemed sensible.
For a scraper, the reason matters less than the fact. Something about the request or the client led the site’s edge to want more evidence, and the site has configured its edge to ask.
When a challenge appears and when it doesn’t
The same site can challenge one client and not another, or challenge a client on one page and not the next. Understanding the triggers explains a lot of confusing scraper behaviour.
Site-wide settings come first. Many edge products have a security level setting, from off to strict, and a mode for sites under attack that challenges nearly every visitor. A site in that mode will challenge you however well-behaved your client is.
Path rules come next. Sites often apply stricter rules to specific paths: login pages, account areas, search endpoints, APIs, checkout. A scraper may pass freely on product pages and hit a challenge the moment it touches search.
Scores drive most of the rest. The edge computes a risk score for each request from network reputation, the client’s fingerprint, the request’s shape and the session’s behaviour. The pieces on IP reputation feeds and on behavioural signals cover those inputs. A request above a threshold gets a challenge; one below goes through.
Rate triggers are common. A client that exceeds a rate on a path or a site may start receiving challenges instead of outright blocks, giving a real person a way through while stopping volume.
Geography sometimes plays a part. Some sites challenge traffic from countries outside their market.
And novelty matters. A client with no history, no cookies and no previous clearance is more likely to be challenged than one returning with a valid token. That’s one reason rotating exits and discarding cookies tends to increase challenge rates rather than reduce them.
First check: can the client run a script at all
The most basic thing a challenge tests is whether the client executes JavaScript. That’s the dividing line that filters out the largest volume of automated traffic.
A plain HTTP client, whether a library in Python, Go or Node, or a command-line tool, receives the challenge page as text. It doesn’t run the script. It never sends the results back, never gets a clearance token, and every subsequent request looks exactly like the first: a client with no token asking for a protected page. The edge serves the challenge again. From the scraper’s side this looks like the site returning the same odd page forever.
That’s the end of the road for plain HTTP clients on a challenged path, and it’s worth recognising quickly. Scrapers often waste time and bandwidth retrying the same URL through different exits, on the theory that the address is the problem. It isn’t. The client simply can’t satisfy the check, whatever address it uses.
A browser, including a headless one driven by automation, does run the script. So it passes this first check. What happens next depends on the other checks.
The fact of not running scripts is also recorded. A client that received a challenge and never attempted it has told the edge something about itself, and that can feed into how later requests from the same address or pattern are scored.
For a scraper, the practical takeaway is to detect challenge pages explicitly in the pipeline and stop retrying them blindly. A challenge on a plain HTTP client is an answer, not a transient error.
Second check: is the environment consistent
For clients that do run the script, the challenge looks at the environment the script finds itself in, and whether it’s consistent with what the client claimed to be.
At a high level, the script can inspect which browser features and interfaces exist, how they behave, and properties of the rendering and runtime environment. It compares those against what a real browser of the claimed type and version would show. A client whose headers say it’s one browser, whose TLS handshake looks like another, and whose script environment looks like a third doesn’t fit together.
Automation frameworks leave traces in the environment unless they’re specifically configured otherwise, and vendors know what many of those traces look like. The pieces on TLS fingerprinting and on what your scraper leaks besides its IP describe some of the network-level signals that feed into the same comparison.
The comparison is also made against the network. A client claiming to be a mobile browser on a phone, arriving from a datacenter address, with a desktop screen size, is internally inconsistent in several ways at once.
The details of what’s checked change often, are deliberately obscured in the challenge scripts, and differ between vendors. That’s by design. It’s also why this piece stays at the level of categories. The specifics are an arms race, and the side with visibility into millions of real browsers holds the advantage.
For a scraper using a headless browser, the implication is that running a real browser engine isn’t the same as being an ordinary visitor. Everything around the engine is part of what the challenge evaluates: the way it’s launched and driven, and the machine and network it runs on.
Third check: computation, timing and the token
Some challenges include a computational task: the script has to compute something that takes a measurable amount of work before the edge will issue a token. This is a proof-of-work idea, similar in spirit to systems used to deter spam. For a single page view, the cost is small enough that people don’t notice. For a client making many requests, the cost multiplies, and it becomes a real expense in compute and time.
Timing is observed too. How long the script took to run, whether it ran at a speed plausible for the claimed device, and how the page load unfolded around it are all available to the edge. The details are vendor-specific, but the principle is that the challenge measures more than whether a script ran.
Then there’s the token itself. When a client passes, the clearance it receives is usually bound to aspects of that client. Common bindings include the address or a range derived from it, and properties observed during the challenge. The token also expires.
That binding explains a pattern many scraper operators see. A browser passes the challenge through one exit, gets a token, and then the scraper rotates to a different exit for the next request. The token doesn’t match the new address, so the edge ignores it and challenges again. The scraper now has to pass a fresh challenge on every exit, which multiplies cost, and every repeated challenge adds to the evidence that this client is unusual.
Keeping one session on one exit for a sensible period, with its cookies, is simply how browsers behave. It’s also the only way a clearance token works as intended.
Newer attestation-based checks
Alongside script-based challenges, some sites and CDNs now support approaches where the client’s device or platform vouches for it, without the site running an elaborate check.
The general idea is that a trusted party, such as the device’s operating system or another issuer, confirms that a request comes from a genuine device and a real user session, and provides a token that proves this without revealing who the user is. Standards work in this area has produced private, unlinkable token schemes that some browsers and platforms support, and some edge providers accept these tokens as an alternative to showing a challenge.
For ordinary visitors on supported devices, that means fewer challenges. For automated clients, it means that the population of visitors who pass without any challenge at all may increasingly consist of those carrying a token their platform issued, and a client without one is, by comparison, more likely to be asked for evidence.
This is still evolving, adoption varies, and the details differ between implementations. The relevant point for scraping is directional. Sites are moving towards ways of recognising genuine visitors that an automated client has no honest way to take part in. That makes the honest routes described below more important over time, not less.
What challenges look like from inside a scraper
Because challenges come in several forms, it helps to know what they look like in a scraper’s own data, so they can be counted and handled instead of mistaken for something else.
Plain HTTP clients see an HTML response with little or no content, a script, and often a 403 or 503 status. The body is usually small and nearly identical across different URLs on the same site. Retrying gives the same page.
Headless browsers may see the challenge page briefly, then either the real content, a second and harder challenge, a visible CAPTCHA, or the same challenge again. In logs, that can show up as repeated navigations to the same URL, unusual redirects, or page loads that take much longer than normal.
Sessions that did pass may start failing later, when the clearance token expires or when something it was bound to changes, such as the exit.
Some sites return a soft version: a 200 with a page that looks like content but is actually a stripped template or a challenge embedded in a normal layout. The piece on soft blocks that return 200 describes how to catch those.
The practical step is to write a classifier. Recognise the challenge page by its structure: its size, its script references, the absence of the elements your parser expects, specific response headers or cookies the edge sets. Record each occurrence as a challenge, separately from errors and from successful fetches, with the URL, the path type, the exit and the session. That gives you a challenge rate per target and per section, which is the signal the rest of this piece is about.
Build the classifier from saved examples rather than guesses. When a challenge first appears, keep a copy of the response, headers included, and write a test that the classifier recognises it. Keep a copy of a normal page from the same site too, and test that the classifier doesn’t flag it. Challenge pages change when vendors update them, so when the classifier starts missing, the saved examples tell you what changed. The piece on recorded fixtures covers keeping that kind of test set.
Make the classifier conservative in one direction. It’s better to flag an unusual real page as a possible challenge and have a person look than to let challenge pages flow into the dataset as empty records. An empty record that looks like data is the most expensive failure a scraper can have, because nobody notices it until a decision has been made on it.
And once a response is classified as a challenge, stop there. Don’t retry it through another exit, don’t hand it to a browser to try again. Record it, count it, and let the rate decide what happens next.
One job, three different answers
A made-up example shows how the same scraper can meet challenges for three different reasons on three sites, and why each deserves a different response.
The job collects public listings from three sites through the same scraper, a plain HTTP client at a modest pace with one session per exit.
On the first site, listings load normally. But every request to the site’s search endpoint returns a small HTML page with a script and a 403. Category pages are unaffected. The site has put a path rule on search. The response is simple: stop using search and build the crawl from category pages and the sitemap instead. The challenge rate on that site drops to zero, and nothing about the proxies changed.
On the second site, everything works for the first few hundred requests of each run, then challenges start appearing and climb steadily. Slowing the scraper to half its rate and dropping concurrency to one delays the first challenge and lowers the rate, but doesn’t remove it. The site is reacting to volume. The response is to find the pace at which challenges stay near zero, accept that the job takes longer, and spread it across the day.
On the third site, the very first request of every session returns a challenge, on every page, at any pace, from every exit type tried. That site has decided it wants script-based evidence from every visitor. For a plain HTTP client, that’s a clear no. The honest options are an API if one exists, a data feed, a partnership or a direct request for access. Running a browser farm to pass the challenge would be the dishonest option, and an expensive one.
Three sites, three causes, three responses. In none of them was a different proxy the answer.
Why defeating it is the wrong goal
It’s common to treat a challenge page as an obstacle to engineer around. There are services that sell challenge solving, and there’s a steady stream of techniques for configuring automation to pass. This piece won’t point to any of them, and the reasons are practical as well as principled.
The first reason is what the page means. A site that serves a challenge has decided it wants more evidence before serving content to that client. Engineering a client to supply convincing evidence that it’s something it isn’t is deception, directed at a system whose only job is to tell visitors apart. Most sites’ terms of service prohibit it explicitly, and depending on the site, the data and the jurisdiction, it can carry legal exposure beyond a breach of terms.
The second reason is durability. Challenge scripts change frequently and without notice. A technique that passes today is being studied by the vendor’s team, and when it stops working, it stops all at once, often across every site that vendor protects. A scraping operation built on passing challenges is permanently one update away from failure.
The third reason is cost. Every challenge means running a browser, executing scripts, sometimes doing computation, and repeating it whenever a token expires or an exit changes. Compared with fetching a page from a site that’s happy to serve you, it’s much more expensive.
The fourth reason is collateral. Traffic that fights challenges feeds the vendor’s models and the reputation of every exit it uses. On a shared pool, other customers inherit that.
A challenge is information. The useful thing is to act on it.
Treating the challenge as a signal
When a challenge rate appears in a scraper’s data, the sensible response depends on what’s driving it, and the scraper’s own logs are usually enough to tell.
If challenges appear only on certain paths, such as search or login, the site has chosen to protect those paths. The right response is to leave them alone. Search results can often be replaced by category pages or a sitemap. Login-protected content is out of scope for a scraper that doesn’t have explicit permission.
If challenges rise with your request rate, the site is reacting to volume. Slow down, reduce concurrency, keep sessions consistent on one exit with their cookies, and see whether the rate falls. If it does, you’ve found a pace the site will accept.
If challenges appear from the first request at a gentle pace, on ordinary pages, the site has decided it wants evidence from every client like yours. Further tuning is the wrong response. Check the robots rules and terms, look for an API or a data feed, and consider asking the site directly.
If challenges appear suddenly on a target that was fine yesterday, the site or its CDN changed configuration. The piece on what changes when a target adds a CDN covers that transition. Treat it as a new target and reassess from the start.
In every case, alert on the challenge rate so a person looks at it quickly. A job that keeps running into challenges for a week spends bandwidth collecting nothing and adds to its exits’ records the whole time.
What a challenge costs both sides
It helps to be clear-eyed about cost, because it’s often what finally changes a team’s mind.
For the site, a challenge is cheap. The edge serves a small page and evaluates a small response. The origin servers never see the request. That’s why sites can afford to challenge broadly, and why they can leave challenges switched on indefinitely.
For a scraper, a challenge is expensive in every direction. A plain HTTP client gets nothing for its request, and every retry costs bandwidth on a metered exit. A browser-based scraper pays for the browser: memory, CPU, rendering, the scripts and resources the challenge page loads, and any computation it asks for. It pays again every time a token expires, and again for every exit if it rotates. Multiply that across a large job and the challenge becomes the dominant cost of the whole operation.
There’s also the cost that doesn’t show up on an invoice. Every challenged request adds to the evidence the vendor holds about the exit and the pattern. On a shared pool, that makes the pool worse for everyone using it, including you on your next job.
Compare that with the cost of the alternatives. An API key, a licensed dataset, or an hour spent writing to the site’s owner can look expensive up front. Set against a browser farm that fights challenges indefinitely and breaks whenever the vendor updates, they’re usually the cheaper route by a long way.
When the automation is legitimate
Not every automated client that meets a challenge is unwelcome. Some are yours, or are wanted, and there are proper ways to handle them.
If you run monitoring or testing against your own site behind a CDN, the CDN almost certainly lets you configure rules that skip challenges for your own tools. That’s typically done by matching a known address, a specific header with a secret value, or an authenticated route. It’s the right fix, because the site owner is the one making the exception.
If a partner or client needs your crawler to access their site, ask them to add a rule for it. Explain what the crawler does, how much traffic it sends, which paths it touches and how it identifies itself. Site owners can create exceptions easily when they want to, and a rule they created is stable in a way no workaround is.
If your crawler is a well-known, well-behaved service, look at whether the relevant CDNs or bot management vendors run a verified crawler programme. These schemes recognise automated clients that identify themselves verifiably and behave well. They require the opposite of disguise: a consistent, honest identity. The piece on identifying your crawler honestly with a contact header covers why that approach tends to work better than anonymity.
And if the site publishes an API, use it. APIs usually sit behind different rules from the HTML, designed for programs rather than people.
What I can’t tell you
I can’t tell you what a particular vendor’s challenge script checks, how it weights what it finds, or how long its tokens last on a given site. That’s proprietary, it changes often, and it’s deliberately hidden inside obfuscated code.
I can’t tell you how to pass a challenge, and I won’t. The categories above are there so that a challenge page stops being a mystery, not so it can be defeated.
I can’t promise that a slow, honest scraper will never be challenged. Some sites challenge everyone, and some challenge every automated client regardless of manners. That’s their call to make.
What I can say is that a JavaScript challenge is a site’s edge asking for evidence it didn’t get from the request alone. Plain HTTP clients can’t provide it at all. Browsers provide it in ways that reveal how they’re run. The honest responses are to leave protected paths alone, slow down where volume is the trigger, keep sessions consistent, look for an API or ask for access, and stop when the answer is clearly no.
If you want the related pieces on CAPTCHAs, CDNs and TLS fingerprinting, Proxy Scraping has the full set of guides from the operator side.
Get new guides and videos first — join the Telegram channel.