Behavioural signals that anti bot vendors score
The scraper that changed address and changed nothing
A team runs a scraper through a rotating pool, with a fresh exit on every request. It works for a while, then the challenge pages start. The team’s first instinct is that the addresses are burned, so they switch to a pool with a better reputation. The challenges continue at much the same rate. They switch again. Same result.
What they’re running into is a score that was never mainly about addresses. Modern bot management products look at what a client does over time: how fast it asks for pages, in what order, what else it loads alongside them, and, when scripts run in a browser, how the page was interacted with. Change the address and all of that stays exactly the same. In some cases changing it makes things look worse, because the same pattern turning up from many addresses is itself a pattern.
This piece describes, at a high level, the categories of behaviour those systems weigh. It’s meant to explain why a scraper gets the treatment it gets, and why the durable answers are a slower, compliant scraper or an official channel for the data. It deliberately doesn’t describe how to imitate human behaviour. That route is an arms race against well-funded vendors, it runs against most sites’ terms, and it’s the kind of deception this site won’t help with.
The companion pieces here cover what a CAPTCHA actually measures, what a scraper leaks besides its IP, and what header order gives away. Those are mostly about what a client is. This one is about what a client does.
What behavioural means in this context
Bot detection signals fall roughly into two groups.
Static signals describe the client at a single moment. The address and the network it belongs to. The TLS handshake. The HTTP version and header set. In a browser, properties such as screen size, installed fonts, graphics capabilities and whether automation flags are present. These are the fingerprints covered elsewhere on this site. They can be read from one request or one page load.
Behavioural signals describe what the client does across time. How many requests it makes and how they’re spaced. Which pages it visits and in what order. Whether it loads the page’s images, scripts and styles, or only the HTML. Whether cookies set on one response come back on the next. In a browser, whether and how the page is scrolled, clicked, typed into or touched. None of these can be read from a single request. They emerge from a sequence.
Vendors combine both groups, and the combination is what makes them effective. A static signal can be ambiguous: a datacenter address might be a corporate VPN, a missing header might be an unusual but legitimate client. A behavioural pattern adds context. A datacenter address that loads one page, lingers, loads its images and follows a link a person would follow looks different from one that requests two hundred product pages in alphabetical order without loading anything else.
Behaviour is also harder to change than a static signal, because it’s a consequence of what the program is built to do. A scraper exists to fetch a lot of pages efficiently. That goal shapes its behaviour in ways that are visible from the outside, whatever it claims to be.
Where the signals are collected
Behavioural data comes from two places, and knowing which is which explains a lot about what vendors can and can’t see.
The first is the server side: the requests themselves as they arrive at the site or at the CDN in front of it. Every request carries a timestamp, a path, headers and usually cookies. From the sequence of requests associated with a client, the vendor can reconstruct cadence, navigation paths, which resources were fetched and whether session state was kept. This works for every kind of client, including plain HTTP libraries that never run any scripts. There’s nothing to opt out of here; it’s simply the traffic.
The second is the client side: scripts delivered with the page that run in the visitor’s browser and report back. These can collect interaction events such as pointer movement, scrolling, key presses, focus changes and touch events, as well as timing information about how the page loaded and how long the visitor stayed. The data is typically sent to the vendor in the background, sometimes encrypted or obfuscated, and combined with the server-side view.
A plain HTTP client that doesn’t execute scripts produces no client-side data at all. For a site that expects it, that absence is itself a signal: the page was fetched, but nothing ran. A browser-based scraper does run the scripts, so it produces client-side data, and that data reflects how the automation behaves.
Both views feed a model, usually run by the vendor across many customer sites. That cross-site view matters. A vendor that sees similar behaviour on many sites it protects can learn a pattern from one and apply it to all.
Request cadence
The simplest behavioural signal is how requests are spaced in time.
People browse in bursts and pauses. They load a page, read it, click something, go back, open two tabs, leave for ten minutes, come back. The gaps between their requests vary a lot, and their total volume over an hour is modest. Even heavy users of a site produce nothing like the volume of an automated job.
Automated clients tend to show one of a few shapes. Perfectly regular spacing, one request every fixed interval, is the most obvious. High sustained volume from one client is another. So is concurrency: many requests in flight at once from one session or one client identity, far more than a browser tab would generate. And so are unusual hours relative to the site’s audience, such as sustained traffic at times when the site’s real visitors are asleep.
Vendors don’t need any one of these to be conclusive. They look at the distribution of gaps, the volume per time window, the concurrency, and how those compare with the site’s normal traffic. A client whose cadence sits far outside what the site usually sees gets a higher score, and that score accumulates over the session.
For a compliant scraper, cadence is also the signal it has the most control over, and the one where good behaviour and low scores line up naturally. A scraper that makes few requests, spaced generously, with low concurrency, is doing what the site’s robots rules and rate limits generally ask for. It still won’t look like a person, and it shouldn’t try to. But it stops looking like a load problem, which is often what a site cares about most.
Navigation paths and timing
The second category is the route a client takes through the site, and how long it spends at each step.
People arrive through a small number of entry points: the homepage, a search engine result, a link from elsewhere, a bookmarked page. From there they follow links the page presents to them, in an order that reflects interest rather than structure. They go back, they get distracted, they abandon things halfway.
Automated clients often show a different shape. Many enter directly on deep URLs without ever loading a page that links to them. Many walk the site in a systematic order: every product in a category, sequential identifiers, alphabetical slugs, every page of a listing from first to last. Many request a page and move to the next one faster than anyone could have read it. Many never revisit anything.
The Referer header adds to this. A request that claims to come from a page the client never loaded, or that carries no referer where a browser would normally send one, doesn’t fit a normal navigation path.
Vendors use these patterns in aggregate. A single deep entry is ordinary; people follow links from search engines all the time. A session that consists entirely of deep entries in sequential order, at a pace nobody reads at, stands out.
This is also why sitemap-led crawling, which is right for other reasons, doesn’t look like a person browsing. It shouldn’t be made to. A crawler reading the sitemap is doing what sitemaps are published for. The honest version of that behaviour is a scraper that identifies itself and paces itself, not one that pretends to wander.
What gets loaded with the page
A browser loading a web page doesn’t make one request. It makes dozens: the HTML, then stylesheets, scripts, fonts, images, sometimes API calls made by the page’s own scripts, analytics beacons and more. The order and timing of those requests follows from how the page is built and how browsers work.
A plain HTTP client requests the HTML and stops. From the server’s side, the pattern is unmistakable: page after page with none of the resources a browser would have fetched. For a site that serves ordinary browser traffic, that’s a clear sign the client isn’t a browser.
Headless browsers do load the resources, but automation setups often block some of them, usually images and fonts, to save bandwidth. That’s sensible for cost. It also produces a resource pattern that differs from ordinary browsers in a consistent way.
Caching adds another layer. A returning visitor’s browser sends conditional requests for resources it already has, and skips some entirely. A client that fetches every resource fresh every time, or never fetches any, looks different from one with a normal cache.
None of this should push a scraper towards loading resources it doesn’t need. Fetching images purely to look like a browser costs bandwidth on both sides, and it’s the kind of mimicry that doesn’t hold up. The useful reading is the opposite one: a plain HTTP client is visibly a program, and on sites where that’s a problem, the site is saying something about whether it wants programs there. Where the data is available through an API or a feed, that’s the channel built for programs.
Interaction events
The category people usually think of first is interaction: how the page is used once it’s loaded. This only applies to clients that run the page’s scripts, which in practice means browsers, including automated ones.
Scripts on the page can observe pointer movement, clicks, scrolling, key presses, touches, focus and blur on the window, visibility changes when a tab is hidden, and the timing between all of them. They can observe whether these events happened at all during a page view, and some statistical properties of how they happened.
At a high level, vendors are interested in a few questions. Did anything happen? A page that loads in a real browser and receives no interaction events before the next navigation is unusual for many kinds of pages. Is what happened consistent with a physical input device and a person, or with events generated programmatically? Is it consistent with the device the client claims to be, since a phone produces touch events and a desktop produces pointer events? And is it consistent across the session?
The details of how vendors analyse these events are proprietary, change frequently and are deliberately hard to observe from outside. That’s by design, and it’s also why attempting to generate fake interaction is a poor strategy. Anyone building that is working against teams whose full-time job is detecting exactly that, with data from millions of real sessions to compare against, and the effort buys a temporary result on terms the site never agreed to.
For a scraper, the practical point is simpler. Interaction signals are one more reason why a browser-based scraper is still clearly an automated client to a site that’s looking. Running a real browser doesn’t make a program into a person.
Session continuity and state
The fourth category is whether a client behaves like one continuous visitor.
Sites set cookies, and browsers return them. A session cookie set on the first response comes back on every later request, until it expires or the visitor clears it. Some sites set tokens through scripts, or require a value from one response to be sent with the next. A normal visitor keeps all of this state for the duration of their visit.
Automated clients break continuity in several ways. Some discard cookies between requests, so every request looks like a new visitor. Some keep cookies but change address on every request, so one session hops across dozens of networks within minutes. Some keep one address but rotate user agents, so one session claims to be several different browsers. Each of these is a mismatch between signals that should agree.
Vendors track these inconsistencies closely because they’re cheap to detect and rarely innocent. A session identifier seen from many addresses in a short time is a strong signal. So is a client that never accepts the site’s cookies at all, on a site where every browser would.
This is one place where the rotation habit backfires directly. A scraper that rotates exits on every request, while keeping cookies, produces exactly the “one session, many networks” pattern. A scraper that rotates exits and drops cookies produces a flood of first-time visitors with no history, which is its own pattern. The piece on rotating proxies without losing your session covers why sticky sessions matter for correctness. They matter for how the site reads you as well.
How the scores combine
No single behavioural signal decides anything on its own. Vendors combine them, along with static signals and reputation data, into a score for the request or the session, and the site’s configuration maps scores to actions.
The combination is typically done by models trained on large amounts of traffic, including traffic the vendor has labelled from known bots and known humans. The exact features and weights are proprietary and change often. What’s broadly true is that the models look at sequences rather than single requests, that they combine many weak signals into a stronger one, and that they can learn patterns across all the sites a vendor protects.
The actions vary. A low score passes through untouched. A middling score might trigger an invisible check, a JavaScript challenge or a CAPTCHA. A high score might get a block, a rate limit, or a soft block that returns a 200 with different content, which the soft blocks piece on this site describes. Some sites log scores without acting on them, using the data for later decisions. Some serve deliberately degraded data to high-scoring clients.
Scores attach to whatever identity the vendor can track: a session cookie, a device identifier derived from browser properties, an account, and yes, an address and its range. The more identities line up, the more confidently the score applies. That’s why changing one of them, such as the address, while keeping the behaviour the same, doesn’t reset anything. The behaviour is re-scored on the new address within a few requests, and the pattern links the new address to the old one.
Why changing the address doesn’t help
Put the pieces together and the reason IP rotation fails against behavioural scoring becomes clear.
The score is driven by what the client does. Cadence, navigation, resource loading, interaction, session continuity: all of these come from the program’s logic, and all of them survive an address change untouched. A new exit sends the same sequence of requests with the same timing and the same gaps in its behaviour. The model sees the same pattern and reaches the same conclusion.
Rotation can also make things worse in two ways. First, as described above, rotating mid-session breaks continuity, and that inconsistency is itself scored. Second, the same behavioural pattern appearing from many addresses at once looks like a distributed operation, which is precisely the kind of traffic that bot management exists to stop. The vendor doesn’t need to know every address. It needs to recognise the pattern, and then every new address that shows it inherits the score quickly.
There’s a cost dimension too. Every request sent through a premium exit only to be challenged is paid bandwidth that bought nothing. Teams that respond to rising challenge rates by buying a bigger or more expensive pool often find the bill going up while the success rate stays flat, because the pool was never the problem.
The piece on what a CAPTCHA actually measures makes a parallel point for challenge pages specifically. Address quality is one input among many, and for behavioural scoring it’s one of the least important.
Why imitation is a dead end
The obvious next thought, for some people, is to make the scraper behave more like a person: randomise delays, wander through pages, generate pointer movements. This piece won’t describe how, and the reasons deserve a direct answer.
First, it’s an arms race you’re set up to lose. Vendors see traffic from huge numbers of real users across many sites, which gives them a detailed picture of real behaviour that no individual scraper operator has. They update their models continuously. A technique that seems to work today is being studied by people whose job is to make it stop working, and when it stops, it stops without warning.
It’s deceptive. Imitating a person to get past a system built to tell people from programs is a deliberate attempt to be treated as something you aren’t. Most sites’ terms of service prohibit it, and depending on the site, the data and the jurisdiction, it can carry legal risk as well as the obvious contractual one.
It’s expensive. Simulated browsing means real browsers, real rendering and real bandwidth for pages and resources you don’t need, multiplied across every page of the job.
And it misses what the site is saying. A site that pays a vendor to score behaviour has made a choice about automated access. The honest options are to respect that choice at a pace and scope it will tolerate, or to find a route to the data the site actually offers.
What a compliant scraper looks like to these systems
A slow, well-behaved scraper won’t look like a person to a behavioural model, and it shouldn’t try to. What it can do is look like the kind of automated client many sites are content to allow.
That means low volume and generous spacing, well inside whatever the site’s robots rules and rate limits suggest. Low concurrency, often a single connection. A consistent identity: one session kept for a sensible period, cookies accepted and returned, one address per session, one honest user agent. The piece on identifying your crawler with a contact header makes the case for a descriptive user agent with a way to reach you.
It means fetching only what the job needs, from URLs the site publishes for crawling, and following robots on every request. It means backing off hard when challenged or rate limited, and treating a rising challenge rate as a signal to slow down or stop, not as a problem to route around.
Some sites will still challenge or block that scraper. That’s their answer, and it’s a legitimate one. A behavioural model that flags a slow, honest crawler is doing its job: the site has decided it doesn’t want automated clients at all, or not in that section. At that point, the right move is to find an official channel or ask permission.
Reading your own challenge rate as feedback
Behavioural scoring happens out of sight, but its results show up in your own logs, and those are worth reading as feedback rather than noise.
Track the challenge rate per target, per section of the site, and per session. A challenge is any response that isn’t the content you asked for: an interstitial, a CAPTCHA, a block page, a soft block with missing data. Count them separately from errors, because they mean something different. An error says something broke. A challenge says the site made a decision about you.
Then look at how the rate responds to your own changes. If slowing the scraper down, reducing concurrency and keeping sessions consistent brings the challenge rate down, the site was reacting to load and pattern, and you’ve found a pace it will tolerate. If the rate stays high even at a gentle pace, the site is telling you it doesn’t want automated clients in that section, and further tuning is the wrong response.
Watch for the rate rising over time with no change on your side. That usually means the site or its vendor changed something: a new model, a stricter threshold, a new section under protection. It’s a prompt to re-read the site’s terms and robots rules, and to check whether an official channel has appeared.
Alert on it. A challenge rate that doubles overnight deserves a person’s attention the next morning, before a job has spent a week of bandwidth collecting block pages. The piece on alerting on block rate instead of errors covers how to set that up.
And resist the urge to treat a rising challenge rate as a proxy problem. It almost never is.
Two scrapers against one scored site
A made-up comparison helps show how the categories add up.
The first scraper runs twenty concurrent workers through a rotating pool, a fresh exit per request. It drops cookies between requests to “stay clean”. It enters on deep product URLs in the order they appear in the sitemap, requests only HTML, and moves to the next URL as soon as the previous one returns. It sends a browser user agent it picked from a list.
To a behavioural model, almost every category lights up. Cadence: high volume, high concurrency, regular spacing. Navigation: sequential deep entries, no referers, no revisits. Resources: HTML only, from a client claiming to be a browser. Continuity: every request a first-time visitor, from a different network. The claimed browser and the observed behaviour disagree. The score climbs within a few dozen requests, and rotating exits spreads the same pattern across the pool.
The second scraper runs one worker, keeps one session for a sensible period on one exit, accepts cookies, waits several seconds between requests, honours robots and any crawl-delay, and identifies itself with a descriptive user agent and a contact address. It still enters on deep URLs and still fetches only HTML, because that’s what the job needs.
The second scraper doesn’t look human either. What it looks like is a small, honest program at a modest pace, and many sites’ configurations let that through or apply only light checks to it. Some won’t, and for those the answer is an official channel. But the difference between the two scrapers has nothing to do with proxies. It’s entirely in how the program was built to behave.
The durable routes to the data
When behavioural scoring stands between a scraper and the data, the lasting solutions are the ones that don’t involve beating it.
Official APIs are the most direct. Many sites that defend their HTML heavily publish APIs, sometimes free, sometimes paid, with documented rate limits and terms. The data is usually cleaner, the load is lower, and the arrangement is explicit.
Data feeds and exports are next. Retailers publish product feeds for comparison engines and affiliates. Public bodies publish bulk downloads. Many data providers license datasets that would take months to scrape. The piece on when buying the data beats building the scraper covers that trade-off.
Permission is often underused. A short, clear message to a site explaining what you want, why, how much traffic it would involve and how you’d identify yourself gets a surprising number of positive answers, especially for research or non-competitive uses. Some sites will allowlist a crawler that asks.
Verified crawler programmes exist too. Some CDNs and bot management vendors run schemes through which well-known, well-behaved automated clients can be recognised and treated accordingly. They typically require verifiable identity and good conduct, which is exactly the opposite of evasion.
Each of these trades some upfront effort for a result that doesn’t break every time a vendor updates its model.
What I can’t tell you
I can’t tell you how any specific vendor weighs these signals, which inputs its models use, or what thresholds a given site applies. That information is proprietary, it changes constantly, and anything specific I wrote would be out of date or wrong for some site you care about.
I can’t give you a scraper configuration that avoids behavioural scoring, and I wouldn’t if I could. The categories above are described so you can understand why a scraper is treated the way it is, not so you can build around them.
I can’t promise that a slow, compliant scraper will be allowed. Some sites will decide they don’t want it, and that decision is theirs to make.
What I can say is that the reason IP changes don’t fix challenge pages is now fairly clear. The score is about behaviour, and behaviour comes from the program. The sustainable paths are a scraper whose behaviour a site can live with, or a channel the site built for programs in the first place.
If you want the related pieces on CAPTCHAs, fingerprints and what your scraper leaks besides its IP, Proxy Scraping has the full set of guides from the operator side.
Get new guides and videos first — join the Telegram channel.