Imperva and Incapsula: what their bot checks look at

If you scrape for a living, you have hit the page that says “Request unsuccessful. Incapsula incident ID” followed by a string of digits. It is one of the most recognisable block pages on the web, and it usually shows up at the worst time: after a scraper has run clean for three weeks, or right after you swap proxy providers. The frustrating part is that the page tells you almost nothing about why you were blocked.

This article is about what the Imperva stack actually inspects, based on what is publicly documented and what I have seen in my own logs running scrapers out of Singapore. I will be clear about the line between the two. Imperva does not publish its scoring model, and nobody outside the company knows the weights. What I can describe reliably is the observable surface: the cookies, the challenge flow, the signals that visibly change outcomes, and the failure modes that waste the most time.

The stakes are practical. Imperva sits in front of a lot of banks, airlines, government portals and retail sites, so you will meet it on exactly the targets where data is worth the effort. Treating it as a generic “Cloudflare-like” wall leads to wrong fixes, like rotating IPs faster when the real problem is a cookie jar you are resetting on every request. I want to give you a mental model that makes the first ten minutes of debugging more productive. For the wider set of proxy troubleshooting write-ups, the blog index has the rest of the series, and the site homepage lists our proxy guides by topic.

background and prior art

Incapsula was a cloud WAF and CDN company founded in 2009 and acquired by Imperva in 2014. The Incapsula brand was folded into the Imperva name over the following years, and what customers now buy is sold as Imperva Cloud WAF, with Advanced Bot Protection as a separate product. In 2023 Thales announced it would acquire Imperva, and the deal closed in December of that year. None of that changed the technical surface much, which is why you still see incap_ses_ cookies and “Incapsula incident ID” strings on pages in 2026. The old branding lives on in cookie names and error text.

Two layers matter for a scraper. The first is the Cloud WAF layer, which does what a WAF does: IP reputation, rate limiting, request header sanity, known bad signatures, and geo rules the site owner configures. The second is the bot protection layer, which adds client classification using JavaScript challenges, behavioural telemetry and fingerprinting. Many sites only buy the first. Some buy both. A lot of confusion in forums comes from people describing one layer while the other person is fighting the second. If your plain HTTP client gets through on one Imperva site and dies on the next, that is usually why, and not because you changed something.

The prior art worth knowing is the general design of cookie-based session tracking in RFC 6265, which defines how Set-Cookie and Cookie headers work and why a client that mishandles them gets treated as a new visitor each time. Imperva’s whole session model rests on that boring mechanism. If you want to see how Imperva describes its own products and documentation, the vendor docs hub is at docs.imperva.com. I recommend reading what the site owner can configure, because it tells you what knobs exist on the other side of the wall.

the core mechanism

I find it easiest to think of the checks as four stacked questions, asked in roughly this order: who are you on the network, how do you speak HTTP and TLS, do you carry a believable session, and can you run and behave like a browser. A failure at an earlier level often never reaches the later ones, which matters for debugging.

the network layer

The first question is the IP. Imperva keeps reputation data on source addresses and ranges, and a site owner can set rules by country, ASN type, or known proxy and hosting ranges. Datacenter ASNs get the harshest treatment on sites with any bot policy at all. Residential and mobile ranges get more benefit of the doubt, but that benefit is spent quickly if the address shows abnormal request volume against a single host.

What I have seen consistently:

• a fresh datacenter IP often fails on the very first request on strict sites, before any cookie is set
• a residential IP usually gets a first response, then degrades over tens of requests if the behaviour looks automated
• mobile carrier IPs behind CGNAT are the most tolerant, partly because blocking one address risks blocking thousands of real phones

This is the layer where proxy choice is decisive, and it is the only layer where money alone solves the problem. Everything below it is engineering.

the HTTP and TLS layer

The second question is whether your client looks like the browser it claims to be. Two things get inspected here. First, the HTTP request itself: header presence, header order, casing in HTTP/1.1, pseudo-header order in HTTP/2, and whether the User-Agent matches the rest. A Chrome user agent with a header order that no Chrome release produces is a cheap, high-confidence signal. Semantics for the headers themselves are in RFC 9110, but browsers add conventions on top that the RFC does not require, and those conventions are what get fingerprinted.

Second, the TLS handshake. The ClientHello carries cipher suites, extensions, supported groups and ALPN values in an order that is characteristic of the TLS library. Python’s requests through OpenSSL, Go’s net/http, and Chrome’s BoringSSL all produce different ClientHellos, and the handshake structure is defined in RFC 8446 for TLS 1.3. Fingerprint schemes like JA3 and its successors hash these fields. I cannot tell you exactly which fingerprint scheme Imperva uses internally, and I will not pretend to. What I can tell you from testing is that swapping a stock Python client for one that mimics a real Chrome ClientHello changes the outcome on some Imperva sites with no other change at all. That is a strong hint that the handshake is being looked at, directly or through the CDN edge.

the session layer

This is where Imperva is most distinctive, and where most scraper bugs live. The cookies you will see:

• visid_incap_<siteid>: a long-lived visitor identifier, set on first contact
• incap_ses_<n>_<siteid>: a session cookie, which rotates and which the edge uses to bind your requests to one visit
• nlbi_<siteid>: a load balancer related cookie that appears on many protected sites
• reese84: the token issued by the bot protection layer after a successful client-side challenge, seen only on sites with Advanced Bot Protection
• ___utmvc: an older JavaScript challenge cookie that still turns up on legacy configurations

The point to absorb is that these cookies are the identity. If you throw away the jar between requests, every request is a first visit with no history, and first visits from automation are exactly what gets scrutinised. If you keep the jar but change the IP mid-session, you present an established identity from a new network location, which is also suspicious. The correct pairing is one jar, one IP, one TLS profile, for the lifetime of the session.

The visid_incap_ cookie is worth singling out because it persists. If a visitor ID has been flagged, reusing it carries that flag forward. Starting a new session with a clean ID after a block is often the right call, but only together with a new IP. A new ID from the same blocked address just looks like a bot clearing cookies.

the client challenge layer

On sites running Advanced Bot Protection, the first HTML response is sometimes a stub that loads an obfuscated JavaScript file. That script collects browser signals, runs checks, and posts a payload back to the origin. If the payload passes, the server sets reese84 and subsequent requests go through. The script is obfuscated, changes over time, and is not something to reverse for sport. Doing it well means maintaining it against regular updates, and in my experience that is a worse deal than running a real browser.

What the script looks at is the usual browser integrity surface: whether navigator properties are consistent with each other, whether canvas and WebGL output look like real hardware, timing and event behaviour, and the presence of automation markers. A headless browser with default settings fails the obvious parts. A patched, headful browser with consistent properties passes more often. I do not have a precise list of what is checked, and anyone who claims one is working from a snapshot that is probably out of date.

Behaviour matters on top of that. Mouse movement, scroll patterns and the timing between page load and first action feed into classification on some deployments. Zero interaction followed by a burst of deep-link requests is a pattern I would expect any bot system to flag.

how the layers combine

The result is a score or a decision at the edge, and the response you see depends on the site owner’s policy: pass, serve a challenge, silently degrade (empty results, stale data, decoy pages), or hard block with the incident page. Silent degradation is the one that hurts, because your pipeline stays green while the data is wrong. I will come back to that in the production notes.

worked examples

The numbers below come from my own test runs and are illustrative, not benchmarks. Your targets, volumes and Imperva configurations will differ, so use the shape of the result and not the percentages. I am deliberately not naming the target sites. Naming them would add nothing for you and could point at people who did nothing wrong.

example 1: a retail site on Cloud WAF only

The target was a mid-sized retail site with category and product pages, protected by Imperva’s WAF layer without a visible reese84 flow. I ran 2,000 product page requests from a rotating pool of datacenter IPs using a stock Python HTTP client, one new cookie jar per request.

The outcome was bad: roughly 70 percent of requests got the incident page, and the rest returned usable HTML. I changed one variable at a time. Keeping a persistent jar per IP for 50 requests moved the success rate up noticeably. Switching the same client to a Chrome-like TLS profile moved it up again. Moving to a small pool of residential IPs with sticky sessions of around ten minutes got me to the point where blocks were rare enough to handle with a simple retry. The lesson was that session persistence and TLS profile did most of the work before I paid for better IPs, so I tested those first on every Imperva target afterward.

A minimal version of the session handling looks like this:

# one jar, one proxy, one TLS profile per worker
import httpx

def make_session(proxy_url: str) -> httpx.Client:
    return httpx.Client(
        proxy=proxy_url,
        http2=True,
        headers={
            "accept-language": "en-SG,en;q=0.9",
            "user-agent": CHROME_UA,  # must match your TLS profile
        },
        follow_redirects=True,
        timeout=20,
    )

# reuse the same client for the whole visit, so Set-Cookie
# values for visid_incap_ and incap_ses_ are replayed automatically

Note that httpx on its own still produces a Python ClientHello. For sites that fingerprint TLS you need a client that can impersonate a browser handshake, such as curl_cffi, which wraps curl-impersonate. The header and jar logic above stays the same either way.

example 2: an airline-style booking flow with reese84

This target ran Advanced Bot Protection. A plain HTTP client received the stub page and never obtained reese84, so every API call returned a block. I switched to a headful Chromium driven by Playwright with a persistent profile, residential sticky IPs, and a warm-up of two or three ordinary page loads before touching the search endpoint.

Across a test of 300 searches, the persistent profile with warm-up completed the large majority. The same setup in default headless mode failed far more often, and a fresh profile each time failed more often again. Two details mattered. First, I let the challenge script finish before navigating, which meant waiting for the reese84 cookie to exist and not just waiting for load. Second, I kept search volume per session low, around 15 to 20 searches, then rotated the whole identity: profile, IP and cookies together.

A check I now put in every browser-based worker:

async def wait_for_token(context, timeout_ms=15000):
    deadline = time.monotonic() + timeout_ms / 1000
    while time.monotonic() < deadline:
        cookies = await context.cookies()
        if any(c["name"] == "reese84" for c in cookies):
            return True
        await asyncio.sleep(0.25)
    return False

If the function returns false, I treat the session as burned and rotate, instead of pushing requests through and learning about it from the block page.

example 3: a government portal with strict geo rules

A public records portal behind Imperva allowed traffic from Singapore and a handful of other countries. Datacenter ranges from the right country still failed. The same requests from a Singapore mobile carrier IP passed with an ordinary client and a persistent jar. The cost difference was real: the mobile route cost me several times more per gigabyte than datacenter traffic, but the page volume was small, a few hundred lookups a day, so the total spend stayed modest. Matching the proxy type to the actual volume mattered more than finding the cheapest per-GB price. I would not have found that without testing each network type against the same request, which is a habit worth building. Only collect data you are allowed to collect, and read the portal’s terms first. This is not legal advice.

edge cases and failure modes

These are the pitfalls that cost me the most hours, with what I do about each.

silent degradation looks like success

Some configurations do not block. They serve a 200 response with a thinner page, missing prices, an empty results list, or cached content. Your status-code monitoring sees nothing. The counter-strategy is content validation: assert that the fields you need are present and plausible on every response, and track the ratio of valid to total pages per proxy and per session. A drop in that ratio is your early warning, usually well before status codes change. I also keep a small canary set of URLs with known expected values and run them every hour.

cookie jar and IP drift

The classic self-inflicted wound is a proxy pool that rotates exit IPs while your client keeps the cookie jar. The edge sees incap_ses_ cookies from one address suddenly arriving from another country. Fix it by making sticky sessions the default and binding the jar to the session ID of your proxy provider. If the provider’s sticky session expires, discard the jar with it. The reverse bug exists too: a jar lost on every request because a worker pool creates a new client per task. Check by logging Set-Cookie counts. If every response sets a new visid_incap_ value, you are a new visitor each time.

headless and automation tells

Default headless Chromium leaks in several ways, and bot scripts look for them. Running headful, using a real profile directory, and keeping navigator values consistent with the user agent and platform fixes the cheap tells. Do not mix an approach where you patch one property and ignore the others. A user agent claiming Windows with a Linux platform string and software rendering in WebGL is worse than an honest default, because the inconsistency is itself the signal. When in doubt, run on a real desktop OS image and avoid mismatches between declared and observed values.

challenge timing and navigation races

On reese84 sites, navigating away before the token is set leaves you with a half-initialised identity. Requests then fail in confusing ways, including intermittent success. Always gate on the cookie, as in the earlier snippet, and not on a fixed sleep. Also watch for the challenge re-running mid-session: the token has a lifetime, and long sessions need to refresh it. If you see success followed by a sudden run of blocks at about the same elapsed time across workers, suspect token expiry before you suspect IP burn.

aggressive concurrency on one identity

Imperva rate limits per visitor and per IP, and the thresholds are set by the site owner. One identity firing 10 concurrent requests behaves unlike any human. Spread load across identities, cap per-identity concurrency at one or two, and add jitter. Fixed intervals are easy to spot, so randomise delays. Respect the site’s published crawl rules too, which are standardised in RFC 9309 for robots.txt. Politeness is not only ethical, it also keeps your identities alive longer.

what we learned in production

The biggest lesson was to make the identity a first-class object in the code. For each Imperva target we now define a unit made of a proxy session, a TLS profile, a cookie jar or browser profile, and a request budget. The unit is created together, used together, and retired together. Before I did this, bugs showed up as vague “the success rate dropped” tickets, and I spent days looking at IPs when the actual problem was a pool bug that reset the jar. With the identity as an object, I can log per-identity outcomes and see which part is failing.

The second lesson is to test cheapest first. For a new Imperva target I go in this order: a persistent jar with a stock client, then a browser-like TLS profile, then a headful browser with a warm-up, then a better proxy class. Each step costs more than the last, and a surprising number of targets fall at step one or two. I keep a record of which step each domain needed so the next person on the team does not repeat the ladder. If you are weighing proxy types for a given step, the blog index has our proxy reviews and comparisons, and if you are building out a multi-identity setup for other kinds of account work, the guides on multiaccountops.com cover session isolation from that side. For browser profile tooling that backs the headful approach, antidetectreview.org is where we compare the options.

One more honest note. Bot protection changes constantly, and anything I write here has a shelf life. The cookie names have been stable for years, but the challenge script and scoring change without notice. Build monitoring first, tuning second. If a target matters to your business, assume you will revisit it every few months, and treat a sudden drop as a normal event, not as a personal affront. You can also browse the full blog index for related troubleshooting notes on other vendors, or head back to the homepage to start from our proxy overview.

references and further reading

• RFC 6265: HTTP State Management Mechanism, the cookie specification that Imperva’s session identity depends on
• RFC 9110: HTTP Semantics, for header and request semantics that fingerprinting builds upon
• RFC 8446: TLS 1.3, for the handshake structure behind TLS fingerprinting
• RFC 9309: Robots Exclusion Protocol, the standard for crawl rules
• Imperva documentation, the vendor’s own docs for Cloud WAF and bot protection configuration

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-10-20.