← all guides

Recorded fixtures for testing scrapers offline

Testing against the live site is the expensive habit

Someone changes one line in a scraper’s parser, maybe a selector for the price field. To check it works, they run the scraper against the real site. It fetches a few hundred pages through the proxy, the output looks right, and the change ships. The next person does the same for their change, and so does the one after.

Each of those runs costs proxy bandwidth, adds load to someone else’s server, and produces a result that depends on what the site happened to serve that afternoon. If the page changed between two runs, you can’t tell whether your code broke or the site did. If the proxy exit got a soft block, the test “fails” for a reason unrelated to the change. And none of it can run in CI without handing the CI system proxy credentials and letting it hit the target on every commit.

Recorded fixtures fix that. You fetch a representative set of real responses once, politely, save them exactly as they arrived, and run your tests against the saved copies from then on. Parser changes get verified in seconds, offline, with the same input every time.

This is a different problem from caching responses so a production rerun doesn’t refetch the site. That piece on this site is about the live job. Fixtures are about the test suite: a deliberately small, hand-picked set of inputs, kept for years, that lets you change code with confidence. They also connect to the piece on parsers that break weeks after a redesign, because a good fixture set is how you notice what changed and prove the fix.

What follows covers what to record, how to store it, how to replay it at the right level, how to keep secrets and personal data out of it, and how to tell when fixtures have gone stale.

What a fixture actually is

A fixture, in this sense, is a saved HTTP exchange that stands in for the real one during a test. At minimum it holds the response body. A useful one holds more.

The request URL and method, so you know what was asked for. The response status code. The response headers, because content type, encoding, redirects and caching headers all affect how your code behaves. The raw body bytes, exactly as received, before any decoding your code does. And a small metadata record: when it was captured, from which kind of exit, with which client, and why it was chosen.

Saving the raw bytes matters. If you save the body after your client has decoded it to text, you lose information about encoding, and encoding bugs are exactly the kind of thing fixtures should catch. A page served in a legacy encoding with a misleading header is a real case; a decoded copy hides it.

The metadata matters for the future. A year from now, someone looking at a fixture called product_page.html needs to know it was captured before the site’s redesign, from a residential exit in a particular country, as the example of a product with a sale price. Without that, nobody knows whether it still represents reality or what it was meant to test.

There are two broad kinds of fixture. Page fixtures are single responses fed straight into a parser: here’s the HTML, what records come out? Session fixtures are sequences of exchanges replayed at the HTTP level: here’s what the server returned for page one, page two and the product pages they link to, so the crawler’s navigation logic can run end to end without a network. Most scrapers need both.

Recording once, politely

The recording run is the only time your test suite touches the real target, so do it carefully.

Fetch at a normal, polite pace, through the same kind of exit and the same client configuration your production scraper uses. The point is to capture what production actually sees. A fixture recorded from your laptop’s home connection may differ from what a datacenter exit or a proxy in another country gets, and the piece on sites that serve different HTML to datacentre IPs explains why that matters.

Record through a layer that sees the raw exchange. There are several ways to do this. In Python, libraries such as vcrpy and the pytest-recording plugin intercept HTTP calls made by common clients and save them as “cassettes” on first run, replaying them afterwards. In Node, Polly.js and nock can record and replay. For browser automation, Playwright can record network traffic to a HAR file and later serve requests from it. A recording proxy such as mitmproxy can capture everything passing through it, whichever client made the request. Choose whichever sits closest to how your scraper makes requests.

Keep the recording run small and targeted. You want a chosen sample, not a crawl. A few dozen exchanges covering the page types and edge cases you care about is usually enough. Recording thousands of pages makes the fixture set slow to load, hard to review and heavy in a repository, and it adds nothing a well-chosen sample doesn’t already cover.

Respect the target’s rules while recording, the same as in production: robots, rate limits, terms of service, and staying away from logged-in areas and personal data you shouldn’t hold. A fixture set built by ignoring those rules has the same problems as a scraper built that way, plus it now lives in your repository.

Choosing which responses to keep

The value of a fixture set comes almost entirely from what you choose to record. A set with twenty ordinary product pages tests the same thing twenty times.

Start with one clean example of each page type the scraper handles: a listing page, a detail page, a search result, a category index, whatever the job involves. These are your baseline.

Then add the edge cases, which are where parsers actually break. A product with no price, or a price on request. A sale price alongside a crossed-out original. An out-of-stock item. A listing page with zero results. The last page of pagination, where the next link is missing. A page with an unusually long title or a field containing characters outside ASCII. A page in a second language or currency, if the job spans regions. A detail page where an optional section is missing entirely.

Add the pages your scraper should reject. A soft block page that returns 200 with a challenge or an empty template. A not-found page served with 200. A redirect to a login page. A maintenance page. The soft blocks piece on this site describes how these look in the wild. Your scraper’s validation logic should recognise them, and the only way to test that offline is to have copies.

Add the responses that exercise error handling: a 404, a 429 with a retry-after header, a 503, a truncated body, a response with a content type that doesn’t match its body.

Finally, give each fixture a name that says what it’s for, not where it came from. product_sale_price.html tells the next reader what it tests. page_17.html tells them nothing.

A set like this might be thirty or forty files. That’s plenty to catch most regressions.

Parser tests against page fixtures

The simplest and most valuable tests feed a page fixture into the parser and check what comes out.

That works best when the parser is a pure function: HTML in, structured records out, with no network calls, no global state and no dependence on the current time unless it’s passed in. If your parser currently fetches pages itself, separating fetching from parsing is worth doing for this reason alone. Then a test is a few lines: load the fixture bytes, call the parser, assert on the result.

For assertions, there are two useful styles. Targeted assertions check specific fields: this fixture should produce one record with a price of 19.99 and a currency of EUR, and the in_stock flag false. They’re precise and they document intent. Snapshot or “golden file” tests save the parser’s full output for each fixture to a file, and the test compares new output against the saved version. Any change in any field shows up as a diff. Snapshot tests catch changes you didn’t think to assert on, which is exactly what you want when refactoring a parser.

A good suite uses both. Targeted assertions on the fields that matter most, and a snapshot of the whole output as a safety net.

When a snapshot diff appears, read it rather than blindly accepting it. A change you intended shows up as the expected field changing. A change you didn’t intend, like a description field suddenly empty on three fixtures, is the bug the snapshot exists to catch.

Rejection fixtures get their own tests. Feed the soft block page to your validator and assert that it’s flagged, that no records are emitted, and that the result is marked as a block, not as a page with zero products. That distinction is easy to lose in a refactor and expensive to lose in production.

Replaying sessions for crawl logic

Parsers aren’t the only thing that breaks. Crawl logic, the code that decides which links to follow, when to stop paginating, how to handle redirects and retries, has its own bugs, and page fixtures alone can’t test it.

Session fixtures replay at the HTTP level. The test runs the actual crawler, but every request it makes is answered from the recorded set instead of the network. If the crawler asks for page two of the listing and there’s a recording for that URL, it gets the recorded response. If it asks for something that wasn’t recorded, the replay layer can raise an error, which tells you the crawler’s behaviour changed.

That last property is useful in itself. A crawler that suddenly requests URLs it never requested before has changed its navigation, and a test that fails on an unrecorded request makes that visible. It might be a bug, or it might be intended, but either way someone should look.

Matching rules deserve attention. Replay libraries match incoming requests to recordings by some combination of the method and URL, the query string, and sometimes headers or the request body. Too strict, and harmless differences such as a changed header order or a timestamp parameter cause misses. Too loose, and two different requests get the same answer. Configure matching to ignore volatile parts like cache-busting parameters and to respect the parts that matter, like page number.

Session fixtures also let you test retry and backoff. Record a 429 with a retry-after header followed by a 200 for the same URL, and check that the crawler waits and retries correctly, using a fake clock so the test doesn’t actually sleep. The same goes for 503s and connection errors. Testing politeness logic this way is far better than discovering it’s broken because a target started returning errors in production.

Time, randomness and other things that break replay

Replay only works if the code under test asks for the same things it asked for during recording, and scrapers often don’t.

Time is the usual culprit. A crawler that adds a timestamp to a query string for cache busting, or that computes “yesterday’s date” to build a URL, will request different URLs every day. A parser that compares a listing’s date against today to decide whether it’s new will produce different output tomorrow. Pass time in as a parameter, or use a fake clock in tests, and configure the replay layer to ignore volatile query parameters when matching.

Randomness is next. Crawlers that shuffle their queue, pick a random user agent from a list, or add jitter to delays are harder to replay deterministically. Seed the random generator in tests, or inject the randomness as a dependency you can control.

Ordering matters for session fixtures. If the crawler fetches pages concurrently, requests may arrive at the replay layer in a different order each run. Most libraries match by request rather than by position, which handles that, but a recording that includes the same URL twice with different responses, such as a 429 then a 200, needs ordered playback for that URL. Check how your library handles repeats.

Finally, watch for pages that embed values unique to each fetch: a CSRF token, a nonce in a script tag, a session ID in a link. If the crawler reads one of those and sends it back in the next request, the replay layer must match on a request that includes the recorded value, which it will, as long as the value came from the recording. Problems start when part of the value comes from the live code instead.

Making accidental live calls impossible

A test suite that’s supposed to be offline will, sooner or later, make a live call by accident. Someone adds a new request path that wasn’t recorded, and the replay library is configured to fall through to the network. Or a dependency makes its own HTTP call. Or a test uses a real client instead of the one wired to the replay layer.

In CI, that call either fails because there’s no network or proxy configured, which is confusing, or it succeeds and hits the target from your CI runner’s address, which is worse.

So block the network in tests, deliberately. In Python, a plugin such as pytest-socket can disable socket connections for the test run, so any attempt to open one raises an error pointing at the offending test. Replay libraries usually have a mode that refuses to record or pass through, and CI should run in that mode. Node libraries like nock have a setting to disallow unmatched network connections. At the infrastructure level, CI jobs for the scraper can run in a container with no outbound network at all.

The result is that any live call fails loudly, in the test that made it, with a message saying which URL was requested. That’s the moment to decide whether to record a new fixture or fix the code.

This also keeps proxy credentials out of CI entirely. The test suite never needs them, because it never uses the proxy. Fewer places holding credentials means fewer places they can leak from, and CI logs are a common place for them to leak.

Browser pages and what to record

Pages that need a browser to render are harder to fixture, but not impossible.

The first option is to record the network traffic the browser generates, typically as a HAR file, and replay it. Playwright supports this directly: record with a HAR option on the browser context, then route requests from the HAR in tests. The browser loads the page, runs its scripts and makes its API calls, all answered from the recording. This tests the whole rendering path, but HAR files can be large, and pages with randomised or time-dependent requests may not replay cleanly without tuning the matching.

The second option is to save the rendered DOM after the page has loaded, and test the parser against that. It’s much smaller and simpler, and it tests the part most likely to change, the extraction. It doesn’t test whether the page renders the same way next time, but a scheduled live check can cover that, as described further down.

The third option, often the best, is to notice that the browser was fetching structured data from an API behind the page, record those API responses, and test against them directly. If your scraper can read that data without a browser in production, you’ve also cut the job’s bandwidth and complexity, which the piece on reading a target’s API before scraping its HTML covers.

Whichever you choose, record with images, fonts and media blocked where the test doesn’t need them. They bloat fixtures and add nothing to a parser test.

Sanitising what you store

A recorded exchange contains more than the data you wanted, and some of it shouldn’t sit in a repository.

Request headers can hold proxy authorisation and cookies, along with API keys or session tokens. Response headers can set cookies tied to a session. If you recorded through a proxy with credentials in the URL, some recording tools will capture that URL too. Scrub all of it before committing. Most replay libraries have filter hooks for exactly this, letting you replace sensitive header values with placeholders at record time. Use them, then check the saved files by eye anyway.

Bodies can hold personal data. A page might show reviewer names, a seller’s contact details, or user comments. If the scraper’s job doesn’t need that data, the fixture shouldn’t keep it. Replace it with obvious placeholders, or pick a different page that doesn’t include it. The piece on personal data in a scrape you didn’t plan for covers why holding it creates obligations you probably don’t want.

Think about what the pages themselves are. Fixtures are copies of someone else’s content. Keep the set small, keep it in a private repository, use it only for testing, and don’t redistribute it. Trimming a fixture to the parts of the page your parser actually reads is a reasonable way to keep it minimal, as long as you keep enough structure around those parts that the test stays realistic.

And keep an eye on size. A few dozen HTML pages is fine in a repository. Hundreds of megabytes of HAR files isn’t. Store large fixtures with a large-file mechanism or in a separate artefact store, and keep the everyday set small.

Fixtures from more than one exit and region

A scraper that runs through proxies in several countries, or through more than one type of exit, may see materially different pages depending on where each request leaves from. A fixture set recorded from one exit only tests one version of the site.

If the job covers regional storefronts, record a baseline set for each region the parser has to handle. Currency, number formatting, date formats, the language of labels, and even which fields exist can differ. A price parser tested only on a page that writes 1,299.00 will meet a page that writes 1.299,00 in production, and the fixture set is the cheapest place to find that out.

If the site treats exit types differently, record from the exit type production uses, and consider recording from an ordinary connection too, so you have a reference copy. When the two differ, the difference is worth keeping as a fixture pair. It documents what the site does, and it gives your validator something to test against: given the datacenter version of this page, does the scraper recognise that fields are missing?

Tag each fixture with the region and exit type in its metadata. When a test fails on the German fixtures and passes on the British ones, you want to know that immediately, not after opening the files.

This is also where fixtures pay back their recording cost fastest. Testing five regional variants against the live site means five sets of proxy exits, five sets of target load, and five chances for the network to add noise. Testing against five fixture sets takes seconds.

One parser change, worked through

Here’s how the whole thing plays out on a routine change, to make the flow concrete.

The scheduled live check flags the product detail page type. The selector for the price no longer matches on the live sample; everything else still does. Nobody’s pipeline has broken yet, because the check runs ahead of production.

Someone records a fresh detail page, politely, through the production exit type. They sanitise it, name it product_detail_after_redesign.html, and add a metadata note: captured on this date, price block moved into a new component, old fixture kept for comparison.

They run the parser tests. The new fixture fails the targeted assertion on price, as expected. The snapshot diff shows price empty and every other field intact. They update the selector, preferring a structured source where one exists, such as a JSON-LD block on the page, since that tends to survive redesigns better than markup does.

They rerun. The new fixture passes. The old fixture, if the parser is meant to handle both during a rollout, still passes too. The snapshot for the new fixture gets reviewed and accepted.

The change goes through CI offline in seconds, with no proxy credentials involved, and ships. The next scheduled live check passes.

Total live traffic for the whole change: the scheduled check’s handful of requests and one recording fetch. Without fixtures, the same change would typically be tested by running the scraper against the site a few times, which is more traffic and far less certainty.

Knowing when fixtures have gone stale

Fixtures freeze a site at a moment. The site keeps moving. Eventually your fixtures describe a site that no longer exists, your tests pass, and production quietly extracts nothing.

That’s the main risk of fixture-based testing, and it needs its own control. The usual answer is a small scheduled live check, separate from the test suite. Once a day or once a week, depending on how often the site changes, it fetches one live example of each page type, politely, through the production proxy setup. It then compares what it got against the fixture for that page type.

The comparison shouldn’t be a byte diff, because live pages always differ in small ways. Compare structure instead. Do the selectors the parser relies on still match? Does the parser produce records with all required fields present? Is the page size within a plausible range of the fixture’s? Has the set of script and style URLs changed in a way that suggests a redesign? The redesign piece on this site lists the early signals worth watching.

When the live check flags a difference, that’s the trigger to record fresh fixtures for that page type, look at the diff between old and new, update the parser, and let the snapshot tests show exactly what changed in the output. Keep the old fixture around for a while, renamed, if the site might roll back or serve both versions during a transition.

This check is the only part of the setup that sends live traffic, and it’s tiny: a handful of requests on a schedule. It replaces the hundreds of pages that ad hoc live testing would have fetched for every code change.

Fixtures in CI, step by step

Putting it together, here’s what the CI side looks like for a typical scraper repository.

Fixtures live in a directory next to the tests, organised by page type, each with a small metadata file or a shared index recording capture date, exit type and purpose.

The test job runs with the network blocked, either through a test plugin or by running in a container without outbound access. The replay layer runs in a mode that refuses to record or pass through.

Parser tests load page fixtures and run targeted assertions and snapshot comparisons. Crawl tests replay session fixtures through the real crawler with a fake clock. Validation tests feed block pages and error responses and check they’re classified correctly.

The job has no proxy credentials, no target URLs it can reach, and no dependence on the time of day or what the site served this morning. It passes or fails for reasons that are entirely about your code.

Separately, a scheduled job runs the live structure check with the production proxy configuration and credentials, pages the owner on a mismatch, and does nothing else.

Recording new fixtures is a manual, deliberate step, run by a person, with a review of the resulting files before they’re committed. Automatic re-recording defeats the purpose, because it would silently absorb a site change into the fixtures and hide it.

With that in place, a parser change is tested in seconds on every commit, and the target sees a few requests a week from your team instead of a few hundred per change.

What fixtures can’t tell you

Fixtures test your code against inputs you’ve seen. They’re very good at that and useless for almost everything else.

They can’t tell you how the live site behaves today. That’s what the scheduled check is for, and even it only samples.

They can’t tell you anything about blocking, rate limits or how a target treats your traffic. A fixture of a clean page says nothing about whether the next thousand requests will get clean pages. Those questions belong to monitoring in production: block rate, soft block detection, content validation on every run.

They can’t tell you about proxy behaviour. A fixture doesn’t know whether your exits are healthy, whether the pool rotates correctly, or whether a new country’s exit gets different content. If you add a new region, record new fixtures from that region.

They can’t tell you about timing or performance under real network conditions.

And they can drift into false comfort if nobody watches them. A test suite with stale fixtures passes on every commit while production extracts empty records. The live structure check and the habit of re-recording on any flagged change are what keep the fixtures honest.

Used within those limits, fixtures are the cheapest reliability improvement most scraping codebases can make. Record a well-chosen set once, test against it forever, check the live site on a small schedule, and stop paying proxy bandwidth to find out whether a selector change works.

If you want the related pieces on caching, keeping scrapers alive through redesigns and catching soft blocks, Proxy Scraping has the full set of guides from the operator side.

Get new guides and videos first — join the Telegram channel.

proxies
Need proxies that survive the block wall?

Singapore Mobile Proxy runs real 4G/5G mobile IPs on rotating SIMs — the carrier-grade addresses most of these targets still trust.

see plans →
read on
More scraping guides

The rest of the field manual: target-site playbooks, library walkthroughs, provider reviews, and anti-bot troubleshooting.

browse all guides →