Caching Responses So A Rerun Does Not Rehit The Site

The problem with rerunning a scraper from scratch

Most scrapers get rerun a lot more than people plan for. You write a script, it breaks halfway through on a parsing bug, you fix the bug, and you run it again from the top. If nothing is cached, that second run hits every single page again, including the thousands of pages that parsed fine the first time and didn’t need to change at all.

We run proxy infrastructure for scraping workloads, and this pattern shows up constantly in support tickets: someone burns through a day’s worth of proxy bandwidth and rotation budget re-fetching pages they already had a perfectly good copy of, sitting in a log file nobody thought to reuse. It’s not a proxy problem. It’s a caching problem, and it’s one of the cheapest fixes available.

What caching actually means here

Response caching means storing the raw output of a request (the HTML, the JSON body, the headers you care about) keyed by something deterministic, usually the URL plus any parameters that change the response. Before you make a request, you check the cache. If there’s a valid entry, you use it and skip the network call entirely. If not, you fetch, store the result, and move on.

This is different from retry logic or deduplication within a single run. Caching is about persistence across runs, so that killing your script and starting it again tomorrow doesn’t mean paying the full cost of yesterday’s crawl a second time.

Why this matters more for scraping than for typical API caching

In an internal API, caching is usually about latency. In scraping, caching is about not being the reason a site’s traffic graph spikes. A site you’re pulling from has no way to distinguish “a scraper being efficient” from “a scraper hammering us for no reason” except by looking at request volume and pattern. Every request you don’t need to make and skip is one less data point pushing you toward rate limiting or a block.

It also matters for your own infrastructure. If you’re running requests through rotating residential or mobile proxies, every cached hit is a request you didn’t pay bandwidth for and didn’t burn a rotation on. On a large crawl, the difference between caching and not caching can be the difference between finishing a rerun in ten minutes and finishing it in six hours.

The simplest version: cache by URL

The easiest cache to build is a key-value store where the key is the normalized URL and the value is the response body plus a timestamp. Normalize the URL first (sort query parameters, strip tracking parameters like utm_source, lowercase the host) so that two URLs that are functionally identical don’t create two cache entries.

import hashlib
import json
import time
from pathlib import Path

CACHE_DIR = Path("cache")
CACHE_DIR.mkdir(exist_ok=True)

def cache_key(url):
    return hashlib.sha256(url.encode()).hexdigest()

def get_cached(url, max_age_seconds):
    path = CACHE_DIR / cache_key(url)
    if not path.exists():
        return None
    entry = json.loads(path.read_text())
    if time.time() - entry["fetched_at"] > max_age_seconds:
        return None
    return entry["body"]

def store_cached(url, body):
    path = CACHE_DIR / cache_key(url)
    path.write_text(json.dumps({"fetched_at": time.time(), "body": body}))

This is deliberately unglamorous. A directory of files keyed by hash, a timestamp, and a max-age check is enough for most single-machine scraping jobs. You don’t need Redis or a database until you’re running distributed workers that need to share a cache, or your cache is big enough that filesystem lookups get slow.

Deciding how long a cached response stays valid

This is the actual design decision, and it depends entirely on how often the underlying page changes. A product listing page on an e-commerce site might change hourly. A company’s “about us” page might not change for a year. Caching everything for the same duration wastes either freshness or requests, depending on which way you get it wrong.

A few practical patterns:

• Static or slow-moving pages (documentation, company profiles, historical records): cache for days or weeks. There’s no reason to refetch these on every run.
• Pages you expect to change but not urgently (product descriptions, category pages): cache for hours, tuned to how often you actually need fresh data for your use case.
• Pages where freshness is the whole point (prices, stock levels, live scores): cache for minutes, or don’t cache the volatile fields at all while still caching the parts of the page that don’t change, like the layout scaffolding you’d otherwise re-parse.

If you’re not sure, start with a conservative cache window (a few hours) and shorten it only for the specific pages where you’ve confirmed you need fresher data. It’s easier to tighten a cache than to explain to a site owner why your scraper suddenly tripled its request rate.

Respecting the site’s own signals

A lot of sites tell you how long content is good for, and it’s worth reading those signals instead of guessing. HTTP responses can include Cache-Control and Expires headers, and some sites set them meaningfully even for pages that aren’t behind a CDN. If a response says Cache-Control: max-age=3600, that’s the site operator telling every client, including browsers, how long that content is expected to stay valid. Using that as your cache window instead of inventing your own number means your scraper’s refresh rate matches what the site itself considers reasonable.

Conditional requests are the more precise version of this. If you store the ETag or Last-Modified header from a previous response, you can send it back on the next request (If-None-Match or If-Modified-Since). If nothing changed, the server can reply with a 304 Not Modified and an empty body instead of sending the full page again. This is lighter on both sides: your scraper does less parsing, and the site does less work generating a response it knows you already have. Not every site implements this correctly, but where it’s supported, it’s the cleanest way to check “did this change” without pulling the whole page down each time.

Cache invalidation you control on purpose

Time-based expiry handles the common case, but sometimes you know a specific page changed and want to force a refetch regardless of the cache window. Build in a way to bypass or clear the cache for specific URLs, either through a command-line flag or by deleting the specific cache file. Don’t make cache invalidation something that requires wiping the entire cache directory. If your only invalidation tool is “delete everything,” you’ll end up either never using it or using it too often, both of which defeat the purpose.

It also helps to log what the cache is doing: how many requests were served from cache versus fetched fresh, on each run. This number is the clearest signal you have that caching is actually working, and it’s the first thing worth checking if a run that used to take ten minutes suddenly takes an hour again.

Where caching fits with proxies and rate limits

Caching and proxy rotation solve different problems, but they work together. Proxies exist to distribute request volume across different IPs so you’re not sending an unrealistic number of requests from a single address. Caching reduces the total volume you need to send in the first place. A scraper that caches aggressively needs fewer proxy IPs and fewer rotations to stay under any given site’s rate-limiting thresholds, because it’s simply making fewer requests.

This doesn’t mean caching is a workaround for a site’s bot detection or rate limits, and it shouldn’t be framed as one. A site that blocks based on request patterns, headers, or behavioral signals will still evaluate the requests you do make on their own merits. What caching changes is how many of those evaluations happen at all. Fewer unnecessary requests is a legitimate operational improvement, not a way to slip past a defense.

What to actually build first

If you’re adding caching to an existing scraper for the first time, start with the simplest version: hash the URL, store the response body and a timestamp, check age before fetching. Add header-aware invalidation (ETag, Last-Modified) once you’ve confirmed the sites you’re working with actually send those headers. Tune per-page expiry windows once you know which pages on your target sites change often and which don’t. Log cache hit rates so you can see the effect instead of guessing at it.

None of this requires new infrastructure. It requires treating the second run of your scraper as a different problem than the first, because it is one.

For more on running scrapers efficiently, comparing proxy types honestly, and keeping crawl behavior clean at scale, visit Proxy Scraping.

Get new guides and videos first — join the Telegram channel.