Storing what you scraped so tomorrow's run is cheaper

The part of the pipeline nobody budgets for

Most scraping write-ups spend all their time on the collection side: which proxy pool to use, how to rotate sessions, how to space requests so a target site’s rate limiter doesn’t flag the traffic. Storage gets an afterthought, usually a line that says “then save to CSV.” That’s a mistake if you’re running this more than once.

Every proxy request costs something. Residential and mobile IPs are billed by the gigabyte on most providers, and even flat-rate datacenter plans have a practical ceiling before you’re buying more IPs or more bandwidth. If your storage layer can’t tell you what you already have, every run starts from zero and pays full price for data you fetched last week. Good storage design is a cost lever, not just a place to dump JSON.

What “cheaper tomorrow” actually means

There are three separate savings a storage layer can produce, and they don’t require any change to how you talk to proxies or how you space out requests:

• Skip fetches for unchanged pages. If you know a product page hasn’t changed since Tuesday, you don’t need to re-render it Wednesday.
• Skip re-parsing on partial failures. If a run dies halfway through, you want to resume from record 4,001, not re-fetch 1 through 4,000.
• Skip duplicate targets across jobs. If two different scrapers on your team are both hitting the same URL list, one of them shouldn’t be paying proxy costs the other already paid.

None of these touch how a request looks to the target site. They’re entirely on your side of the wire. That matters, because it means storage optimization is one of the few places in a scraping pipeline where you can cut cost without adding any risk of looking more like a bot.

Content hashing: the cheapest check you can run

The simplest thing to store alongside every scraped record is a hash of the content itself, not the URL, not the timestamp: a hash of the page body or the extracted fields, whichever you actually use downstream. SHA-256 over the raw HTML works if you need forensic certainty; a hash over just the fields you extract works fine if you only care whether the data changed, and it’s cheaper to compute and compare.

On the next run, before you decide to spend a proxy request re-fetching a URL, check whether you have reason to think it changed. For pages that update on a schedule (news, listings, prices) this reasoning is usually “enough time has passed that it’s worth checking.” For pages that rarely change (static profile pages, archived listings) it might be “only re-check if something upstream told us to.” Either way, once you do fetch, compare the new hash to the stored one before you touch storage or trigger downstream processing. If they match, you’ve confirmed nothing changed, but you’ve also confirmed the fetch was worth doing to know that, so the hash isn’t saving the fetch cost here, it’s saving everything after the fetch: no re-parsing, no re-writing rows, no re-triggering whatever consumes the data.

The actual proxy savings come from a layer above the hash: deciding which URLs are even eligible for re-fetch on a given run. That decision should be driven by change frequency you’ve observed, not a blanket “re-scrape everything daily” policy, which is the single most common source of wasted proxy spend I see in scraping setups.

Structuring storage so “have I seen this” is a fast question

A lot of scrapers store output as flat files, one per run, named by date. That’s fine for a one-off pull. It’s a poor foundation for anything recurring, because answering “have I already got this URL, and when” means opening every file and scanning it.

A minimal relational structure fixes this cheaply. One table for targets (URL, last fetched timestamp, last content hash, fetch status), one table for the actual scraped payloads keyed by target ID and fetch timestamp. Even SQLite is enough for this at moderate scale, since the win is being able to ask “which of these 10,000 URLs do I actually need to hit today” in one query instead of a file scan, rather than anything to do with query throughput.

This also gives you a natural place to record failures. If a fetch through a given proxy returns a 403, a captcha page, or just times out, log that against the target with the outcome type. On the next run you can decide, deliberately, whether to retry it, retry it through a different exit node, or leave it alone for now. Without that record, failed fetches tend to either get silently dropped or blindly retried every run, both of which cost money for no return.

Deduplication across jobs, not just within one

If you’re running several scrapers, or the same scraper reused for different clients or projects, it’s worth checking whether they overlap on target URLs before anyone starts a job. This is a five-minute check against your storage layer: pull the URL list for the new job, diff it against what’s already fresh in storage, and only queue the ones that are missing or stale.

This sounds obvious written down, and it’s still the thing I see skipped most often in practice, because the scraper and the storage layer get built by different people at different times and nobody wires the check back in. The proxy bill doesn’t care whose job it was.

Retention: don’t store what you’ll never use again

The flip side of storing enough to skip re-fetching is not storing so much that the storage itself becomes the cost center. Raw HTML for every fetch, kept indefinitely, adds up fast if you’re pulling thousands of pages a day. Decide up front what you actually need long-term: usually it’s the extracted fields plus the content hash plus a fetch timestamp, not the full page body. Keep raw HTML around only as long as you need it for debugging parser breakage, then let it expire.

This isn’t a proxy cost saving directly, but it keeps the storage layer cheap and fast enough that the freshness checks above stay fast too. A bloated storage layer that takes ten seconds to answer “have I seen this URL” will get bypassed under deadline pressure, which puts you back to re-fetching everything.

Where this intersects with proxy choice

None of the above changes which proxy type you should use for a given target. That decision still comes down to how the target site treats traffic: sites with light bot defenses are fine on datacenter IPs, sites that fingerprint aggressively and rate-limit by ASN tend to need residential or mobile exit nodes to get a representative view of the page at all, and no proxy type is unblockable regardless of what the sales page says. What good storage does is reduce how many times you need to make that request in the first place. Cutting request volume by skipping known-unchanged targets is the one place where a scraping operator can lower proxy spend without touching anything about how the requests are made, which is why it’s worth building before you build a bigger proxy pool.

If you’re scaling a scraping operation up, storage is usually the cheaper problem to solve first, before you spend more on IPs.

For more on choosing proxies and running scrapers that hold up over time, head back to the proxyscraping.org homepage.

Get new guides and videos first — join the Telegram channel.