Deciding how often to recrawl a page: a practical recrawl frequency framework

What recrawl frequency actually means

Recrawl frequency is just the interval between the first time you fetch a URL and the next time you fetch that same URL again. It sounds trivial until you’re running a scraper against a few hundred thousand URLs, and you realize the interval you pick for each one has real consequences: stale data if you wait too long, wasted requests and proxy spend if you check too often, and a request pattern that starts to look mechanical if every page gets hit on the same clock regardless of whether anything on it ever changes.

Most people who are new to scraping at scale default to one number. Recrawl everything every hour, or every day, or every week. It’s simple to implement and it’s almost always wrong, because it treats a page that updates every few minutes the same as one that hasn’t changed in two years.

Why a fixed schedule wastes proxy budget

Every recrawl is a request, and every request through a rotating residential or mobile pool has a cost attached, whether that’s billed per GB, per request, or against a session budget. If you’re recrawling ten thousand product pages daily and eight thousand of them haven’t changed since the last pass, you’ve spent proxy budget and target-site load on nothing. That’s money and bandwidth going toward confirming the obvious.

The flip side is just as real. If you recrawl a fast-moving inventory or price page once a week, you’re publishing stale numbers for six days out of seven, which defeats the point of scraping it at all. Neither failure mode is about the proxies themselves. It’s a scheduling problem, and proxies are just where the cost of a bad schedule shows up first, because that’s the line item you actually watch.

Reading the signals a page gives you for free

Before you build any kind of adaptive schedule, check what the page or server is already telling you, because a lot of sites answer this question without you having to guess.

Last-Modified and ETag headers. A well-behaved server sends one or both. You can store the value from your last fetch and compare it on the next check without downloading the whole page.
Sitemap lastmod timestamps. If a site publishes a sitemap with per-URL last-modified dates, that’s a direct signal from the publisher about when something actually changed, and it’s often more reliable than guessing from the rendered HTML.
RSS or Atom feeds. For content sites, a feed timestamp tells you when new items were published without you having to poll the page itself.
Your own change history. The most useful signal is one you build yourself: log every time a recrawl actually detects a diff, and you get an empirical change rate per URL or per URL pattern. That history is worth more than any header, because it’s specific to what you’re tracking, not what the server assumes matters.

If none of those signals are available, you’re working blind, and blind means you should start conservative and let observed behavior correct you rather than assuming a high-frequency need up front.

Building a tiered recrawl schedule

In practice, most of our recrawl scheduling comes down to bucketing URLs by observed or expected volatility, then assigning each bucket its own interval instead of pretending every page belongs in the same tier.

A rough structure that holds up across most scraping targets:

High volatility (minutes to hours): live pricing, stock levels, sports scores, breaking news homepages. These need short intervals and usually justify the extra request cost because the value of freshness is high.
Medium volatility (daily): product listings, job postings, forum threads with ongoing replies. Daily or twice-daily checks catch most changes without hammering the target.
Low volatility (weekly to monthly): static reference pages, older articles, company “about” pages, archived listings. These rarely change, and checking them daily is close to pure waste.
Effectively frozen: pages you’ve observed haven’t changed across several long-interval checks. Push these to the back of the queue or drop them from active recrawl entirely, with an occasional spot check to catch the rare edit.

The key move is that a page can migrate between tiers based on what you actually observe. A product page that starts throwing frequent price changes during a sale period should move up a tier temporarily. One that goes quiet for a month should drift back down. This is closer to an adaptive backoff curve than a fixed cron schedule: tighten the interval after a detected change, loosen it after a run of no-change results.

Conditional requests: checking without paying full price

One of the most underused tools here is the conditional GET. If you stored an ETag or Last-Modified value from the previous fetch, you can send If-None-Match or If-Modified-Since on the next request. A server that supports this returns a 304 Not Modified with no body if nothing has changed, which means you get your answer for a fraction of the bandwidth and a fraction of the load on the target.

This matters for recrawl frequency because it changes the economics. If a full fetch costs you real proxy bandwidth but a conditional check costs almost nothing, you can afford to check more often for pages where freshness genuinely matters, without paying full price every time. Not every server implements this correctly, and some return 200 regardless of the conditional headers, so it’s worth verifying behavior per target rather than assuming it works everywhere.

Where recrawl frequency intersects with block risk

Detection systems generally aren’t looking at a single request in isolation. They’re looking at patterns over time: how often a given identity or IP hits a given URL, how uniform the timing is, whether the traffic shape matches what a human visitor or a legitimate crawler would produce. A recrawl schedule that hits every URL on an identical fixed interval, from the same rotation pattern, every single day, is a pattern, and patterns are exactly what anomaly detection is built to notice.

None of this means there’s a trick to avoid detection. It means that tying your recrawl frequency to actual content change rates has a side benefit beyond cost savings: it naturally produces a less uniform, less mechanical request pattern than a blanket schedule does, because different pages get checked at different intervals for reasons tied to their own behavior. That’s a byproduct of good scheduling, not a bypass technique, and it doesn’t make a scraper undetectable or guarantee it won’t get flagged. Respecting robots.txt crawl-delay directives and a site’s published rate limits still matters regardless of how smart your recrawl tiering is.

Common mistakes we’ve made running this in production

A few patterns worth naming because we’ve hit all of them:

Treating “no diff detected” as “check less” without checking why. Sometimes a page looks unchanged because your parser broke and is silently extracting the same cached value every time, not because the underlying content is stable. Recrawl tiering assumes your change detection is trustworthy, so it’s worth auditing that separately.
Never revisiting the “frozen” bucket. Pages do come back to life, seasonal listings especially. A frozen page with zero recrawl forever will eventually be wrong and you won’t know.
Scheduling by URL count instead of by value. Ten thousand low-value archive pages don’t deserve the same queue priority as a hundred pages that actually drive decisions downstream.

A simple framework to start with

If you’re setting this up for the first time: start every new URL on a moderate default interval, log every recrawl result as changed or unchanged, and after a handful of cycles let the observed change rate move that URL into a faster or slower tier. Use conditional requests wherever the target supports them. Check sitemap and feed timestamps before you poll blind. And treat the schedule as something that adjusts itself from real data, not a number you pick once and forget.

Getting recrawl frequency right isn’t about squeezing more out of a proxy pool. It’s about not paying for information you already have and not going stale on information you actually need.

Want more breakdowns like this on running scrapers and proxy infrastructure honestly? Head back to the Proxy Scraping homepage for the rest of our guides.

Get new guides and videos first — join the Telegram channel.