The Bandwidth Bill That Doubled Without More Pages

The invoice that didn’t make sense

Somewhere around the middle of last month I pulled up a residential proxy invoice and did a double take. Same job, same schedule, same target site. Twice the bandwidth. No new pages added to the crawl list, no new brand, nothing in the changelog that touched the scraper’s page count. When a bill moves like that with page volume flat, the answer is never “the site got heavier.” It’s almost always something in the request pipeline quietly pulling more bytes per page than it used to, and it usually takes checking four or five separate layers before you find which one.

This is the walkthrough I use when a bandwidth line jumps and the obvious explanation (more pages, more targets) doesn’t hold up.

Start with what “bandwidth” actually bills

Residential and mobile proxy providers meter bandwidth as total bytes moved through their network in both directions, not per successful page scraped. That distinction matters more than people think. A request that gets rate-limited and retried five times before succeeding bills for all five attempts. A response that comes back as a 200 but is actually a JavaScript challenge page still costs the same bytes as the real page, and now you fetch it again on retry. None of that shows up as “more pages” in your own logs if your logging counts jobs completed rather than bytes transferred per request. So the first move isn’t guessing, it’s turning on byte-level logging per request if it isn’t already there, because without it every other step here is a guess.

Full page loads where you only needed the DOM

The single biggest bandwidth multiplier I’ve seen in production scrapers is a switch from a lightweight HTTP client to a headless browser, or a headless browser that isn’t configured to block subresources. A plain requests-style GET for an HTML page might pull 50-200KB. Point a headless Chromium instance at the same URL with default settings and it will happily fetch every image, font, ad script, tracking pixel, and video preview embedded on that page, because a real browser is supposed to render the whole thing. That can be 5-10x the raw HTML weight, sometimes more on media-heavy pages, and every one of those bytes routes through the proxy and gets billed.

If a target site added client-side rendering or tightened its bot detection and the scraper had to move from a simple HTTP client to a headless browser to keep working, that alone explains a doubling. Check whether the browser instance has resource blocking configured, specifically blocking image, media, font, and stylesheet request types at the network layer. Most headless frameworks (Playwright, Puppeteer) support this natively through request interception. If nobody set it, the browser is proxying every asset on the page for a job that only needed the text content.

Retries and rotation stacking on each other

Rotating proxies are supposed to reduce failure rates, and a badly tuned rotation interval can do the opposite. If a session rotates IPs mid-workflow, on a site that ties session state (cart, login, pagination cursor) to the IP or a cookie that gets invalidated on IP change, every rotation can force the scraper back to a landing page or a login flow it has to re-walk. That is the whole entry sequence repeated per rotation, not one page fetched twice.

Separately, retry logic that doesn’t back off correctly will hammer a target with the exact same request on a fresh IP every time it gets a non-200. If the site is issuing soft blocks, like serving a CAPTCHA page or a “verify you’re human” interstitial instead of a hard 403, and the scraper’s success check only looks for a 200 status code, it will treat that CAPTCHA page as a success, move on, and silently produce garbage data while still burning full bandwidth on every attempt. This is the scenario worth taking seriously precisely because it doesn’t fail loudly. It just costs money and quietly degrades data quality at the same time. The fix isn’t a workaround for the CAPTCHA, it’s validating response content, not just status code, so retries stop when a page is actually a block page and the run alerts a human instead of looping.

Compression getting dropped somewhere in the chain

This one is boring but it’s caught real money before. If a proxy is doing man-in-the-middle TLS interception, which some corporate and some cheaper proxy setups do for logging or filtering, it can strip or fail to pass through the Accept-Encoding: gzip, br header correctly, causing the origin server to send uncompressed responses. Text-heavy pages compress 70-80% smaller under gzip, so losing compression on a text-dominant scrape target can nearly double payload size on its own, with literally nothing else about the job changed. Worth a direct check: pull raw response headers through the proxy and confirm Content-Encoding is actually present on responses, not just that the request asked for it.

Mobile and residential exit nodes aren’t uniform

Different exit nodes on the same rotating pool can carry very different overhead depending on the carrier network and how the provider’s client software wraps traffic. Mobile IPs in particular often route through carrier-side compression proxies or CGNAT layers that can add protocol overhead invisible in a simple total-bytes count. If the rotation logic shifted its exit node mix, say from a pool that was mostly one carrier to a broader mix, the per-request overhead baseline can shift even though nothing in the scraper code changed. This is the hardest one to diagnose because it requires comparing bandwidth per request against exit node metadata, and not every provider dashboard exposes that breakdown. Worth asking the provider directly whether they can segment usage by node type before assuming this is the cause, since it’s a slower check than the others.

The five minute audit, in order

Before opening a support ticket with a provider or budgeting for a rate hike, walk through this in order, cheapest checks first:

1. Bytes per request, not per job. Turn on per-request logging if it isn’t there. This alone usually points at the culprit.
2. Headless vs plain HTTP. Confirm whether the client changed, and if it’s headless, confirm subresource blocking is actually configured, not assumed.
3. Response validation. Check whether “success” is defined by status code alone. If a block page counts as a win, retries and downstream reprocessing are burning bandwidth on bad data.
4. Compression headers. Spot-check raw headers through the proxy for Content-Encoding on a sample of responses.
5. Retry and rotation counts. Pull actual attempt counts per completed page, not just completed-page totals. A 1.3 attempts-per-page average instead of 1.05 shows up as a real bandwidth jump at scale.

None of this requires guessing about what the target site changed on their end. It’s entirely observable from data the scraper and the proxy provider already have. A bandwidth bill that doubles without more pages is a signal that something downstream of “how many URLs are in the queue” changed, and in nearly every case one of these five checks finds it within an hour.

If bandwidth accounting and proxy behavior like this is useful, more breakdowns like this one live at Proxy Scraping.

Get new guides and videos first — join the Telegram channel.