Targets where plain datacenter proxies are still enough
The reflex that quietly eats the budget
A team wants to pull a few thousand pages of product documentation from a vendor’s public docs site. Somebody has read that residential proxies are what serious scrapers use, so the job goes out through a residential pool billed by the gigabyte. It runs fine. It would also have run fine from a plain datacenter exit, or from the office connection at a gentle pace, and nobody checks, because the job never failed.
That’s the pattern this piece is about. The residential reflex is rarely wrong in the sense of breaking anything. It’s wrong in the sense of paying premium rates for traffic that a cheaper exit would have carried without a single extra error.
I run proxy infrastructure, including mobile lines that sit at the expensive end of the market, so I’ve got no reason to talk anyone out of spending. But the quickest way to lose a customer’s trust is to sell them an exit type their target doesn’t care about. A lot of the public web still doesn’t care where a polite request comes from.
The comparison between residential, datacenter and mobile exits is already on this site, and it starts from the proxy. This one starts from the other end. You look at the target first, decide what it’s likely to tolerate, and only then pick the exit. When you work in that order, datacenter comes out as the right answer more often than the marketing around proxies suggests.
None of this is a promise that any particular site will serve you. Targets change, and the checklist further down is about recognising likely candidates, then confirming with a small test before you commit a whole job. Think of it as a way to avoid overpaying, and keep treating the target’s rules as the thing that decides what you’re allowed to fetch at all.
Start from the target, not the proxy type
Proxy shopping usually begins with a question about the proxy. Which type is best, which provider has the biggest pool, which one people on a forum recommend. Those questions have no answer in the abstract, because the right exit depends entirely on what’s on the other end.
So flip it. Before you look at a single pricing page, write down three things about the target. Who runs it. What they gain from blocking automated traffic. What they lose if they block a legitimate visitor by mistake.
A public statistics office publishes data so people will use it. Blocking a researcher’s script makes their job harder and helps nobody, so they tend not to bother beyond basic rate limiting. A documentation site exists to be read, indexed and linked; the vendor wants search engines and developer tools crawling it. A small independent shop running on a modest server mostly wants to stay online during a traffic spike, and it may not have any bot management at all.
Compare that with a large marketplace, a ticketing site, or an airline fare page. Those operators have direct commercial reasons to control who reads their data at volume, and they pay vendors to sort visitors into buckets. Exit type genuinely matters there, along with a dozen other signals.
The honest middle is big. Most of the public web sits between “wants to be read” and “actively defends its data”, and much of it leans towards the first. When you describe the target before you describe the proxy, you stop buying for the hardest case every time.
What a datacenter exit actually tells a site
A datacenter address is one registered to a hosting company or a cloud provider. Anyone can look up the autonomous system that announces it, and the organisation name usually says it plainly: a cloud platform, a colocation firm, a hosting reseller. Reverse DNS often gives it away too, with hostnames that look like server names rather than home broadband lines. The whois and reverse DNS piece on this site walks through that lookup in detail.
So a site that wants to know can find out, cheaply and in advance, that a request came from a server rather than a person at home. Bot management products ship with lists of hosting ranges for exactly this purpose, and some sites apply stricter rules or a challenge page to those ranges by default.
Knowing is one thing. Acting on it is a choice the site makes, and many sites have good reasons not to. Plenty of legitimate traffic comes from datacenters: search engine crawlers, uptime monitors, link preview services, corporate VPN gateways, RSS readers, accessibility tools, and every company that routes staff browsing through a cloud security proxy. Blocking all of it by default breaks things the site owner actually wants.
That’s the mechanical reason datacenter exits keep working on a large share of the web. The site can tell, and it has decided the distinction doesn’t matter for its pages. What does matter, on almost every site, is how the traffic behaves. A datacenter exit making one request every few seconds with an honest user agent looks like the monitoring and indexing traffic the site already accepts. The same exit hammering hundreds of requests a minute looks like trouble from any network.
Documentation and reference sites
Documentation sites are the clearest case. Software vendors, open source projects and standards bodies publish docs because they want developers to find and use them. The pages are usually static or generated at build time, served from a CDN, and cached aggressively. Traffic from servers is normal there, because developer tools, search engines, offline doc readers and internal knowledge bases all fetch these pages automatically.
What you’ll usually see on a docs site is a sitemap that lists every page, a robots file that allows most of the site, and HTML that renders fully without JavaScript. Sometimes there’s also a downloadable archive or a public repository holding the source files. In that last case, the scrape itself is the wrong tool; cloning the repository gets you the content in its original form with zero load on the web server.
Reference and knowledge sites follow similar logic, with more variation. A public wiki, a glossary, a specification site or a technical encyclopedia wants readers. Some of the larger ones publish database dumps or APIs precisely so that people stop scraping the HTML, and using those is both kinder and more reliable.
The catch with this category is that the CDN in front may belong to a vendor that also sells bot management. A docs site on a big CDN might have the basic tier with no bot rules, or it might have a security setting that challenges hosting ranges. You can’t tell from the domain alone. That’s what the test later in this piece is for. In practice, though, docs and reference sites are the category where a datacenter exit at a sensible pace is the default right answer, and paying per gigabyte for residential traffic mostly buys you nothing.
Government portals and open data
Government statistics offices, open data portals, legislative databases, court listings in jurisdictions that publish them, planning registers and procurement notices all share a purpose. They exist so the public can read the information. Many are legally required to publish it. Their audience includes researchers, journalists and companies that pull the data into their own systems.
These sites often have the least sophisticated bot defence of anything on the public web. What they do have, and you should respect it, is limited capacity. A regional planning portal may run on one ageing server behind a basic firewall. A datacenter exit won’t bother it. Fifty concurrent connections from any exit might take it down for the local residents who actually need it that afternoon.
So the constraint on government targets is almost never exit type. It’s pace, timing and whether a better channel exists. Many portals publish bulk downloads, CSV exports or a documented API with its own rate limits and terms. When one exists, use it. When the only route is the HTML, crawl slowly, outside local business hours if you can, and identify yourself with a descriptive user agent and a contact address. The site owner can then reach you instead of blocking you.
One edge case worth flagging. A few government services sit behind the same commercial CDN and security products as private sites, especially anything that handles logins, payments or identity. Those areas are out of scope anyway. A public statistics table and a citizen login portal might share a domain and have completely different protections; judge each section on its own.
Small shops and independent storefronts
Small independent shops are the category people most often overpay on, because “e-commerce” sounds like a hard target. Big retailers are hard targets. A shop with a few hundred products on a self-hosted cart, or a specialist supplier with a catalogue page, usually is not.
These sites typically run on a small hosting plan or a managed platform, and the owner’s priority is keeping the site up and fast for customers. They may have a security plugin with basic rate limiting, or nothing at all. Publicly listed prices and stock levels are the usual target for a scraper, and those are shown to every visitor on purpose.
The complication is the platform. A small shop on a large hosted commerce platform inherits that platform’s edge protections, which can include bot scoring applied to everyone on the platform. So two shops of the same size can behave very differently: the self-hosted one serves anything polite, and the hosted one challenges hosting ranges because the platform decided so network wide. That difference shows up in the response headers and in how the checkout and cart pages behave, and it’s something you learn from a test fetch, not from the shop’s size.
The other complication is fragility. A small shop on shared hosting can be slowed down by a scraper that a big retailer wouldn’t even notice. The owner sees a spike, their host sends them a warning, and they add a firewall rule that blocks your whole range. At that point it was your rate that caused it, and a residential pool would only have spread the same load across more addresses. For small shops, crawl the catalogue slowly, fetch each product page once per recrawl cycle, and stay well away from cart, checkout and account pages.
Feeds, sitemaps and anything published for machines
A surprising amount of what people scrape was published for machines in the first place. RSS and Atom feeds, sitemaps, public JSON endpoints behind a site’s own search box, product feeds for comparison engines, and schema markup embedded in pages are all built on the assumption that software will read them.
Machine-facing resources are the easiest possible case for a datacenter exit. Feed readers and aggregators run from servers. Search engines fetch sitemaps from their own infrastructure. Nobody expects a sitemap to be fetched from a home broadband line, and blocking hosting ranges from a sitemap would break search indexing, which almost no site owner wants.
So before planning a crawl of HTML pages, look for the machine version. A sitemap tells you which URLs exist and often when they last changed, which lets you skip unchanged pages entirely. A feed may carry the full content you wanted. A public endpoint that the site’s own front end calls may return clean structured data with a fraction of the bytes. The sitemap planning piece on this site covers the crawl side of that in depth.
This matters for the cost argument too. The cheapest request is the one you don’t make, and the second cheapest is one that returns a small structured response instead of a heavy rendered page. Once you’re reading feeds and sitemaps, exit type barely registers as a cost, because the traffic volume drops so far that even premium bandwidth would be cheap. Datacenter is simply the natural fit for traffic that looks exactly like the aggregator and indexer traffic the site was built to serve.
The checklist before you pay premium rates
Here’s the checklist I’d run on any new target before deciding the exit type. None of these points proves anything alone. Together they tell you whether the target is a likely datacenter candidate.
- Who runs it and why it’s public. Docs, government data, reference material and small independent shops lean towards “wants to be read”.
- Whether a sanctioned channel exists: an API, a bulk export, a data dump, a feed, a public repository. If it does, that’s usually the answer and the exit question disappears.
- What the robots file says, and whether it names specific crawlers differently from the default. A site that allows general crawling and publishes a sitemap is signalling that automated reading is expected.
- Which CDN or security vendor sits in front, visible in response headers and cookies. A plain CDN with caching headers is different from a bot management product setting challenge cookies on the first response.
- Whether the first page loads fully as HTML, or arrives as an interstitial that needs JavaScript before the content appears.
- Whether the terms of service say anything about automated access, and what.
- How large the site’s infrastructure looks. A shared hosting plan needs a gentler pace than a global CDN, whatever exit you use.
If most of those point towards “low defence, wants to be read”, a datacenter exit is the reasonable starting point. If several point the other way, particularly a bot management vendor or a challenge page on first load, then this is a target where exit type is one of many signals, and you should think carefully about whether and how to scrape it at all.
A two-exit test that settles most cases
The checklist gives you a guess. A small test turns it into evidence. Fetch the same handful of pages twice: once from a datacenter exit, once from an ordinary connection such as your own home or office line, at a slow pace, with the same client and the same headers.
Then compare the two sets of responses properly. Status codes are the start, but the soft blocks piece on this site explains why a 200 can still be a block page. Compare the response sizes. Compare the actual content: the product count on a category page, the prices, the presence of the data fields you care about. Check for challenge cookies or scripts in one response and not the other. Look at the response headers for signs that a security layer made a decision about you.
If both exits return the same content, same size, same fields, and neither triggers a challenge, you’ve got decent evidence that the target doesn’t treat hosting ranges differently at this pace. That’s a datacenter target for now. Repeat the check every so often. Sites switch vendors or flip settings without telling anyone.
If the datacenter exit gets different HTML, the site is making a distinction. The piece on sites that serve different HTML to datacentre IPs covers what to do with that finding, and the short version is that the difference itself is information about what the site wants. If the datacenter exit is blocked or challenged outright, that’s a clear statement too.
Here’s what a worked version looks like. Say the job is a public procurement portal that lists tender notices. Pick twelve URLs: the index, two pages deep into pagination, a search result, and eight individual notices. Fetch each from both exits a few seconds apart. Save every response body to disk with its headers. Then diff them. If the notice pages match byte for byte apart from a timestamp or a session token, and the index shows the same notice count from both exits, the portal doesn’t care where you come from at that pace. If the datacenter copy of the search result is shorter, or the notice count on page two differs, you’ve found a section that treats you differently, and the job plan should reflect it.
Keep the test small, a dozen pages is plenty, and keep it slow. The aim is to learn how the site treats your traffic. Running a full load test against someone else’s server is out of scope.
Where the checklist misleads you
A few traps catch people even after the test passes.
The first is the CDN that arrives later. A docs site or a shop can switch to a new edge provider, or turn on a bot protection setting, on any Tuesday. The job that ran cleanly for months starts getting challenge pages, and nothing in your code changed. The piece on what changes when a target adds a CDN goes into that transition. For the purposes of this checklist, it means your datacenter decision has an expiry date, and you should monitor for it.
The second is the section that differs. A site can be relaxed on its public catalogue and strict on search results, pagination beyond a certain depth, or anything that looks like an API. Test the actual pages the job needs, not just the homepage.
The third is volume. The test at a dozen pages says nothing about what happens at fifty thousand. Many sites rate limit by address, and a small set of datacenter addresses doing a large job will hit those limits sooner than a large residential pool would. Hitting a rate limit is the site telling you the pace is too high. Spreading the same load across more addresses so the limit never triggers goes against what the site asked for, and it’s the kind of thing this site won’t help anyone do. Slow down, spread the job over more time, or ask for access.
The fourth is the soft block that looks like success. A datacenter exit can get a page that returns 200 and looks normal but carries stale prices, missing stock data or a trimmed product list. Validating content, not just status, is the only protection.
Pace still decides the outcome on an easy target
It’s tempting to read “low-defence target” as “anything goes”. It means the opposite. When a site has little automated protection, the only thing between your scraper and a slow or broken site for real visitors is your own restraint.
A datacenter exit has a lot of bandwidth behind it. It can send requests faster than a home connection and faster than a small server can answer. That’s the real risk on an easy target. The exit never gets you noticed on its own terms; running at full speed does, because it degrades the site, the owner notices, and the fix they reach for is a firewall rule that blocks your provider’s entire range. After that, every customer of that provider pays for your hurry.
So on easy targets, set the pace deliberately. Respect any crawl-delay in the robots file. Keep concurrency low, often a single connection for a small site. Honour retry-after headers. Back off hard on any 429 or 503. Cache responses so a rerun doesn’t fetch everything again. Use conditional requests where the server supports them, so unchanged pages cost a tiny response instead of a full download.
Identify yourself where you can. A user agent that names your project and gives a contact address is reasonable on public data work, and it gives a site owner a way to talk to you before they reach for a block. Some owners will simply email and ask you to slow down, which is a much better outcome for both of you than a silent ban. None of this makes a job immune to being blocked. It makes the job one that site owners have little reason to block.
How the cost shapes actually differ
The money argument is simple once you see how each type is sold. Residential and mobile exits are usually metered, most often by the gigabyte, sometimes with a monthly commitment. Datacenter exits are more often sold per address or per port, with bandwidth either generous or not metered at all. Exact terms vary by provider and change often, so read the current pricing page rather than trusting any number in an article, including this one.
What follows from that shape is the part that matters. On a metered residential plan, every byte of every page costs money: images you didn’t need, scripts, retries, redirects, the soft block page you had to fetch before you knew it was a soft block. On a per-address datacenter plan, the marginal cost of a byte is close to zero until you hit a fair use limit, so the cost of a job depends on how many addresses you need, which depends mostly on pace and parallelism.
That means the saving from moving an easy target to datacenter is largest on heavy pages and large jobs. A small daily job fetching fifty light pages may cost next to nothing either way, and the decision barely matters. A weekly crawl of a large docs site or a big open data portal, pulling heavy HTML, is where residential billing turns into a line item someone eventually questions.
A rough way to see it for your own job, without trusting anyone’s benchmark: take one representative page, note its transfer size from the browser’s network panel with images blocked, multiply by the page count and the number of recrawls per month, and put that total against the per-gigabyte rate on the residential plan you were about to buy. Then put the same job against the monthly price of the handful of datacenter addresses the pace actually needs. You’ll know within ten minutes whether the choice is worth a meeting.
There’s also a reliability side to the cost. Residential exits are someone’s home connection, and they drop, change address and vary in latency. Datacenter exits sit in racks with stable power and fibre. For a target that doesn’t care about exit type, you’re paying more for a less predictable connection.
Running a mixed setup without making it complicated
Most real scraping work touches several targets, and they won’t all fit the same exit. The practical setup is a default and an escalation path.
Make datacenter the default for any new target that passes the checklist and the two-exit test. Record the result, including the date, so you know when the decision was made. Route every job for that target through the datacenter pool.
Escalate per target, and only on evidence. If monitoring shows challenge pages, content differences between exits, or a sustained rise in soft blocks that isn’t explained by your own pace, then the target has changed or was misjudged. At that point the questions come in order: is our pace still reasonable, is there an official channel, does the target’s terms allow what we’re doing, and only then, would a different exit type be appropriate and permitted.
Keep the configuration per target rather than per job. When two teams scrape the same docs site, they should share one decision and one rate limit, which is the same argument the internal rotation service piece on this site makes about shared infrastructure generally. Otherwise one team’s residential spend on an easy target never gets questioned, because nobody sees it next to the other team’s datacenter job hitting the same site for a fraction of the cost.
Review the list every few months. Targets drift in both directions. A site that once needed special handling may have published an API, and a site that was relaxed may have added a vendor.
What I can’t tell you from here
I can’t tell you that any specific site will accept datacenter traffic. The categories above describe tendencies, and every one of them has exceptions: a docs site that challenges hosting ranges because its CDN default does, a government portal behind a commercial security product, a tiny shop on a platform that scores every visitor. The two-exit test tells you what one site does at one pace on one day, and that’s all it tells you.
I also can’t give you a threshold for pace that’s safe on every target. It depends on the site’s capacity, its stated rules and what else is hitting it. A crawl-delay in robots, a documented rate limit, or a retry-after header is the site’s own answer, and it beats any rule of thumb.
And none of this changes what you’re allowed to collect. A target being easy to fetch from a server says nothing about whether its terms permit automated access, whether the content is yours to republish, or whether the pages contain personal data you shouldn’t be storing. Those questions come first, before the exit type, and cheap bandwidth doesn’t make them go away.
What the target-first habit does do is stop the default overspend. Describe the target, run the checklist, test with two exits, pick the cheapest exit that serves the job at a polite pace, and keep watching for the day it changes.
If you want the full comparison of exit types, or more on how targets change what they serve depending on where a request comes from, Proxy Scraping has the rest of the guides, written from the operator side.
Get new guides and videos first — join the Telegram channel.