The 2026 Cheerio guide for production scraping
Most people reach for Puppeteer or Playwright the moment they need to scrape anything, and most of the time that’s overkill. If the page you’re targeting ships its data in the initial HTML response, and a huge share of e-commerce, directory, and listing sites still do, you don’t need a browser. You need an HTTP client and a fast HTML parser, and that’s exactly what Cheerio is: a server-side implementation of jQuery’s API for parsing and querying static markup, with none of the memory and CPU overhead of a headless Chromium instance.
This guide is for operators who are already running scrapers at some volume and are paying for it, either in compute costs on a headless browser fleet or in wasted proxy bandwidth from over-fetching. I’m going to walk through a Cheerio setup that survives contact with production: proxy rotation, retry logic, selector drift, and the concurrency limits that keep you from getting your IPs burned in the first hour. By the end you’ll have a script that can run unattended and a sense of what changes as you push it from a hundred requests a day to a hundred thousand.
One thing this guide won’t do is pretend Cheerio is a universal tool. It doesn’t execute JavaScript. If your target renders content client-side, you need a browser-based approach, and I link to that at the end.
what you need
- Node.js 20 or later (nodejs.org), since Cheerio 1.0’s ESM-first build assumes a modern runtime
- the
cheeriopackage (npm install cheerio), currently on the 1.x line per the official Cheerio docs - an HTTP client:
undici(bundled with Node) oraxiosfor the actual fetching, since Cheerio only parses, it doesn’t fetch - a proxy pool. datacenter proxies are fine for sites with light protection; for anything fronted by Cloudflare or a bot-detection vendor you’ll want residential or ISP proxies. I’ve written up specific providers in the Decodo review if you’re picking a vendor
- a concurrency limiter (
p-limitis the smallest one that does the job) - somewhere to persist output: a local SQLite file is enough to start, Postgres once you’re running multiple jobs
- budget: figure $50 to $200/month for a starter residential proxy plan depending on GB usage, plus whatever compute you’re already running the script on. Cheerio itself is free and open source
step by step
1. set up the project
mkdir cheerio-scraper && cd cheerio-scraper
npm init -y
npm install cheerio undici p-limit better-sqlite3
Expected output: a package.json and node_modules with no install errors. If it breaks: better-sqlite3 compiles a native binding, so on Windows you’ll need the Visual Studio Build Tools (or just swap it for sqlite3 with the prebuilt binaries, or skip persistence for now and write to a JSON file).
2. write the minimal fetch-and-parse script
import { load } from 'cheerio';
import { request } from 'undici';
async function fetchAndParse(url) {
const { statusCode, body } = await request(url, {
headers: { 'user-agent': 'Mozilla/5.0 (compatible; research-bot/1.0)' }
});
if (statusCode >= 400) throw new Error(`status ${statusCode} for ${url}`);
const html = await body.text();
const $ = load(html);
return $('h1').text().trim();
}
fetchAndParse('https://example.com').then(console.log);
Expected output: the page’s <h1> text printed to stdout. If it breaks: check the status code you’re throwing on, if you’re getting 403s on a plain GET, the target is likely doing basic bot detection on headers or TLS fingerprint, not JavaScript checks, so the fix is usually a proxy and a more complete header set, not a browser.
3. add proxy rotation
import { ProxyAgent, request } from 'undici';
const proxies = [
'http://user:[email protected]:8000',
'http://user:[email protected]:8000',
];
function pickProxy() {
return proxies[Math.floor(Math.random() * proxies.length)];
}
async function fetchViaProxy(url) {
const agent = new ProxyAgent(pickProxy());
return request(url, { dispatcher: agent });
}
Expected output: requests exiting from different IPs on each call, verifiable by hitting https://ifconfig.me through the same function. If it breaks: a lot of proxy providers require you to whitelist your egress IP rather than use inline auth, check your vendor’s dashboard, that’s the single most common reason a proxy connection just hangs.
4. harden selectors against markup drift
Sites change their HTML without warning. Don’t select on generated class names like .css-1a2b3c, those are build-tool output and rotate on every deploy. Prefer semantic attributes:
const price = $('[data-testid="price"]').first().text().trim();
Expected output: a stable extraction that survives a CSS refactor. If it breaks: your selector returns an empty string instead of throwing, which is worse because it fails silently. Add a check that throws or logs when a required field comes back empty, so you find out the day the site changes instead of a month later when your dataset has a hole in it.
5. add retries with backoff
async function withRetry(fn, attempts = 3) {
for (let i = 0; i < attempts; i++) {
try {
return await fn();
} catch (err) {
if (i === attempts - 1) throw err;
const delay = 500 * 2 ** i + Math.random() * 250;
await new Promise(r => setTimeout(r, delay));
}
}
}
Expected output: transient failures (timeouts, connection resets, 5xx) resolve on retry without you re-running the whole job. If it breaks: if retries never succeed, you’re not dealing with a transient error, you’re dealing with a block. Rotate the proxy before retrying, not just the request, or you’ll hammer the same flagged IP three times in a row.
6. control concurrency and add jitter
import pLimit from 'p-limit';
const limit = pLimit(10);
const results = await Promise.all(
urls.map(url => limit(() => withRetry(() => fetchAndParse(url))))
);
Expected output: at most 10 requests in flight at once, instead of firing hundreds simultaneously and getting rate-limited immediately. If it breaks: if you’re still getting HTTP 429 responses at low concurrency, add a fixed delay between requests to the same domain (200-500ms is a reasonable starting point) rather than relying on concurrency limits alone, since a target can rate-limit by requests-per-second regardless of how many are technically “in flight.”
7. store, dedupe, and log
Write results with an upsert keyed on a natural ID (product SKU, listing URL, whatever’s stable), not an autoincrement row per run, or you’ll accumulate duplicate rows every time the job re-runs. Log every failed URL with its status code and timestamp to a separate table or file so you can distinguish “site changed its markup” from “proxy got blocked” from “target is just down” without re-scraping to find out.
Expected output: a dataset that grows correctly on re-runs and a failure log you can actually act on. If it breaks: if your row count keeps climbing on every run, your dedupe key isn’t stable, check whether the field you’re keying on actually stays constant across page loads.
common pitfalls
Trying to make Cheerio do what a browser does. It doesn’t run JavaScript, evaluate fetch calls the page makes, or handle infinite scroll. If the data isn’t in the raw HTML response (view source, not inspect element), Cheerio can’t see it.
Rotating proxies but not headers. A rotating IP behind a static user-agent and no accept-language header is still a fingerprint. Vary your header set alongside your proxy, and match the accept-encoding your client actually supports, or you’ll get served content encodings you can’t decompress.
Brittle selectors tied to build output. Covered above, but it’s the single biggest source of silent data loss I’ve seen operators run into.
No respect for robots.txt or rate limits. Even for public data, checking the target’s robots.txt directives and pacing your requests is the difference between a scraper that runs for months and one that gets your whole proxy subnet blocked in a week. This isn’t legal advice, robots.txt compliance and the legality of scraping a given site are separate questions, and you should get actual legal counsel if you’re unsure about a specific target.
Treating all 4xx responses the same. A 403 usually means detection, a 429 means you’re too fast, a 404 means the URL is genuinely gone. Log the status code, not just “failed,” or you’ll misdiagnose the fix.
scaling this
At 10x (roughly a few thousand requests a day), a single Node process with p-limit and a handful of rotating proxies is enough. The main thing to add is a scheduler (cron or a simple queue) so runs don’t overlap.
At 100x, a single process stops being reliable. Move to a real job queue (BullMQ on Redis is the common choice) so you can distribute work across multiple worker processes, retry failed jobs independently, and see queue depth. You’ll also need proxy pool management rather than a static array, providers like Decodo and Smartproxy offer rotating gateways that handle this for you, or you roll your own with health checks that pull a proxy out of rotation after N consecutive failures.
At 1000x, you’re running distributed workers across multiple machines, and the bottleneck usually isn’t Cheerio’s parsing speed (it’s fast, it’s not the constraint) but proxy bandwidth cost and target-side rate limits. This is also the point where a subset of targets will have added JavaScript rendering or bot-detection challenges that plain HTTP requests can’t clear, so expect to run a hybrid setup: Cheerio for the sites that still serve static HTML, a headless-browser fallback for the ones that don’t. Monitoring becomes non-optional at this scale, you want alerting on error rate per domain, not just a log file nobody reads. It’s also worth reviewing how your requests look at the TLS and header level, since large-scale HTTP scraping gets fingerprinted the same way browser automation does; antidetectreview.org covers fingerprinting detection in more depth if that’s a gap in your setup.
where to go next
If you’re seeing 429s climb as you scale up, read debugging 429 errors: rate limits, proxy quality, and behavioural patterns for how to tell a proxy quality problem from a pacing problem. If your IPs are getting outright banned rather than rate-limited, diagnosing IP bans: when it’s the proxy vs when it’s your fingerprint walks through the diagnosis. And when you hit a target that Cheerio genuinely can’t handle because the content is client-rendered, how to bypass Cloudflare 403s with Playwright plus residential proxies is the natural next step. For everything else, the full archive is at the blog index.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-23.