← all guides

The 2026 Puppeteer guide for production scraping

Plain Puppeteer, the way most tutorials still teach it, gets flagged fast. puppeteer.launch(), navigate, scrape, done, that pattern worked in 2019. It doesn’t hold up against Cloudflare, Akamai Bot Manager, or HUMAN (formerly PerimeterX) in 2026. Chrome’s own “headless” flag became fingerprintable years ago, which is part of why Chrome 112 shipped a new headless mode that runs the full browser binary instead of a stripped-down variant. Sites still catch you on canvas fingerprints, WebGL renderer strings, timing anomalies, and IP-to-timezone mismatches, headless mode or not.

This guide is for people running scraping as an actual operation, price monitoring, lead data, SEO tracking, competitive intel, not someone running a one-off script for a weekend project. I’m assuming you’re comfortable with Node.js and can SSH into a Linux box without hand-holding.

By the end you’ll have a Puppeteer setup built for sustained runs: stealth plugin, per-session proxy rotation, consistent fingerprints, proper waits instead of arbitrary sleeps, retry logic that tells block pages apart from real errors, and enough logging that you find out a target started blocking you from a dashboard, not from three days of empty CSVs.

what you need

  • Node.js 20 LTS or newer
  • puppeteer, puppeteer-extra, and puppeteer-extra-plugin-stealth (npm packages)
  • A rotating proxy pool. residential or ISP proxies, not datacenter, if the target does any bot scoring. budget roughly $4 to $15 per GB for residential (Decodo, Bright Data, Oxylabs are the usual names) or $1 to $3 per IP per month for static ISP proxies
  • A queue, even a simple one. BullMQ plus Redis works fine for a single box, doesn’t need to be fancier than that at low volume
  • A Linux VPS or dedicated box. plan for roughly 300 to 500MB RAM per concurrent browser context, so a 4GB box realistically runs 6 to 8 contexts before you start seeing OOM kills
  • Somewhere to store output: Postgres, SQLite, or flat files depending on volume
  • A webhook or Slack channel for alerts, you want to know about a block spike before your data does

step by step

1. install puppeteer and the stealth plugin

npm install puppeteer puppeteer-extra puppeteer-extra-plugin-stealth

Expected output: package installs clean, no peer dependency warnings about Chromium version mismatches. puppeteer pulls down its own pinned Chromium build automatically.

If it breaks: on a headless Linux box the install often fails because Chromium’s system dependencies (libnss3, libatk-bridge2.0-0, etc.) aren’t present. Install them via your distro’s package manager before retrying, not by passing --ignore-scripts.

2. launch chrome with production flags

const puppeteerExtra = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteerExtra.use(StealthPlugin());

const browser = await puppeteerExtra.launch({
  headless: 'new',
  args: [
    '--no-sandbox',
    '--disable-setuid-sandbox',
    '--disable-dev-shm-usage',
    '--disable-blink-features=AutomationControlled',
  ],
});

Expected output: a browser process starts, navigator.webdriver reads false on any page you open, and the stealth plugin patches the usual automation tells (plugins array, chrome.runtime, permissions API).

If it breaks: --disable-dev-shm-usage matters more than it looks. Docker containers default to a tiny /dev/shm, and Chrome crashes silently mid-render without it. If contexts die with no error, check that flag first.

3. wire proxy rotation into each browser context

Don’t set the proxy at browser launch and reuse one browser for everything, that ties every page to the same IP. Use a fresh incognito browser context per proxy instead:

const context = await browser.createBrowserContext({
  proxyServer: `http://${proxyHost}:${proxyPort}`,
});
const page = await context.newPage();
await page.authenticate({ username: proxyUser, password: proxyPass });

Expected output: each context routes through a distinct IP, verifiable by hitting https://ip.decodo.com/json or similar before the real navigation.

If it breaks: page.authenticate() has to be called before page.goto(), and only once per page. Calling it twice, or after the first navigation, silently drops the credentials and you get a 407 further down the chain.

4. normalize the fingerprint per session

Match viewport, timezone, and Accept-Language to the proxy’s geolocation. A US residential IP paired with an Asia/Singapore timezone and a zh-CN locale header is the single most common flag I see, and it’s an easy one to fix:

await page.emulateTimezone('America/Chicago');
await page.setExtraHTTPHeaders({ 'Accept-Language': 'en-US,en;q=0.9' });
await page.setViewport({ width: 1366, height: 768 });

Expected output: fingerprint checkers (BrowserLeaks, CreepJS) show a consistent geo/timezone/language story matching the exit IP.

If it breaks: check the MDN User-Agent header reference if you’re overriding UA manually, an outdated Chrome version string in your UA while the real Chromium build reports a newer one in navigator.userAgentData is a mismatch bots detect for free.

5. wait for real signals, not fixed timeouts

await page.goto(url, { waitUntil: 'domcontentloaded' });
await page.waitForSelector('.product-price', { timeout: 15000 });

Expected output: the scraper waits exactly as long as the page needs, no more, no less. See the Puppeteer API docs for the full set of wait conditions.

If it breaks: waitUntil: 'networkidle0' looks safe but hangs forever on pages with polling analytics scripts that never go idle. Prefer waiting on the specific element or API response you actually need.

6. add retry and backoff with error classification

Not every failure is the same failure. A 429 means slow down. A 403 or a CAPTCHA page means the proxy or fingerprint got burned. A timeout might just be a slow proxy.

async function fetchWithRetry(fn, retries = 3) {
  for (let i = 0; i < retries; i++) {
    try {
      return await fn();
    } catch (err) {
      if (err.message.includes('403') || err.message.includes('429')) {
        await new Promise(r => setTimeout(r, 2000 * (i + 1)));
        continue;
      }
      if (i === retries - 1) throw err;
    }
  }
}

Expected output: transient failures recover automatically, block signals get logged separately from network noise.

If it breaks: if every retry hits the same proxy, you’re just hammering a burned IP three times instead of once. Rotate the proxy inside the retry loop, not just the request.

7. persist and reuse sessions

Save cookies and localStorage per proxy IP so you’re not presenting a cold, cookie-less browser on every single run, that pattern itself is a signal.

const cookies = await page.cookies();
fs.writeFileSync(`sessions/${proxyId}.json`, JSON.stringify(cookies));

Expected output: return visits from the same IP carry forward session state, closer to how a real user’s browser behaves.

If it breaks: cookies expire. If a target suddenly 403s a previously-working session, wipe that session file and let it re-establish cold rather than debugging a dead cookie for an hour.

8. extract, validate, and store

Validate scraped fields before writing them anywhere. A silently changed CSS selector will happily write null or an empty string into your database for days if nothing checks it.

const price = await page.$eval('.product-price', el => el.textContent.trim());
if (!/^\$?\d+(\.\d{2})?$/.test(price)) throw new Error('price selector likely broken');

Expected output: bad extractions throw immediately instead of polluting your dataset.

If it breaks: if validation starts failing across the board on a target that hasn’t changed layout, suspect a block page rendering in place of real content, not a selector bug.

9. log, monitor, alert

Track block rate per target and per proxy pool, not just overall success/failure. A pool going from 2% to 40% blocked over an hour is the earliest warning you’ll get.

Expected output: a dashboard or a simple cron job that posts to Slack when block rate crosses a threshold you set, say 15%.

If it breaks: if block rate spikes uniformly across every proxy pool at once, that’s usually the target changing its detection, not your infrastructure. Check their site for a new WAF vendor or a Cloudflare bot-management rollout, Cloudflare documents how its bot scoring works here.

10. containerize and deploy

FROM node:20-slim
RUN apt-get update && apt-get install -y chromium libnss3 libatk-bridge2.0-0
WORKDIR /app
COPY . .
RUN npm ci --omit=dev
CMD ["node", "worker.js"]

Expected output: a reproducible container that runs identically on your box and wherever you deploy it, pulled by your queue worker.

If it breaks: memory limits set too low in your container orchestrator (Docker’s --memory flag, or your host’s cgroup limits) will OOM-kill Chrome mid-scrape with no clear error in your app logs, check dmesg on the host if a worker dies with no stack trace.

common pitfalls

  • IP and fingerprint mismatch. A residential IP from Jakarta paired with a Europe/London timezone and English-only headers is the most common self-inflicted block I see. Fix the fingerprint to match the proxy, not the other way around.
  • Overloading a small box. Running 20 concurrent contexts on a 4GB VPS looks fine for the first ten minutes, then Chrome starts getting OOM-killed mid-render and your error logs fill with vague, unhelpful crashes. Size concurrency to available RAM, not to how fast you want the run to finish.
  • Robotic pacing. Scraping every page at exactly the same interval, with identical click and scroll patterns, is a behavioral signal on its own even with a clean proxy and fingerprint. Add jitter to your request timing.
  • Wrong proxy type for the target. Datacenter proxies are cheap and fine for sites with no real bot management. They get instantly flagged on anything running Cloudflare, DataDome, or HUMAN. Match proxy type to target defenses instead of defaulting to whatever’s cheapest.
  • Scraping data you shouldn’t. Scraping publicly accessible pages is generally different from scraping data behind a login wall or bypassing paywalls, and different jurisdictions treat this differently. This isn’t legal advice, check a target’s terms of service and, if the data involves personal information, consult someone qualified before building a pipeline around it.

scaling this

At 10x (a handful of concurrent scrapers), a single VPS with a queue and a few proxy pools is enough. Manual monitoring, checking a dashboard once a day, is fine.

At 100x, one box stops being enough RAM and CPU-wise. You’re running multiple worker boxes pulling from the same Redis queue, and your proxy budget becomes a real line item, expect to be managing several proxy pools split by target or by geography rather than one pool for everything. Monitoring needs to be automated by this point, not eyeballed.

At 1000x, this becomes infrastructure, not a script. Container orchestration (Docker Swarm or Kubernetes, doesn’t need to be fancy) replaces manually SSH’ing into boxes, proxy spend becomes large enough that IP hygiene (tracking which subnets get burned, rotating pools out) needs its own process, and it’s worth evaluating whether some targets are better served by a paid data API than by scraping at all, since API costs sometimes undercut proxy costs once you factor in engineering time spent fighting blocks. If you’re also managing many separate browser identities at this scale rather than just proxies, the fingerprint isolation techniques covered on antidetectreview.org’s blog are worth a look, that’s a different but related problem from proxy rotation.

where to go next

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-24.

proxies
Need proxies that survive the block wall?

Singapore Mobile Proxy runs real 4G/5G mobile IPs on rotating SIMs — the carrier-grade addresses most of these targets still trust.

see plans →
read on
More scraping guides

The rest of the field manual: target-site playbooks, library walkthroughs, provider reviews, and anti-bot troubleshooting.

browse all guides →