The 2026 Crawlee guide for production scraping

Most scrapers die the same way. Someone writes a quick script with requests or a bare Playwright loop, it works fine against ten test pages, then it hits production and falls apart: IP bans within an hour, a browser process that leaks memory until the box swaps to death, no retry logic so one timeout kills the whole run, and a selector change that goes unnoticed for a week because nothing was watching. Crawlee, Apify’s open source scraping library, exists specifically to handle that boring infrastructure layer so you can spend your time on the part that’s actually specific to your target site.

This is for people who’ve outgrown a single-file script and need something that runs unattended: proxy rotation that doesn’t get flagged in the first ten minutes, a session pool that behaves like a real browsing session instead of a bot hopping IPs, automatic retries with backoff, and a request queue that survives a crash without losing progress.

By the end you’ll have a working PlaywrightCrawler wired to rotating proxies, a session pool, autoscaled concurrency, and a dataset export, plus the operational habits I actually use when running these against real targets, including where they tend to break and what changes as you scale from a few hundred pages a day to a few hundred thousand.

what you need

Node.js 20.x LTS (Crawlee supports 18+, but I run 20 across everything since it’s the current LTS)
npm or pnpm
Crawlee itself (npm install crawlee) plus Playwright’s browser binaries (npx playwright install --with-deps chromium)
a proxy provider with per-request or per-session rotation. residential proxies run roughly $3 to $15 per GB depending on provider and country mix; datacenter proxies are cheaper, often $1 to $3 per GB or a flat monthly per-IP fee. I’ve compared a few of these head to head if you need a starting point, see /blog/decodo-review-2026-honest-pros-cons-and-pricing/
a target site you’re actually allowed to scrape, and a read of its robots.txt before you start. robotstxt.org has the spec if you’re unfamiliar with how directives work
somewhere to land the output: local JSON/CSV for a prototype, Postgres or S3 for anything that needs to survive a server restart
a few hours for the initial setup, after that it mostly runs itself

step by step

1. scaffold the project

Run:

npx crawlee create my-crawler

Pick the “Playwright + TypeScript” template when prompted. I default to Playwright over Cheerio for anything with client-rendered content, and switch to CheerioCrawler later if the target turns out to be static HTML and I want the speed.

Expected output: a new my-crawler/ directory with src/main.ts, src/routes.ts, package.json, and a storage/ folder Crawlee uses for local request queues and datasets.

If it breaks: npx pulling an old cached version usually means an outdated npm. Check npm -v, anything below 9 should be upgraded (npm install -g npm@latest) before retrying.

2. install and configure a proxy

npm install crawlee playwright

import { ProxyConfiguration } from 'crawlee';

const proxyConfiguration = new ProxyConfiguration({
    proxyUrls: [
        'http://user:[email protected]:10000',
        'http://user:[email protected]:10001',
    ],
});

If your provider gives you a single rotating gateway instead of a list of IPs (most residential providers work this way), just pass that one URL and rotation happens on their end per connection.

Expected output: nothing visible yet, this just prepares the config object the crawler pulls from.

If it breaks: a 407 on the first request almost always means the auth format is wrong. Check whether your provider expects user:pass@host:port or a separate header, this varies by vendor and their docs are the source of truth here, not guesswork.

3. build the crawler with a session pool

import { PlaywrightCrawler } from 'crawlee';

const crawler = new PlaywrightCrawler({
    proxyConfiguration,
    useSessionPool: true,
    persistCookiesPerSession: true,
    sessionPoolOptions: {
        maxPoolSize: 100,
    },
    maxRequestRetries: 5,
    requestHandler: async ({ page, request }) => {
        const title = await page.title();
        console.log(`${request.url}: ${title}`);
        // your extraction logic goes here
    },
    failedRequestHandler: async ({ request, session }) => {
        session?.markBad();
        console.log(`request ${request.url} failed too many times`);
    },
});

await crawler.run(['https://example.com']);

The session pool is the part people skip and then wonder why they’re getting banned. Each session pairs a proxy IP with its own cookie jar, so requests going out over IP A aren’t carrying cookies from a session that started on IP B, which is one of the fastest ways to get flagged. Crawlee’s own proxy management guide covers the mechanics in more depth than I can here.

Expected output: console lines with page titles as the crawler works through the queue.

If it breaks: check failedRequestHandler logs first. It’s almost always a proxy auth issue, or the target returning a 403 that never reaches requestHandler at all.

4. tune concurrency

const crawler = new PlaywrightCrawler({
    // ...previous config
    minConcurrency: 5,
    maxConcurrency: 50,
    autoscaledPoolOptions: {
        desiredConcurrencyRatio: 0.9,
    },
});

Crawlee’s autoscaler watches CPU, memory, and event loop lag and adjusts concurrency live, so you rarely need to hand-tune this once the min/max bounds are sane.

Expected output: concurrency ramps up gradually in the logs instead of jumping straight to max.

If it breaks: if memory climbs steadily and never plateaus, you likely have unclosed browser contexts. Check you’re not overriding the default page lifecycle handling somewhere.

5. handle blocks and retries deliberately

requestHandler: async ({ page, request, session, response }) => {
    if (response?.status() === 403 || response?.status() === 429) {
        session?.markBad();
        throw new Error(`blocked: ${response.status()}`);
    }
    // extraction logic
},

Throwing inside requestHandler triggers Crawlee’s built-in retry with a fresh session, and if you’ve configured multiple proxy URLs, often a different IP too.

Expected output: blocked requests get requeued automatically instead of silently disappearing from your dataset.

If it breaks: if the same session keeps getting reused right after a block, confirm useSessionPool: true is actually set. It’s easy to leave off in a stripped-down config and not notice until your ban rate climbs.

6. persist the output

import { Dataset } from 'crawlee';

requestHandler: async ({ request, page }) => {
    const data = await page.evaluate(() => ({
        title: document.title,
        url: location.href,
    }));
    await Dataset.pushData(data);
},

Crawlee writes to storage/datasets/default locally by default. For production, export to Postgres or S3 as part of the run, or push directly from the handler.

Expected output: JSON files accumulating under storage/. Dataset.exportToJSON('output') bundles it into a single file if you want that instead.

If it breaks: if storage isn’t updating between runs, check you haven’t set CRAWLEE_PURGE_ON_START=true in an environment that also restarts the process between crawls.

7. add logging and stats

import { log, LogLevel } from 'crawlee';
log.setLevel(LogLevel.INFO);

Crawlee logs request counts, retry counts, and session stats on an interval by default, which is enough to eyeball whether a run is healthy without wiring up a full dashboard on day one.

Expected output: periodic summary lines like “Crawled 1200/5000 pages, failed 3.”

If it breaks: if you need alerts rather than logs, pipe stdout into whatever you already use. I run these through a simple log watcher on the same box that also tracks my proxy pool health.

8. deploy it

For a single scheduled crawl, a Docker container on a small VPS plus cron or pm2 is enough:

docker build -t my-crawler .
docker run --rm my-crawler

For anything you want running continuously with less ops overhead, Apify’s own hosting platform (the company behind Crawlee) will run the same code without you managing servers, at a cost, obviously.

Expected output: a container that exits 0 on a clean run, non-zero on an unhandled crash.

If it breaks: if the container works locally but fails in CI, it’s almost always missing Playwright browser binaries. Add RUN npx playwright install --with-deps chromium to the Dockerfile.

common pitfalls

Rotating IPs without a session strategy. Rotating on every single request looks stealthy but actually reads as very unnatural, real users keep one IP for an entire browsing session. Pair rotation with Crawlee’s session pool, not instead of it.

Ignoring the fingerprint layer. Crawlee handles the network-level rotation, but a stock Chromium instance still leaks canvas, WebGL, and timezone-versus-IP mismatches that give away automation regardless of how clean your proxy is. If you’re up against real bot detection like Cloudflare’s bot score system, PerimeterX, or DataDome, Crawlee alone won’t carry you, you need a proper fingerprint layer on top. I keep a running comparison of anti-detect browsers at antidetectreview.org/blog/ that’s worth a read before you assume proxies solve everything.

Not checking robots.txt or the target’s terms before scaling up. This isn’t legal advice, courts have gone both ways on scraping public data depending on jurisdiction and what’s being accessed, so check the terms and talk to a lawyer if there’s real exposure. Operationally, ignoring robots.txt also just gets you banned faster since most anti-bot systems weight it as a signal.

Unbounded concurrency on day one. Jumping straight to maxConcurrency: 200 against a target you haven’t profiled is how you burn through a proxy plan’s bandwidth and trip every rate limit in one afternoon. Start low, watch the ban rate, scale from there.

Selectors with no fallback. Sites redesign without warning. A crawler that hardcodes one CSS selector with no fallback and no alerting will fail silently for days before anyone notices the dataset stopped growing.

scaling this

10x (a few hundred pages a day to a few thousand): a single VPS handles this fine. Bump maxConcurrency, add a couple more proxy endpoints so you’re not hammering one gateway, and keep session pool size proportional to concurrency.

100x: now you’re paying real money for proxy bandwidth, and cost per successful page matters more than cost per GB. This is usually where I move the request queue to shared storage so multiple worker processes or machines can pull from the same queue without duplicating work. Ban rate per proxy IP becomes a metric worth graphing, not something you eyeball in logs.

1000x: single-machine headless browsing stops being viable on cost and speed. At this scale I split traffic: CheerioCrawler (HTTP plus HTML parsing, no browser) for anything that doesn’t need JS rendering, Playwright reserved only for pages that actually require it. Proxy selection needs to be smarter than round robin, weighting by recent success rate per IP or subnet. You’re also running across multiple machines or containers by now, so centralized logging and a dead man’s switch, an alert if no new dataset rows show up in X minutes, stop being optional.

where to go next

If Cloudflare or similar bot management is what’s actually blocking you rather than raw rate limits, read how to bypass Cloudflare 403s with Playwright plus residential proxies next, it goes deeper into the browser-side fingerprint work Crawlee doesn’t handle for you.

If you’re seeing intermittent 429s rather than hard blocks, debugging 429 errors: rate limits, proxy quality, and behavioural patterns walks through telling a proxy problem apart from a request-pattern problem.

For the session and cookie mechanics I only touched on in step 3, cookie and session handling at scale across rotating proxies covers it properly. And for more build-outs like this one, browse the rest of the writeups at /blog/.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-23.