The 2026 Apify SDK guide for production scraping
Most scrapers die the same way. Someone writes a script with axios and cheerio, it works fine against 50 pages on a laptop, and then it gets pointed at 50,000 pages and falls apart within the hour. No retry logic, one IP address doing all the work, a headless browser instance that never gets closed and eats the machine’s RAM, and no record of which URLs were already handled when the process crashes at 2am.
This is the gap the Apify SDK and its underlying engine, Crawlee, are built for. Crawlee (the open source library that Apify SDK v3 evolved into back in 2022) gives you a persistent request queue, an autoscaling worker pool, session and proxy rotation, and structured storage out of the box, so you’re not rebuilding this plumbing from scratch every time. This guide is for developers and operators who already have a working scraper concept and need it to survive contact with a real target site at real volume, not people writing their first “hello world” fetch request.
By the end you’ll have a Crawlee-based crawler running locally with a rotating proxy and a persistent queue, and a clear path to either self-host it or push it to Apify’s platform as an Actor. I’d test everything locally with a small request cap before you burn proxy bandwidth or platform compute credits on a broken selector.
what you need
- Node.js 18 or newer, installed via the official installer or nvm (get it from nodejs.org)
- npm (ships with Node) or pnpm if that’s your preference
- A code editor, VS Code is fine
- A free Apify account to start; paid plans on Apify’s platform have historically started around $39/month on the Starter tier, but confirm current pricing on their site before you commit to anything
- A proxy provider with rotating residential or datacenter IPs. If you don’t have one lined up, our Decodo review covers pricing and pool size for a mid-tier residential option
- Working knowledge of JavaScript or TypeScript
- Ten minutes with the target site’s robots.txt and terms of service before you start. This isn’t legal advice, and scraping legality depends heavily on jurisdiction and what you’re collecting, so check with counsel if you’re doing this commercially against a site that restricts it
- A small budget for proxy bandwidth, residential traffic typically runs a few dollars per GB depending on provider and volume tier
step by step
1. install Node.js and scaffold the project
Install Node 18+, then scaffold a new project using the Crawlee CLI, which sets up a working template for you instead of wiring imports by hand.
npx crawlee create my-scraper
Pick the Playwright + TypeScript template when prompted, since most production targets need real browser rendering at some point even if you start with a lighter HTTP crawler.
Expected output: a new my-scraper/ folder with src/main.ts, a package.json listing crawlee and playwright as dependencies, and a storage/ directory that Crawlee uses for local state.
If it breaks: npx failing with an old Node error means your version predates 18, install it via nvm install 18 && nvm use 18. A stuck or corrupted scaffold usually means a stale npx cache, clear it with npm cache clean --force and retry.
2. wire in your proxy configuration
Open src/main.ts and add a ProxyConfiguration pointing at your provider’s rotating endpoint.
import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
proxyUrls: [
'http://user-session-1:[email protected]:7000',
'http://user-session-2:[email protected]:7000',
],
});
If you’re deploying to Apify’s platform, you can swap this for their built-in Apify Proxy instead of managing your own list, see the Apify SDK docs for the Actor.createProxyConfiguration() helper.
Expected output: running with verbose logging shows different outbound IPs per request when you check against something like an IP-echo endpoint in your requestHandler.
If it breaks: a 407 Proxy Authentication Required or immediate socket hang up almost always means malformed credentials, most providers expect the sticky session ID embedded in the username field exactly as documented, copy-paste it rather than retyping.
3. write the first requestHandler
This is where you define what happens on each page.
const crawler = new PlaywrightCrawler({
proxyConfiguration,
maxConcurrency: 5,
async requestHandler({ page, request, enqueueLinks, log }) {
const title = await page.title();
log.info(`Scraped ${request.url}: ${title}`);
await enqueueLinks({ selector: 'a.product-link' });
},
});
await crawler.run(['https://example.com/category']);
Expected output: console logs for each visited URL, and a growing storage/request_queues/default/ folder showing handled requests.
If it breaks: a selector returning null or empty results usually means the content loads after initial render. Increase the wait condition, for example await page.waitForSelector('.product-link', { timeout: 10000 }), rather than assuming the DOM is ready immediately.
4. add the request queue and control crawl depth
Crawlee’s RequestQueue persists state between runs automatically once you’re using enqueueLinks, but you need to bound it or it’ll happily crawl the entire site.
const crawler = new PlaywrightCrawler({
proxyConfiguration,
maxRequestsPerCrawl: 500,
requestHandler: async ({ page, enqueueLinks }) => {
await enqueueLinks({ globs: ['https://example.com/product/**'] });
},
});
Expected output: the crawl stops cleanly once it hits the request cap, and storage/request_queues shows a stable count of pending vs. handled requests instead of growing indefinitely.
If it breaks: a queue that keeps growing past what you expect means your glob or selector is matching navigation or filter links you didn’t intend, tighten the glob pattern or add userData depth tracking to cap recursion.
5. turn on the session pool and block detection
const crawler = new PlaywrightCrawler({
proxyConfiguration,
useSessionPool: true,
persistCookiesPerSession: true,
sessionPoolOptions: { maxPoolSize: 50 },
async failedRequestHandler({ request, session }) {
if (request.response?.statusCode === 403) session?.retire();
},
});
Expected output: over a longer run, the “reclaiming failed request” log line shows up less frequently as bad sessions get retired and replaced automatically.
If it breaks: if you’re still getting blocked consistently even with session retirement on, the pool itself is likely tainted, meaning your proxy provider’s IP range is already flagged by the target. Rotate to a different subnet or provider rather than tuning retry logic further.
6. persist output with Dataset
import { Dataset } from 'crawlee';
// inside requestHandler
await Dataset.pushData({ url: request.url, title });
Expected output: numbered JSON files appear under storage/datasets/default/, one per record pushed.
If it breaks: duplicate records across runs usually means uniqueKey isn’t being set correctly on enqueued requests, Crawlee dedupes by that key, so if your URLs contain volatile query params, normalize them before enqueueing.
7. test locally with a request cap
Set maxRequestsPerCrawl to something small like 20 and run it end to end before scaling up.
npm start
Expected output: the run finishes quickly and prints final statistics, requestsFinished and requestsFailed counts, at the end.
If it breaks: a high requestsFailed count means you should inspect the request records in storage/request_queues for the actual error, timeouts and 403/429 blocks need different fixes, don’t lump them together.
8. deploy to the Apify platform as an Actor
npm install -g apify-cli
apify login
apify push
Expected output: your Actor appears in the Apify Console, and you can trigger runs from there or via their REST API.
If it breaks: an auth error on push means your CLI session token is stale, re-run apify login and paste a fresh API token from Console > Settings > Integrations.
9. monitor and set alerts
Set up a webhook on run failure so you find out immediately instead of discovering a dead crawler three days later.
Expected output: a Slack or email notification fires within minutes of a failed or timed-out run.
If it breaks: no notification arriving usually means the webhook is scoped to the wrong event type, confirm it’s set to ACTOR.RUN.FAILED (or TIMED_OUT) specifically and not left on a default that doesn’t match your failure mode, then test it with a manual trigger.
common pitfalls
Running full headless Chromium when you don’t need JS rendering. If the data is in the initial HTML response, a CheerioCrawler is an order of magnitude cheaper on memory and compute than PlaywrightCrawler. Reach for Playwright only when the site actually requires it.
Sharing one proxy across every session. This defeats the purpose of rotation and gets the whole pool flagged fast. Bind sessions to specific proxy IPs via the session pool so a block on one session doesn’t burn your entire IP list.
Skipping the request queue’s persistence. If you’re running one-shot scripts without letting Crawlee manage state, a crash mid-run means starting over from zero, and repeated runs without dedup mean scraping the same pages twice and paying for the bandwidth twice.
Ignoring robots.txt and site terms. The robots.txt protocol is a technical signal, not a legal shield either way, but ignoring it is usually the first thing a site operator points to when they escalate. Again, not legal advice, check your specific situation.
Treating every failure identically. A 403 means the session or IP is burned and needs replacing. A timeout usually means backoff and retry on the same session is fine. Lumping both into one generic retry handler wastes proxy bandwidth on blocks that a fresh session would have avoided.
scaling this
At 10x your original volume, the architecture in this guide holds up as-is. Bump maxConcurrency, add a second proxy provider so you’re not entirely dependent on one pool, and watch memory usage if you’re running Playwright, headless browser contexts are the first thing to blow your RAM budget.
At 100x, a single queue and single machine start to strain. Shard the work, by domain, by URL range, or by category, across multiple Actor runs or worker processes, each with its own request queue so they’re not contending for the same lock. This is also where residential proxies with sticky sessions start earning their higher cost over cheap datacenter IPs, and where bandwidth spend becomes a real line item you need to track, not an afterthought.
At 1000x, throughput stops being the metric that matters and cost-per-record becomes the thing you optimize. You’re likely running a distributed queue (Apify’s platform autoscaling handles this if you stay on it, or a self-hosted Redis-backed queue with multiple worker containers if you don’t), a mixed proxy strategy across residential and mobile pools depending on target sensitivity, and real session and fingerprint management rather than just IP rotation. Read Crawlee’s own docs on autoscaled pools before you build a custom scheduler, they’ve already solved most of the scaling problems you’ll hit.
where to go next
If your target sits behind Cloudflare, pair this setup with our guide on bypassing Cloudflare 403s with Playwright and residential proxies. If you’re seeing inconsistent block rates once you scale past step 9, read debugging 429 errors and separating rate limits from proxy quality issues. And if you’re managing distinct browser identities rather than just rotating IPs, antidetectreview.org’s blog goes deeper into fingerprint-level stealth than this guide covers. For everything else we’ve written on proxies and scraping infrastructure, check the blog index.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-23.