The 2026 Botasaurus guide for production scraping

Most Python scraping stacks I’ve built over the years fall into one of two camps. Either you write raw requests calls that get blocked the moment a site turns on Cloudflare or Datadome, or you reach for full Selenium/Playwright automation and end up hand-rolling retries, proxy rotation, caching, and parallel workers yourself. Both paths work until they don’t, usually right when a client wants the scraper to run daily instead of once.

Botasaurus is a Python framework built to close that gap. It wraps a patched, anti-detect Chrome driver and a lightweight request-based scraper under the same decorator API, and it bundles the boring infrastructure work, caching, retries, parallelization, output formatting, so you’re not rebuilding it on every project. It’s open source, maintained by Omkar Cloud, and available on PyPI and GitHub.

This tutorial is for operators who already write Python and are past the “scrape one page” stage. You’re running scrapers on a schedule, feeding a database or a lead list, and you need something that survives more than a week of unattended runs. By the end you’ll have a Botasaurus scraper that handles anti-detection, proxy rotation, and caching, and you’ll know what actually changes when you scale it from a laptop job to a thousand-target production pipeline.

what you need

• Python 3.10 or newer, plus pip and a virtualenv (venv or conda, doesn’t matter which)
• Chrome installed locally, Botasaurus drives a patched Chromium under the hood
• The botasaurus package (pip install botasaurus)
• A code editor and basic familiarity with the HTML or API of whatever you’re scraping
• Residential or mobile proxies for anything beyond a handful of requests, datacenter IPs get burned fast on protected sites
• A small VPS for scheduled runs once you move past local testing, $5-10/month covers a single-worker job on providers like Hetzner or DigitalOcean
• Budget for proxy bandwidth, residential proxies typically run $3-15/GB depending on provider and volume tier, this is usually your biggest recurring cost once the scraper is live

step by step

1. install Botasaurus and scaffold a project

Create a virtualenv and install the package:

python -m venv venv
venv\Scripts\activate
pip install botasaurus

Expected output: pip resolves and installs botasaurus along with its bundled browser driver dependencies. Run pip show botasaurus to confirm the version.

If it breaks: on Windows, a common failure is Chrome not being on PATH or being an unsupported version. Update Chrome to the latest stable release and retry. If pip fails to build a dependency, upgrade pip itself first (python -m pip install --upgrade pip).

2. write your first browser scraper

Botasaurus scrapers are plain functions wrapped in a decorator. Here’s a minimal one that pulls the title and price off a product page:

from botasaurus.browser import browser, Driver

@browser
def scrape_product(driver: Driver, data):
    driver.get(data["url"])
    title = driver.select(".product-title").text
    price = driver.select(".price").text
    return {"url": data["url"], "title": title, "price": price}

if __name__ == "__main__":
    scrape_product(["https://example.com/product/1"])

Expected output: a JSON file written to an output folder in your project directory, containing the scraped title and price. Botasaurus handles the browser launch, page load waits, and result serialization for you.

If it breaks: if selectors return nothing, the page is likely rendering content via JavaScript after the initial load. Add an explicit wait or check the network tab in Chrome DevTools to find whether the data is actually loaded via a background XHR call, in which case the @request decorator (step 6) is faster and cheaper anyway.

3. turn on anti-detection settings

The whole point of Botasaurus over stock Selenium is that its driver is patched to avoid the obvious headless-Chrome fingerprints, like navigator.webdriver, worker inconsistencies, and other flags that JS-based bot checks look for. You still have decisions to make: running headless is faster and cheaper on a server, but some sites specifically fingerprint headless Chrome differently from a real desktop session, so headful with a virtual display (xvfb on Linux) is sometimes the safer default in production.

If you’re dealing with a site that runs anything beyond basic JS checks, Cloudflare’s own bot management docs are worth reading to understand what signals they score on: Cloudflare bot management. If your traffic looks too clean, too fast, or too repetitive, you’ll get scored as a bot regardless of the driver you use.

Expected output: a scrape that returns real page content instead of a challenge page or a CAPTCHA wall.

If it breaks: if you’re consistently hitting challenge pages, the fingerprint isn’t your only problem, it’s usually paired with a flagged IP. That’s what proxies are for. For a deeper dive on antidetect browser configuration specifically, antidetectreview.org’s blog covers fingerprint tooling in more depth than I will here.

4. add proxy rotation

Pass a proxy per request rather than hardcoding one IP for the whole run:

@browser(proxy=lambda data: data["proxy"])
def scrape_product(driver: Driver, data):
    driver.get(data["url"])
    return {"title": driver.select(".product-title").text}

tasks = [{"url": u, "proxy": p} for u, p in zip(urls, proxy_list)]
scrape_product(tasks)

Expected output: each task in the batch runs through a different proxy, spreading requests across IPs instead of hammering the target from one address.

If it breaks: if requests time out immediately, the proxy is likely dead or the auth format is wrong (most residential providers expect http://user:pass@host:port). Test the proxy in isolation with curl before blaming the scraper. We’ve written more on distinguishing proxy failures from fingerprint failures in diagnosing IP bans.

5. enable caching so re-runs don’t waste bandwidth

Botasaurus caches results by default keyed on function input, so re-running the same task list skips URLs you’ve already scraped successfully. This matters more than it sounds, if a job of 5,000 URLs dies at URL 3,200 from a network blip, you don’t want to re-burn proxy bandwidth re-scraping the first 3,200.

Expected output: on a second run with the same input list, completed tasks return instantly from the local cache folder instead of hitting the network again.

If it breaks: if you need fresh data on every run (prices, stock levels), you have to explicitly clear or bypass the cache, otherwise you’ll keep getting stale cached results and think the scraper is broken when it’s actually working exactly as configured.

6. switch static pages to request-based scraping

Browser automation is slow and proxy-expensive per page. For any endpoint that doesn’t need JS rendering, an API response, a server-rendered HTML page, use the @request decorator instead:

from botasaurus.request import request, Request

@request
def scrape_listing(request: Request, data):
    resp = request.get(data["url"])
    return resp.json()

Expected output: the same result, but at a fraction of the time and bandwidth cost, since there’s no browser process to spin up.

If it breaks: if the response is a challenge page instead of the expected JSON or HTML, the site is checking TLS fingerprint or headers, not just running JS, and you’ll need the browser path for that endpoint specifically.

7. parallelize and run on a schedule

Botasaurus runs tasks concurrently out of the box, controlled by how many parallel driver instances you allow. Start conservative, 2-4 concurrent browsers on a small VPS, and watch memory usage before pushing higher. Once the scraper is stable, wire it into cron (Linux) or Task Scheduler (Windows) for daily runs, or containerize it so restarts are clean.

Expected output: a batch of tasks completes in a fraction of the sequential runtime, with output files written per run.

If it breaks: if the VPS runs out of memory and the process gets OOM-killed, you’ve over-parallelized for the box size. Chrome processes are not cheap, budget roughly 300-500MB per concurrent instance and size your server accordingly.

common pitfalls

• Running headless by default without testing headful. Some sites score headless Chrome specifically, even a patched one. If you’re getting blocked consistently, test the same target headful before assuming the proxy is bad.
• Burning proxy reputation with too few IPs. Rotating through 5 proxies across 5,000 requests looks exactly like 5 IPs hammering a site. Match your proxy pool size to your request volume, not to your budget.
• Ignoring the cache and re-scraping the same URLs. This is the single most common way operators waste proxy bandwidth on Botasaurus specifically, since the caching is on by default and easy to forget about.
• Not reading the target’s robots.txt or terms of service. This isn’t legal advice, consult a lawyer for anything commercial, but the Robots Exclusion Protocol (RFC 9309) is the actual published standard sites use to declare what they don’t want scraped, and it’s worth checking before you scale a job up.
• Sizing the VPS for the scraper, not for the browser count. A “small” VPS with 1GB RAM will not survive 8 parallel Chrome instances. Test memory headroom before committing to a schedule.

scaling this

At 10x (a few hundred to a couple thousand URLs a day), Botasaurus’s built-in parallelization and caching are enough on a single small VPS. You don’t need a task queue yet, just a sane proxy rotation list and a cron job.

At 100x, you’ll want a real proxy pool with sticky sessions and rotation managed by your provider rather than a static list you maintain by hand, and you should split the scraper into workers behind a queue (Redis with RQ, or Celery) so a crash in one worker doesn’t take down the whole batch. This is also where request-based scraping (@request) starts paying for itself, converting even 30% of your browser-driven pages to request-based cuts both runtime and proxy spend meaningfully.

At 1000x, the scraper itself stops being the bottleneck, proxy cost and IP reputation management become the actual constraints. You’ll be running distributed workers across multiple machines or containers, tracking block rates per proxy subnet, and likely negotiating volume pricing directly with a proxy provider rather than paying retail per-GB rates. Data pipeline concerns (dedup, storage, downstream processing) also become a separate system from the scraper itself at this scale, not something you bolt onto the same script.

where to go next

If you’re hitting Cloudflare challenge pages specifically, read how to bypass Cloudflare 403s with Playwright and residential proxies for a deeper look at that specific failure mode. If you’re not sure whether your blocks are proxy-side or fingerprint-side, diagnosing IP bans: when it’s the proxy vs when it’s your fingerprint walks through how to tell the difference. And if you need to pick a proxy provider before scaling past step 7 above, our Decodo review covers pricing and pool quality in detail. For the full archive, browse /blog/.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-23.