The 2026 Selenium guide for production scraping
I’ve been running Selenium in production since before Selenium 4 shipped the W3C WebDriver protocol by default, and the gap between “Selenium works on my laptop” and “Selenium survives a week of unattended runs” is still where most scraping projects die. A script that opens Chrome, clicks around, and closes cleanly in a demo will leak memory, get IP banned, or silently hang on a stale driver binary the first night nobody’s watching it.
This guide is for operators, not QA engineers. If you’re scraping catalog data, monitoring competitor pricing, or pulling structured data behind a login wall at meaningful volume, and you’ve already outgrown a five-line script, this is written for you. I run Selenium alongside Playwright across scraping pipelines for proxyscraping.org’s own research, and most of what breaks in production isn’t Selenium itself, it’s the plumbing around it: proxy auth, driver version drift, and detection surface nobody budgeted time for.
By the end you’ll have a Chrome setup using Selenium Manager (no manual chromedriver downloads), proxy rotation that actually authenticates, a reduced detection footprint, and a path to running this in Docker or Selenium Grid once one box stops being enough.
what you need
- Python 3.11 or newer (this guide uses Python; the same ideas apply in Selenium’s Java, Node, or C# bindings)
- selenium 4.24 or newer via pip. Selenium Manager, bundled since 4.6, auto-resolves the right chromedriver, so stop pinning driver binaries by hand
- Google Chrome or a Chrome for Testing build installed on the host
- a proxy plan with authenticated, sticky sessions, not a free list. Residential or ISP plans from providers like Decodo, Bright Data, or IPRoyal run roughly $4 to $15 per GB depending on target country and session type as of mid-2026
- a VPS or dedicated box to run this unattended. A 4 vCPU / 8GB box from Hetzner or DigitalOcean in the $20 to $40/month range covers single-digit concurrency
- Docker, if you plan to run Selenium Grid or
standalone-chromecontainers instead of a bare install - a captcha-solving budget (2Captcha or CapSolver) if targets sit behind reCAPTCHA v3 or hCaptcha
- somewhere to dump run logs and crash traces. Even a rotated log file beats nothing
step by step
1. install selenium and let Selenium Manager handle the driver
pip install "selenium>=4.24.0"
Don’t install webdriver-manager or download chromedriver separately. Since Selenium 4.6, Selenium Manager resolves and caches the matching driver for whatever Chrome version is on the box, automatically, the first time you instantiate a driver.
Expected output: your first webdriver.Chrome() call downloads a driver binary to ~/.cache/selenium (or %LOCALAPPDATA%\selenium on Windows) and opens a browser window in a few seconds.
If it breaks: a SessionNotCreatedException mentioning a version mismatch means Chrome and the cached driver disagree, usually because Chrome auto-updated. Delete the Selenium Manager cache directory and let it re-resolve on the next run.
2. configure Chrome for headless, unattended use
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
opts = Options()
opts.add_argument("--headless=new")
opts.add_argument("--no-sandbox")
opts.add_argument("--disable-dev-shm-usage")
opts.add_argument("--disable-gpu")
opts.add_argument("--window-size=1366,900")
opts.page_load_strategy = "eager"
driver = webdriver.Chrome(options=opts)
Use --headless=new, not the old --headless flag. The new headless mode, added in Chrome 109, runs the same rendering engine as headed Chrome instead of a stripped-down variant, which matters for rendering fidelity and because the old mode has a distinct, easily fingerprinted signature. Full flag reference is in Selenium’s WebDriver options docs.
Expected output: a headless Chrome process that renders JS-heavy pages the same as a visible browser would, confirmable via ps aux | grep chrome on Linux.
If it breaks: crashes on a Docker host almost always trace back to --no-sandbox and --disable-dev-shm-usage being missing, since containers default to a tiny /dev/shm.
3. wire up authenticated proxy rotation
Chrome strips inline username:password credentials from proxy URLs, so Options.proxy alone won’t authenticate. The fix is a small unpacked extension that answers Chrome’s proxy auth prompt via the webRequest API:
import zipfile, string
manifest = """
{
"version": "1.0.0",
"manifest_version": 2,
"name": "proxy auth",
"permissions": ["proxy", "webRequest", "webRequestBlocking", "<all_urls>"],
"background": {"scripts": ["bg.js"]}
}
"""
bg_js = string.Template("""
chrome.webRequest.onAuthRequired.addListener(
() => ({authCredentials: {username: "$user", password: "$pw"}}),
{urls: ["<all_urls>"]}, ["blocking"]
);
""").substitute(user="YOUR_PROXY_USER", pw="YOUR_PROXY_PASS")
with zipfile.ZipFile("proxy_auth.zip", "w") as zp:
zp.writestr("manifest.json", manifest)
zp.writestr("bg.js", bg_js)
opts.add_extension("proxy_auth.zip")
opts.add_argument("--proxy-server=http://gate.yourproxyvendor.com:10001")
Expected output: driver.get("https://ifconfig.me") returns your proxy pool’s IP, not your host’s, with no auth popup blocking the page load.
If it breaks: a stuck auth popup usually means the extension registered too late, or your Chrome build is rejecting manifest v2. If so, skip the extension and IP-whitelist the proxy on your vendor’s dashboard instead.
4. cut down the automation fingerprint
opts.add_argument("--disable-blink-features=AutomationControlled")
opts.add_experimental_option("excludeSwitches", ["enable-automation"])
opts.add_experimental_option("useAutomationExtension", False)
This doesn’t make Selenium undetectable, nothing does reliably in 2026, but it removes the cheapest signals: the navigator.webdriver flag and the “Chrome is being controlled by automated test software” infobar. Sites doing serious bot detection go further, checking CDP timing artifacts and canvas or WebGL fingerprints, which is a different problem than a Chrome flag can solve. antidetectreview.org tracks which fingerprinting layers various anti-detect tools actually defeat, worth a read before you assume config flags alone handle this.
Expected output: driver.execute_script("return navigator.webdriver") returns false or undefined instead of true.
If it breaks: if a site still flags you instantly, the tell usually isn’t Selenium, it’s a residential proxy IP with a bad reputation, or a fingerprint that doesn’t match the declared user agent. Check the proxy before you rewrite browser flags.
5. set sane timeouts and retry logic
driver.set_page_load_timeout(25)
driver.implicitly_wait(0) # use explicit waits instead
Wrap navigation in a retry loop with a hard cap, two or three attempts, not infinite. A page that hangs on driver.get() for 25 seconds and then fails cleanly is recoverable; a driver with no timeout that hangs forever will quietly stall an entire worker.
Expected output: a hung page load raises TimeoutException at 25 seconds instead of blocking indefinitely.
If it breaks: if timeouts fire constantly on pages that load fine in a real browser, your proxy’s median latency is probably too high for the target. Check with a plain curl -x timing test outside Selenium first.
6. quit the driver properly, every time
try:
run_scrape(driver)
finally:
driver.quit()
driver.close() closes a tab. driver.quit() kills the browser process and the driver session. Mixing these up, or letting an exception skip cleanup, is the single biggest cause of servers slowly filling up with zombie chrome processes until the box runs out of memory.
Expected output: ps aux | grep chrome shows zero lingering processes after a run finishes or crashes.
If it breaks: if processes still pile up despite finally: driver.quit(), check for SIGKILL‘d workers, an OOM killer or a supervisor timeout, that never reach the finally block at all. Add a cron job that force-kills orphaned chrome processes older than N minutes as a backstop.
7. containerize it for repeatability
# docker-compose.yml
services:
selenium:
image: selenium/standalone-chrome:127.0
shm_size: 2gb
ports:
- "4444:4444"
Point your script at webdriver.Remote(command_executor="http://localhost:4444", options=opts) instead of webdriver.Chrome(). This pins the Chrome and driver version to the image tag, so a host-level Chrome auto-update can’t break your pipeline overnight, and it’s the same setup you’ll scale out with Selenium Grid later.
Expected output: docker compose up -d, then a Selenium session connects to localhost:4444 and runs identically to the local install.
If it breaks: session not created errors from the container usually mean shm_size is too small. Chrome needs real shared memory, not Docker’s default 64MB.
8. run it as a supervised service, not a foreground script
# /etc/systemd/system/scraper.service
[Unit]
Description=selenium scraper worker
After=network.target
[Service]
ExecStart=/usr/bin/python3 /opt/scraper/run.py
Restart=on-failure
RestartSec=15
User=scraper
[Install]
WantedBy=multi-user.target
Expected output: systemctl status scraper shows an active, auto-restarting process that survives an SSH disconnect.
If it breaks: a restart loop, Restart=on-failure firing every 15 seconds, usually means the crash happens at startup. Check journalctl -u scraper -n 50 before assuming it’s a Selenium problem at all.
common pitfalls
Reusing one proxy session across thousands of requests. A sticky session that never rotates burns its reputation on the target site, and then you’re debugging “Selenium is broken” when it’s actually an IP problem. See diagnosing IP bans versus fingerprint bans for how to tell which one you’re hitting.
Running the old --headless flag instead of --headless=new. It’s a smaller, more obviously non-standard rendering path and one of the easiest bot-detection tells to check for. It’s a one-line fix.
No cookie or session hygiene between runs. Wiping cookies on every launch means re-solving logins and captchas repeatedly instead of reusing a warmed-up session. If you’re scraping anything authenticated, read up on session handling across rotating proxies before you architect this.
Letting exceptions skip driver.quit(). Covered above, but worth repeating: it’s the pitfall that takes down a whole box weeks after launch, not on day one, which is exactly why it doesn’t get caught in testing.
Treating every block as a code bug. A 403 or a captcha wall is frequently a proxy quality or fingerprint issue, not a selector that changed. Check the proxy and the fingerprint before you start rewriting scraper logic.
scaling this
At 10x, a single script on one box running a ThreadPoolExecutor with 5 to 10 concurrent driver instances is fine. One proxy provider, one log file, done.
At 100x, one box stops being enough. Move to Selenium Grid or several standalone-chrome containers spread across two or three hosts, feed them a URL queue through Redis or a simple database table instead of a hardcoded list, and expect your proxy bill to become the real bottleneck before your compute does. You’ll also want country and ASN diversity in your proxy pool, not just more IPs from the same subnet, since detection systems increasingly flag by subnet clustering rather than single IPs.
At 1000x, seriously reconsider whether every page needs a full browser. The economical move at that volume is splitting the pipeline: raw HTTP requests, with proper header and TLS fingerprint hygiene, for static or API-backed pages, and Selenium or Playwright reserved only for pages that genuinely require JS execution. Orchestrate with Kubernetes or a job queue like Celery, run dedicated proxy infrastructure instead of a shared vendor pool, and budget real engineering time for session and cookie state, because at this scale that state machine, not the browser automation, is what actually breaks.
where to go next
If Selenium’s detection ceiling frustrates you, Playwright with residential proxies against Cloudflare covers the same production concerns with a different automation stack that some operators find easier to keep undetected. Pair whichever one you run with a proper read on rate limits, proxy quality, and 429s, since most “Selenium is slow” complaints turn out to be throttling. Browse the full article index for the rest of the proxy and scraping infrastructure writeups.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-24.