The 2026 Requests guide for production scraping
Most scrapers start the same way: a for loop, a call to requests.get(), and a print statement. That works for 50 URLs against a friendly API. It falls apart the first time you run it unattended overnight against a site that rate-limits you, rotates its WAF rules, or just times out on a bad connection. I’ve had scrapers die at 3am because one proxy in the pool hung forever with no timeout set, and the whole run just sat there until I woke up and killed it.
This is for anyone running Python’s Requests library in production, not a notebook. You’re pulling product listings, job postings, pricing data, whatever, at a volume where a single unhandled exception or a missing retry policy costs you real time and real proxy spend. I’m not covering headless browsers here, Requests is still the right tool when the target doesn’t need JavaScript rendering, and it’s lighter and cheaper to run at scale than Playwright or Selenium.
By the end you’ll have a Requests-based scraper with connection reuse, retry and backoff logic, proxy rotation, concurrency, and logging that actually tells you what broke. None of this is exotic. It’s the stuff that separates a script that works on your laptop from one you can leave running for a week.
what you need
- Python 3.10 or newer, installed via python.org or your package manager
pip install requests(this guide assumes requests 2.32.x, the current 2.x line as of mid-2026)- a proxy pool from a provider like Decodo, Smartproxy, or SOAX (I’ve reviewed all three on this site) — residential IPs are usually billed per GB, datacenter IPs per port or per IP, budget accordingly since proxy spend is your main recurring cost once you’re past a hobby run
- a virtualenv or
venvso your scraping dependencies don’t collide with anything else on the box - a test target that won’t get you in trouble: httpbin.org is fine for checking headers and your outbound IP before you point anything at a real site
- basic comfort with a terminal, and optionally Redis if you plan to scale past a single process later
step by step
1. set up an isolated environment
Create a venv and install requests inside it, not globally.
python -m venv .venv
source .venv/bin/activate # .venv\Scripts\activate on Windows
pip install requests
Expected output: pip show requests returns a version string like Version: 2.32.3. If it doesn’t, your venv isn’t activated, check the shell prompt for the (.venv) prefix.
If it breaks: pip install failing with a permissions error usually means you’re not actually inside the venv and pip is trying to write to system site-packages. Reactivate and retry.
2. use a Session, not bare requests.get calls
A requests.Session() reuses the underlying TCP connection (via urllib3’s connection pooling) instead of opening a new one per request. Against a site you’re hitting hundreds of times, this alone cuts noticeable latency.
import requests
session = requests.Session()
session.headers.update({"Accept-Language": "en-US,en;q=0.9"})
resp = session.get("https://httpbin.org/get", timeout=10)
print(resp.status_code, resp.json()["headers"]["Accept-Language"])
Expected output: 200 and your header echoed back. Cookies set by the server also persist automatically across calls on the same session, which matters for anything gated behind a login or a CSRF token.
If it breaks: if you’re spinning up a new Session() per request inside a loop, you’ve defeated the whole point. Instantiate it once, outside the loop.
3. configure timeouts and retries with an HTTPAdapter
Requests has no default timeout, a hung connection will block forever. It also doesn’t retry failed requests unless you wire that up yourself using urllib3’s Retry class mounted on an adapter.
from requests.adapters import HTTPAdapter
from urllib3.util import Retry
retry_strategy = Retry(
total=3,
backoff_factor=1.5,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["GET", "POST"],
)
adapter = HTTPAdapter(max_retries=retry_strategy)
session.mount("https://", adapter)
session.mount("http://", adapter)
resp = session.get("https://httpbin.org/status/503", timeout=(5, 15))
The tuple (5, 15) sets a 5-second connect timeout and 15-second read timeout, separately. Expected output: the adapter automatically retries three times with exponential backoff before finally raising or returning whatever the last attempt got.
If it breaks: if requests still hang past your timeout, check you didn’t set timeout=None somewhere upstream, or that a proxy in front of the connection isn’t the one actually stalling (proxies can accept a TCP connection and then just sit on it).
4. rotate proxies through the session
Most proxy providers give you either a rotating gateway endpoint (one hostname, IP changes per request server-side) or a list of sticky IPs you rotate yourself. Either way, it plugs into session.proxies.
proxy_url = "http://username:[email protected]:7000"
session.proxies = {"http": proxy_url, "https": proxy_url}
resp = session.get("https://httpbin.org/ip", timeout=10)
print(resp.json())
Expected output: an IP address that isn’t your own. Run it a few times if you’re on a rotating gateway, the IP should change.
If it breaks: a 407 Proxy Authentication Required almost always means your password has a special character (@, :, #) that needs URL-encoding. Run it through urllib.parse.quote() before building the proxy URL. If you’re getting connection refused entirely, confirm the port matches what the provider gave you, HTTP and SOCKS5 ports are usually different.
5. set realistic headers
A bare Requests call sends User-Agent: python-requests/2.32.3 by default, which is an instant tell to any WAF worth its salt. Set a browser-like user agent and a plausible header set.
session.headers.update({
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
})
Expected output: fewer immediate 403s on sites doing basic bot filtering.
If it breaks: if you’re still getting blocked with correct headers and a clean residential IP, the target is probably fingerprinting your TLS handshake (JA3/JA4), not your headers. Requests’ TLS signature via urllib3/OpenSSL doesn’t match a real Chrome or Firefox stack, and no amount of header spoofing fixes that. That’s a different problem than this guide covers, worth reading up on if you’re pairing scraping with fingerprint-sensitive targets, see the browser fingerprinting write-ups at antidetectreview.org/blog.
6. handle 429s with real backoff, not a fixed sleep
The Retry config from step 3 handles some of this, but you should also read the Retry-After header when present rather than guessing at a sleep duration. Per MDN’s HTTP 429 reference, servers are allowed to specify it in seconds or as an HTTP date.
import time
resp = session.get(url, timeout=10)
if resp.status_code == 429:
wait = int(resp.headers.get("Retry-After", 30))
time.sleep(wait)
resp = session.get(url, timeout=10)
Expected output: a clean pause and successful retry instead of a wall of 429s while you hammer a site that’s already told you to slow down.
If it breaks: if you’re still getting 429s after respecting Retry-After, the block is probably IP-based, not request-rate based, and you need to rotate to a different proxy, not just wait longer on the same one.
7. add concurrency with a thread pool
Requests is synchronous. To get real throughput you need threads (or async via httpx, out of scope here). concurrent.futures.ThreadPoolExecutor is the standard-library option, documented at docs.python.org.
from concurrent.futures import ThreadPoolExecutor, as_completed
urls = [f"https://httpbin.org/anything?id={i}" for i in range(100)]
def fetch(url):
return session.get(url, timeout=10).status_code
with ThreadPoolExecutor(max_workers=10) as pool:
futures = {pool.submit(fetch, u): u for u in urls}
for future in as_completed(futures):
print(futures[future], future.result())
Expected output: 100 URLs completed in a fraction of the time a serial loop would take, roughly bounded by max_workers.
If it breaks: if your proxy provider starts throttling or banning across the whole pool, you’ve likely got too many workers hitting the same target too fast. Start at 5-10 workers and increase gradually while watching your error rate, not the other way around.
8. log status codes, proxies, and exceptions
A scraper that fails silently is worse than one that crashes loudly. Log enough per request to diagnose a bad run without re-running it.
import logging
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
logger = logging.getLogger("scraper")
try:
resp = session.get(url, timeout=10)
logger.info(f"{resp.status_code} {url} via {session.proxies.get('https')}")
except requests.exceptions.RequestException as e:
logger.error(f"FAILED {url}: {e}")
Expected output: a log file you can grep for status code distributions or failure patterns after the run finishes.
If it breaks: if the log volume is too large to read, aggregate instead of logging every line, count status codes per minute and only log full detail on failures.
common pitfalls
- skipping timeouts. This is the single most common cause of a scraper that “just hangs.” Every
.get()or.post()call needs an explicittimeout. - assuming header spoofing beats TLS fingerprinting. A perfect User-Agent doesn’t help if the handshake underneath still looks like Python. If a site is still blocking you with clean headers and a good proxy, that’s usually why.
- retrying blindly without backoff. Hammering the same URL through the same proxy on every failure gets that proxy burned fast, and on a rotating pool you’re paying for that bandwidth either way.
- not URL-encoding proxy credentials. A password with
@or:in it silently breaks the proxy URL and gives you a confusing 407 instead of an obvious parse error. - scaling workers before checking error rate. Throughput numbers look great until half your requests start coming back blocked. Watch failure percentage, not just requests-per-second.
scaling this
At 10x (a few hundred to a few thousand requests per run), a single process with a ThreadPoolExecutor at 10-20 workers and a rotating proxy gateway is enough. One Session per thread (thread-local storage) avoids connection pool contention.
At 100x, a single process starts hitting real limits: GIL contention on CPU-bound parsing, one proxy provider’s rate ceiling, and memory if you’re holding response bodies in a list instead of streaming them to disk or a database. This is where I’d split into multiple worker processes, each with its own proxy sub-pool, coordinated through a queue like Redis or a simple database table of pending URLs. You also want per-domain rate limiting logic here, not just a global one, since different targets tolerate very different request rates.
At 1000x, Requests plus threads usually isn’t the right tool anymore. The GIL caps how much a thread pool actually parallelizes CPU work, and you’re better off with an async client like httpx or aiohttp running thousands of concurrent connections per process, distributed across multiple machines. Proxy strategy also changes: you need enough IP diversity that a provider’s pool doesn’t become the bottleneck, and you’re now managing cost per successful request as a real line item, not just cost per GB.
where to go next
If proxies are the part still tripping you up, read debugging 429 errors: rate limits, proxy quality, and behavioural patterns for how to tell a proxy problem from a rate-limit problem. If you’re hitting Cloudflare specifically, how to bypass Cloudflare 403s with Playwright plus residential proxies covers the case where Requests alone genuinely isn’t enough and you need a real browser engine. For the full archive, see /blog/.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-24.