Connection reuse and multiplexing in a modern scraper

Most scraper performance problems don’t come from the target site being slow. They come from the scraper throwing away a perfectly good connection after every single request and paying the setup cost all over again. If you’ve ever wondered why one scraper pulls 200 pages a minute through a proxy pool and another chokes at 20, connection handling is usually the answer before proxy quality even enters the picture.

What a connection actually costs

Every HTTP request over a fresh TCP connection pays a few taxes before a single byte of the response shows up. There’s the TCP handshake (SYN, SYN-ACK, ACK), then if the request is HTTPS there’s the TLS handshake on top of that, which itself involves a certificate exchange and key negotiation. Depending on the TLS version and whether session resumption is available, that’s another one or two round trips. On a direct connection with low latency this might cost 100-300ms. Route that same connection through a residential or mobile proxy, where the request has to cross the proxy’s own network hop before reaching the target, and the handshake tax climbs, sometimes past a full second per new connection.

Open a new connection for every request and that tax gets paid every single time. Reuse a connection across many requests and you pay it once, then every subsequent request just rides the already-negotiated TCP and TLS session. This is the entire idea behind keep-alive, and it’s been part of HTTP since 1.1. A scraper that ignores it is leaving most of its throughput on the table before it ever hits a rate limit.

Where HTTP/1.1 keep-alive runs out of road

HTTP/1.1 keep-alive lets you reuse a connection, but it’s still fundamentally one request in flight at a time per connection (pipelining exists on paper but almost nothing implements it reliably, and it introduces head-of-line blocking where one slow response stalls everything queued behind it). So a scraper wanting concurrency under HTTP/1.1 opens multiple connections to the same host and spreads requests across them. That works, but it multiplies the handshake tax by however many parallel connections you’re running, and many servers cap concurrent connections per client IP, which through a proxy means per proxy IP.

What HTTP/2 changes

HTTP/2 introduces multiplexing: multiple request/response exchanges, called streams, share a single TCP+TLS connection at the same time. Instead of a text-based request line and headers, HTTP/2 uses binary framing, and headers get compressed with HPACK rather than being sent as raw repeated text on every request. The practical effect for a scraper is that you can issue a dozen concurrent requests to the same host over one connection instead of a dozen connections, which means one handshake tax instead of a dozen.

This matters more through a proxy than it does on a direct connection, because the proxy hop is usually the slowest and least reliable leg of the trip. Fewer connections through a mobile or residential exit means less handshake overhead riding on top of what’s often already a higher-latency, higher-jitter network path than a direct datacenter connection.

The proxy layer complicates this

Not every proxy setup passes HTTP/2 through cleanly. A proxy operating as a plain HTTP forward proxy that terminates and re-issues the request on the target-facing side may downgrade the connection to HTTP/1.1 regardless of what the client asked for, especially with older proxy software. A SOCKS5 proxy, by contrast, just tunnels raw bytes and doesn’t care what protocol rides on top, so HTTP/2 negotiated via ALPN during the TLS handshake generally survives a SOCKS5 hop intact. This is one of the more overlooked differences between proxy types when people compare them purely on IP type (residential vs mobile vs datacenter) and skip over the protocol layer entirely. Two providers with the same IP pool quality can still produce very different scraper throughput because one proxies at the SOCKS5/tunnel layer and the other reverse-proxies HTTP.

Datacenter proxies tend to be the most consistent here because the infrastructure is purpose-built and usually documents its protocol support. Residential and mobile proxy networks route through consumer devices or carrier gateways acting as exit nodes, and behavior can vary between individual exits in the pool, which is worth testing rather than assuming.

Connection reuse and rotation don’t mix cleanly

Rotating proxies exist to spread requests across many IPs, usually to stay under a target’s per-IP rate threshold. Connection reuse exists to avoid re-paying handshake cost. These two goals pull in opposite directions: a connection, once open, is pinned to whichever proxy IP it was opened through. Reusing that connection for many requests means many requests riding the same exit IP, which is the opposite of what rotation is trying to achieve.

The practical resolution most production scrapers land on is rotating per connection rather than per request: open a connection through a given proxy IP, multiplex a batch of requests over it while it’s alive, then close it and open the next batch through a new IP. How large that batch should be depends entirely on the target site’s own tolerance, which isn’t something a scraper operator can know in advance and has to be inferred from response behavior over time.

Fingerprinting is a connection-layer conversation too

Detection systems that classify traffic as automated don’t only look at request headers and cookies. The TLS ClientHello itself, and the HTTP/2 SETTINGS frame and stream priority behavior that follows it, produce a fingerprint that’s often more consistent per HTTP client library than the User-Agent string is. A library like Python’s requests doesn’t speak HTTP/2 at all, and its TLS handshake shape looks nothing like a real browser’s, which is one reason a scraper can send a browser-accurate User-Agent header and still get flagged: the mismatch shows up two layers below the header a scraper author was focused on. Libraries like httpx with HTTP/2 support, or curl_cffi, which impersonates specific browser TLS and HTTP/2 fingerprints, exist because of this gap. None of this is a guarantee against detection. It’s a description of one more signal that detection systems check, on top of behavioral signals like request timing and navigation patterns, and it’s worth understanding for anyone trying to run a compliant scraper that doesn’t accidentally look more suspicious than the traffic it’s meant to represent.

What this means for how you build a scraper

A few concrete takeaways, none of them exotic:

Use an HTTP client library that actually supports connection pooling and, where the target and proxy allow it, HTTP/2. Don’t open and tear down a connection per request out of habit.

Batch requests by proxy session rather than treating rotation as “new IP every request.” Decide your batch size based on what you observe about the target’s own rate tolerance, not a fixed number copied from somewhere else.

Check whether your proxy provider’s forward-proxy layer actually passes HTTP/2 through, or silently downgrades it. This is a five-minute test with a tool that reports the negotiated protocol, and it explains a lot of unexplained throughput differences between providers offering similar IP quality.

Treat TLS and HTTP/2 fingerprint consistency as part of your compliance posture, not just a performance knob. A scraper that behaves like a normal browser at the connection layer, respects the target’s stated rate limits, and only accesses content it has a legitimate right to access is a fundamentally different thing from a scraper trying to look like something it isn’t.

None of this makes a scraper invisible or guarantees it won’t get rate limited or blocked. Sites run detection precisely because automated traffic at scale, done carelessly, causes real load and real problems. Understanding how connections actually work is what separates a scraper that’s a good network citizen from one that just hasn’t been caught yet.

If you’re comparing proxy types or picking tooling for a scraping project, check out the rest of Proxy Scraping for more breakdowns like this one.

Get new guides and videos first — join the Telegram channel.