Proxy SSL certificate errors: why HTTPS interception breaks your scraper
The error nobody explains properly
If you run scrapers behind proxies long enough, you’ll hit it: SSLCertVerificationError, unable to get local issuer certificate, certificate verify failed, or some flavor of “the connection is not private” if you’re testing in a browser. The instinct is to Google the exact error string, find a Stack Overflow answer that says verify=False, paste it in, and move on. That fixes the symptom and hides the actual problem, which is usually that something between your scraper and the target site is intercepting HTTPS traffic and you didn’t know it.
I run proxy infrastructure for a living, and this is one of the most common tickets I see from people running their own scrape jobs through a proxy for the first time. It’s not a proxy provider bug most of the time. It’s a fundamental thing about how HTTPS proxies can be built, and most people have never had to think about which kind they’re using.
What a plain HTTPS proxy actually does
When your client talks to a target site over HTTPS through a normal forward proxy, the proxy doesn’t see your traffic. It sees a CONNECT request to target.com:443, opens a TCP tunnel to that host, and then just shuttles encrypted bytes back and forth. The TLS handshake happens directly between your client and the target server. The proxy is a pipe. It can log that a connection happened and how much data moved, but it can’t read the request path, headers, or body, because it never holds the private key or the session keys.
This is the setup almost every rotating datacenter, residential, or mobile proxy pool uses by default, because it’s simple and it doesn’t require the proxy to hold a trusted certificate for every domain on the internet. Your client verifies the target’s real certificate, issued by a real CA, against its own trust store. If that checks out, the connection is genuinely end-to-end encrypted between you and the site. A proxy operating this way should never cause a certificate error, because it isn’t part of the TLS conversation at all.
What HTTPS interception actually is
Interception is different. A proxy that intercepts HTTPS terminates the TLS connection itself. It presents its own certificate to your client (signed by its own CA, not the site’s real CA), decrypts the traffic, does whatever it’s going to do with it, opens a second TLS connection to the real target on your behalf, and re-encrypts. From your client’s point of view, you’re now trusting the proxy’s certificate authority instead of the real one.
This is a legitimate design in some contexts. Corporate networks run this on their own managed laptops for content filtering and data-loss prevention, with the proxy’s root CA pushed to every device ahead of time so the client trusts it. Some security appliances do it for malware scanning. It works cleanly only when the client has been told, in advance, to trust that CA.
The problem is when your scraper hasn’t been told that, because it’s running on a stock Python environment, a stock Node container, or a CI runner that only trusts the standard public CA bundle. Your HTTP client does the right thing: it sees a certificate signed by an authority it doesn’t recognize, and it refuses the connection. That refusal is the certificate error. The client isn’t broken. It’s doing exactly what TLS verification is supposed to do, which is reject a certificate chain it can’t validate.
Where this shows up in a scraping stack
A few places this actually bites people running proxy-based scrapers:
Corporate or ISP-level networks. If your scraper runs on a machine sitting behind a company firewall or a managed network, that network’s own MITM appliance might be intercepting all outbound HTTPS regardless of which proxy you configured downstream. The certificate error has nothing to do with your proxy provider at all.
Proxy gateway software with inspection features. Some proxy management tools and self-hosted proxy stacks include an optional traffic-inspection mode, usually for debugging or logging request content during development. If that mode gets left on in a production job, every outbound request now needs the tool’s local CA trusted, and any client that wasn’t set up for that will fail verification.
Antivirus and endpoint security suites. A lot of consumer and business antivirus products do their own local HTTPS interception for “web protection,” installing a root cert into the OS trust store but not necessarily into every language runtime’s separate cert store. Python, for instance, often ships its own bundle via certifi rather than reading the OS store, so a cert that Windows trusts fine in a browser can still fail inside a Python script.
Misconfigured CONNECT handling. Some proxy software mishandles the CONNECT method under load or with certain client libraries, and instead of tunneling cleanly it returns an error page over what the client expected to be an encrypted tunnel. That can also surface as a certificate or handshake failure even though nothing is deliberately intercepting anything, it’s just broken.
How to actually diagnose it instead of guessing
Disabling verification tells you nothing except that the error is gone. If you actually want to know what’s happening, a few checks in order:
Run curl -v through the same proxy and look at the certificate chain it prints. It’ll show you the issuer of the cert being presented. If the issuer is a name you don’t recognize, something is terminating TLS between you and the target.
Compare that against hitting the same URL with no proxy at all. If the direct connection presents the site’s normal certificate (Let’s Encrypt, DigiCert, whoever the site actually uses) and the proxied connection presents something else, the proxy path is doing interception.
Check whether your environment has a custom CA bundle configured, and whether it’s actually the one your language runtime is reading. On a lot of systems the OS trust store and the Python/Node/Java trust store are separate, so “it works in the browser” and “it fails in the script” aren’t a contradiction, they’re two different trust stores that disagree.
If you’re running your own proxy or paying for a provider, ask directly whether their forward proxies do TLS termination anywhere in the path, and if so, what CA cert you’re expected to install. A provider running plain CONNECT tunnels shouldn’t need you to install anything at all, and if they’re asking you to trust a custom root cert to make errors go away, that’s worth understanding before you do it, not after.
Datacenter, residential, and mobile: does the proxy type matter here
The short answer is that the exit type (datacenter IP, residential, mobile) doesn’t determine whether interception happens. That’s a function of how the proxy software itself is built and configured, not where the exit node physically sits. A datacenter proxy pool can run pure tunnels just as easily as a residential one can, and either could in theory run an inspecting proxy layer if the operator built it that way, usually for caching, logging, or debugging on their own infrastructure rather than anything aimed at you. The honest way to know is to check the certificate chain yourself with curl -v, not to assume based on which category of proxy you bought.
The fix that isn’t verify=False
If the real cause is a corporate or antivirus MITM on a network you control, the fix is installing that authority’s CA cert into your runtime’s trust store, not turning verification off. If it’s a proxy tool’s inspection mode, the fix is turning that mode off for production traffic. If it’s a broken CONNECT implementation, the fix is switching proxy software or providers, or filing it as a bug.
Turning off certificate verification removes your protection against every one of those cases, plus real man-in-the-middle attacks that have nothing to do with your own infrastructure. It also has nothing to do with getting past a target site’s bot detection. Certificate errors happen between your client and the proxy layer, before your request ever reaches the site you’re scraping. Fixing the trust chain is a correctness problem, not a blocking problem, and it’s worth treating it that way.
If you want more breakdowns like this on how proxy infrastructure actually behaves, from someone running it day to day, take a look around proxyscraping.org.
Get new guides and videos first — join the Telegram channel.