← all guides

Routing a Docker container through a proxy

The container that said it was proxied

A scraper runs fine on a laptop with the proxy variables set in the shell. It gets packaged into a Docker image, deployed to a server, and the logs say every request went through the proxy. A week later someone checks the target’s responses and notices that a share of the traffic came from the server’s own address. Some library in the image was ignoring the proxy, or DNS was going out directly, or the container never had the variables in the first place and a fallback path quietly connected direct.

That pattern is common because Docker has more than one place where proxy configuration can live, and they don’t talk to each other. The daemon has its own settings for pulling images. Builds have their own for fetching packages. Running containers have their own for everything the application does. Setting one doesn’t set the others.

This piece walks through each of those three layers, then the parts that go wrong even when all three are set: libraries that don’t read the environment, DNS that leaves by another route, a proxy on the host that the container can’t reach, and credentials that end up baked into an image.

The broader problem of proxy environment variables reaching further than you meant is covered in its own piece on this site, as is DNS resolution inside a SOCKS tunnel. Those apply here too. What’s specific to containers is the layering, the networking, and the fact that the container is where a lot of scrapers end up running in production, which is exactly where a silent bypass costs the most.

The end goal is a container that can only reach the internet through the proxy, so anything that tries to go around it fails with an error instead of succeeding quietly.

Three layers that don’t share settings

It helps to name the three layers precisely before touching any config.

The daemon layer is the Docker engine itself, the background service that pulls images from registries, pushes them, and manages containers. When you run a pull, the daemon makes the network connection, not your shell and not the container. So the daemon needs its own proxy settings if pulls must go through a proxy, and those settings have nothing to do with what your scraper does.

The build layer is what happens while an image is being built. Each instruction that runs a command, such as installing packages or downloading a dependency, runs inside a temporary container. Those commands need proxy settings if the build environment can only reach the internet through a proxy. Build-time settings are passed in a specific way and, done right, don’t persist into the finished image.

The runtime layer is the running container: your scraper, its HTTP client, its browser, its DNS lookups. This is the one that matters for scraping, and it’s configured through the container’s environment and, more importantly, through the application’s own settings.

Most teams only need the runtime layer for the proxy they scrape through. The daemon and build layers matter when the server itself sits behind a corporate proxy, which is a different proxy with a different purpose. Mixing them up leads to a scraping proxy being used for image pulls, burning metered bandwidth on gigabytes of base images, or a corporate proxy being used for scraping.

So the first decision is which proxy belongs to which layer, and whether the scraping proxy should appear anywhere outside the runtime layer at all. Usually it shouldn’t.

The daemon layer, for pulls and pushes

If the server needs a proxy to reach image registries, the daemon has to be told. On Linux with systemd, the long-standing way is a drop-in file for the Docker service that sets environment variables for the daemon process:

# /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://proxy.internal:3128"
Environment="HTTPS_PROXY=http://proxy.internal:3128"
Environment="NO_PROXY=localhost,127.0.0.1,registry.internal"

After adding it, reload systemd and restart Docker. Recent Docker Engine releases also accept proxy settings in the daemon’s own configuration file, under a proxies key, which avoids editing systemd units. Check the documentation for the version you run, because the supported keys have changed over time.

Two things about this layer catch people. First, it only affects the daemon’s own connections. Setting it doesn’t make containers use the proxy, and it doesn’t affect builds either, apart from the daemon pulling the base image a build starts from.

Second, if you point the daemon at a metered scraping proxy, every image pull goes through it. Base images for browser automation are large. A deployment pipeline that pulls fresh images on every release can move a lot of data through a proxy that bills by the gigabyte, for traffic that never needed to be proxied. Keep registry traffic off the scraping proxy unless there’s a genuine network reason.

Docker Desktop has its own proxy settings in its preferences, which apply to its internal virtual machine. Those behave similarly: they cover pulls and can be passed to containers, but they’re a separate place to check when something doesn’t add up.

The build layer, and keeping it out of the image

During a build, commands that fetch packages need network access. If that access must go through a proxy, Docker provides a set of predefined build arguments for exactly this: HTTP_PROXY, HTTPS_PROXY, FTP_PROXY, NO_PROXY and their lowercase forms. You pass them at build time:

docker build \
  --build-arg HTTP_PROXY=http://proxy.internal:3128 \
  --build-arg HTTPS_PROXY=http://proxy.internal:3128 \
  -t scraper:latest .

These predefined arguments are available to build commands without being declared in the Dockerfile, and Docker leaves them out of the image’s history by default. That’s the property you want: the build can use the proxy, and the finished image doesn’t carry the setting.

The common mistake is writing the proxy into the Dockerfile with an ENV instruction. ENV values persist into the image. Every container started from that image inherits them, and anyone who can pull the image can read them with an inspect command. If the value includes a username and password, which scraping proxy URLs often do, the credentials now travel with the image to every registry and every machine it lands on.

A related mistake is declaring the proxy as an explicit ARG in the Dockerfile and then using it in a way that records it, which can put the value back into the history the predefined behaviour was keeping it out of.

The Docker client can also inject proxy settings automatically. A proxies section in the client’s config file, in the user’s Docker config directory, is passed to builds and to containers that client starts. That’s convenient for a corporate proxy on a developer machine. It’s easy to forget about, and it’s one more place to look when a container has proxy variables nobody remembers setting.

For a scraper image, the clean rule is that the scraping proxy never appears in the build at all. Build with whatever network the build host has, and supply the scraping proxy at runtime.

The runtime layer, where scraping actually happens

At runtime, the usual way to hand a container its proxy is through environment variables:

docker run --rm \
  -e HTTP_PROXY=http://forwarder:8080 \
  -e HTTPS_PROXY=http://forwarder:8080 \
  -e NO_PROXY=localhost,127.0.0.1 \
  scraper:latest

In Compose, the same values go under the service’s environment key, ideally pulled from an env file or a secrets mechanism rather than written into the compose file itself.

The important thing to understand is that environment variables are only a convention. Docker puts the values into the container’s environment. What happens next depends entirely on the software inside. Some HTTP clients read those variables automatically. Some read only the lowercase form, some only the uppercase, some neither. Some read them only when a setting is enabled. Browsers generally ignore them entirely.

So setting the variables at runtime is necessary for many tools, and sufficient for almost none on its own. The runtime layer is really two layers: the container environment Docker controls, and the application configuration your code controls. The second one decides where traffic actually goes.

A better practice for scrapers is to configure the proxy explicitly in the application, from a single setting your code reads, and use environment variables only as the source of that setting. Then each HTTP client and each browser launch is told about the proxy directly, and there’s no reliance on every library in the image happening to honour the same convention.

Libraries that ignore the proxy variables

This is where most container leaks come from. A scraper image rarely contains one HTTP client. It contains the one you chose, plus whatever your dependencies use internally, plus command-line tools called from scripts, plus a browser.

A few behaviours worth knowing, stated generally because versions change:

Python’s requests library reads the proxy variables by default. httpx does too, through its trust_env setting, which is on by default. aiohttp does not read them unless you enable trust_env when creating the session. So an async scraper built on aiohttp can ignore variables that a requests-based helper in the same image honours.

curl reads the lowercase http_proxy variable for plain HTTP, and deliberately ignores the uppercase HTTP_PROXY form for that scheme, a defence against a class of server-side header injection. HTTPS_PROXY works in either case. A shell script calling curl can therefore behave differently from a Python client given the same environment.

Go’s standard HTTP client reads the variables through its default transport, but code that builds a custom transport without setting the proxy function bypasses them.

Node’s built-in fetch has historically not read the proxy variables by default, and recent versions have added an opt-in rather than changing the default. Many Node HTTP libraries need an explicit agent.

Headless browsers launched by Playwright or Puppeteer ignore the environment and take the proxy as a launch option or a command-line flag. The Playwright proxy piece on this site covers the settings that look right and do nothing.

NO_PROXY is its own mess: some tools accept CIDR ranges, some only exact hosts or domain suffixes, some treat a leading dot differently. A NO_PROXY that works for one client may send another client’s traffic direct.

DNS takes its own route out

Even when every HTTP request goes through the proxy, name resolution may not.

How DNS behaves depends on the proxy protocol. With an HTTP proxy, a client sending an HTTPS request uses a CONNECT request naming the hostname, and the proxy resolves it. With a plain HTTP proxy request, the full URL goes to the proxy, which resolves it. But many clients still perform their own lookup first, for connection pooling, for certificate checks, or simply because the code path resolves before it decides whether to use a proxy. With SOCKS, it depends on whether the client sends the hostname or resolves locally and sends an address; the socks5 versus socks5h distinction in many tools exists for exactly this. The DNS inside a SOCKS tunnel piece goes through that in detail.

In a container, local lookups go to Docker’s resolver. On a user-defined network, that’s Docker’s embedded DNS server inside the container’s network namespace, which forwards external names to the resolvers configured on the host. On the default bridge network, containers use a copy of the host’s resolver configuration. Either way, local lookups leave from the server’s network, not the proxy’s.

Why does that matter for scraping? A DNS lookup from the server reveals which hostnames the job is contacting to the server’s resolver, and for some geo-dependent services it can return an answer suited to the server’s location rather than the proxy’s. It’s also a hint to anyone watching that the job’s traffic doesn’t all originate from where the HTTP requests do.

The fix is to prefer proxy protocols and client settings where the proxy does the resolution, and to verify it rather than assume.

The proxy on the host that the container can’t see

A frequent setup has a local proxy or forwarder running on the host, listening on 127.0.0.1, with scraper containers told to use it at 127.0.0.1 too. It doesn’t work, and the failure is confusing.

Inside a container on a bridge network, 127.0.0.1 is the container’s own loopback interface, not the host’s. A proxy listening on the host’s loopback is invisible from there. The client either fails to connect, or, if it’s configured to fall back, connects direct. The second outcome is the dangerous one.

There are three sensible ways around it.

Run the proxy or forwarder as its own container on the same user-defined network, and address it by its service name. Compose makes this natural: a forwarder service and a scraper service on one network, with the scraper pointing at the forwarder by name. This is usually the cleanest option.

Use the special hostname host.docker.internal. Docker Desktop provides it automatically. On Linux, you add it yourself with an extra host entry mapping it to the host gateway. Then the host’s proxy must listen on an address reachable from the bridge network, not only on loopback, and the host firewall should restrict who can reach it.

Use host networking for the scraper container, which puts it in the host’s network namespace so 127.0.0.1 means the host again. That works, but it removes the network isolation that makes the next section possible, so it’s the least attractive of the three.

Whatever you choose, make the scraper’s HTTP client fail on proxy connection errors instead of falling back to a direct connection. Many clients don’t fall back by default. Some wrapper code does, usually through a retry path written with good intentions.

An internal network that makes bypass fail loudly

Everything so far is about configuring the proxy correctly. This part is about making it impossible to get wrong silently.

Docker lets you create an internal network, one with no route to the outside world. Containers on it can reach each other and nothing else. Put the scraper on an internal network, alone with a forwarder container. Attach the forwarder to a second, ordinary network as well, so it can reach the upstream proxy. The scraper’s only path to the internet is now through the forwarder.

docker network create --internal scrape-internal
docker network create scrape-egress

docker run -d --name forwarder \
  --network scrape-egress forwarder-image
docker network connect scrape-internal forwarder

docker run --rm --network scrape-internal \
  -e HTTPS_PROXY=http://forwarder:8080 \
  scraper:latest

In Compose, the same thing is a network declared with internal set to true, the scraper attached only to that network, and the forwarder attached to both.

With this layout, a library that ignores the proxy variables doesn’t leak. It fails to connect, and the failure shows up in logs and tests. A browser launched without its proxy flag can’t load anything. A retry path that falls back to direct has nowhere to go.

DNS on an internal network is worth testing on your own Docker version. Container names resolve through the embedded DNS, which is what lets the scraper find the forwarder. Whether external names resolve at all from an internal network has changed between releases, and in recent ones they don’t, which is the behaviour you want: the scraper can’t even look up a target without going through the forwarder. Run a lookup from inside the container and see.

The forwarder itself can be a small local proxy that holds the upstream credentials and exposes a plain port on the internal network, which also keeps credentials out of the scraper container entirely.

A Compose file that puts it together

Here’s the same layout written as a Compose file, which is how most teams will actually run it. Service names and images are placeholders.

services:
  forwarder:
    image: forwarder-image
    networks: [egress, internal]
    secrets: [upstream_proxy]

  scraper:
    image: scraper:latest
    networks: [internal]
    environment:
      HTTPS_PROXY: http://forwarder:8080
      HTTP_PROXY: http://forwarder:8080
      NO_PROXY: localhost,127.0.0.1
    depends_on: [forwarder]

networks:
  internal:
    internal: true
  egress: {}

secrets:
  upstream_proxy:
    file: ./secrets/upstream_proxy.txt

A few details matter here. The scraper is attached only to the internal network, so it has no route out except the forwarder. The forwarder sits on both networks and reads the upstream credentials from a secret file, which lands inside its container as a file rather than an environment variable. The proxy URL the scraper sees has no credentials in it at all, so nothing sensitive shows up in the scraper’s environment or its logs.

The depends_on entry only controls start order. It doesn’t wait for the forwarder to be ready. Give the scraper a startup check that makes a test request through the forwarder and exits with an error if it fails, rather than starting its job and discovering the problem on the first real request.

The secrets file sits outside the repository, or in a path the repository ignores. The compose file itself contains nothing secret and can be committed safely.

Browsers inside containers

Browser automation in containers has its own set of proxy traps, on top of the general ones.

The browser ignores the environment, so the proxy has to be passed at launch or per context. With Playwright and Puppeteer, that means the proxy option on launch or on each new context. With a raw Chromium command line, it’s the proxy server flag. Forget it, and on an ordinary network the browser connects direct while your HTTP client obediently uses the proxy.

Browsers also make connections your scraping code never asked for. Background requests for updates, safe browsing lists, component downloads and telemetry can go out as soon as the browser starts. Some of those respect the proxy setting and some are controlled by separate flags. On the internal network layout, any that don’t go through the proxy simply fail, which is the outcome you want. On an open network, they leak.

Real-time media features in browsers can open UDP connections, and an HTTP proxy only carries TCP. Browsers have settings to restrict that behaviour, and automation frameworks expose some of them. Again, the internal network makes the question moot for scraping, because UDP has nowhere to go.

There are practical container issues as well. Chromium uses shared memory heavily, and Docker’s default shared memory size is small, which causes tab crashes that look like network failures. Increase the shared memory allocation for browser containers, or configure the browser to avoid relying on it. Browser images are also large, which is one more reason to keep image pulls off a metered scraping proxy.

Test the browser path separately from the HTTP client path. Load an echo page inside the automated browser and read the address it reports, rather than assuming the browser inherited anything.

Keeping credentials out of images and inspect output

Scraping proxy URLs often embed a username and password. In a container setup, there are several places those can end up where they shouldn’t.

Baked into the image through an ENV instruction, as covered above. Anyone with the image can read them.

In the container’s environment, passed with an environment flag. That’s better than the image, but the values are visible to anyone who can run an inspect on the container, and they appear in the process environment inside it. Crash reports and debug dumps sometimes include the environment wholesale.

In compose files committed to a repository, which then travel wherever the repository goes.

In logs, when an HTTP client or a wrapper prints the proxy URL on connection errors. Some do, including the credentials.

The forwarder pattern helps here. If the scraper only knows a plain unauthenticated address on an internal network, there’s nothing secret in its environment, its image or its logs. The forwarder holds the credentials, ideally supplied through a secrets mechanism or a mounted file with tight permissions rather than an environment variable. Swapping providers becomes a change to one container’s configuration.

If you must pass credentials to the scraper directly, read them from a mounted secret file at startup, keep them out of any URL your code logs, and make sure error handlers redact proxy URLs before printing them. And if credentials ever do land in an image or a repository, treat them as leaked: rotate them with the provider and rebuild, since deleting a layer or a commit doesn’t remove copies already pulled or cloned.

Verifying from inside the container

A container configuration you haven’t tested from the inside is only a claim about where its traffic goes.

The first check is the exit address. From inside the running container, request an echo endpoint that returns the caller’s public address, using the same client library your scraper uses, not just curl. Compare the result with the proxy’s expected exit and with the server’s own address. Repeat with each client in the image: the main HTTP library, any helper tools, the browser.

The second check is DNS. On the host, capture traffic on the Docker bridge interface or the host’s outbound interface while the scraper runs, filtered to DNS. If you see lookups for target hostnames leaving the host directly, some part of the stack is resolving locally. On the internal network layout, you should see nothing, because the scraper can’t reach a resolver outside.

The third check is failure behaviour. Stop the forwarder or break the proxy address deliberately and run the scraper. It should fail. If it succeeds, it found a path around the proxy, and that path needs closing.

The fourth check is the image. Inspect it and its history for proxy values, and inspect a running container’s environment. Nothing secret should be visible in either.

Run these checks once when setting up, and again after any change to the base image or dependencies, because a new library version can change how it reads the environment.

Scaling replicas without multiplying the load

Containers make it trivial to run more copies of a scraper. One line in a Compose file or one setting in an orchestrator and there are ten replicas instead of one. That’s useful for throughput and dangerous for politeness.

Each replica usually carries its own rate limiter, configured for what one scraper should send to a target. Ten replicas with that limiter send ten times the traffic. Nothing in the proxy layer notices, because the proxy’s job is routing, and the target sees ten times the load arriving through however many exits the pool gives it.

The fix is to move the rate budget out of the replicas. A shared limiter, whether a small service, a counter in a cache everyone can reach, or a queue that hands out work at a controlled pace, keeps the total traffic per target where it should be regardless of replica count. The internal rotation service piece on this site describes the same idea at team scale: one place that sees all traffic to a domain and caps it.

The forwarder can help here too. Because every replica’s traffic passes through it, the forwarder is a natural place to count requests per target domain and to log them. It shouldn’t be the only control, since the scraper still needs to back off on errors itself, but it gives you one view of total load that no single replica has.

Scaling also multiplies failure. If the proxy goes down, ten replicas retrying at once can turn a brief outage into a burst of traffic the moment it recovers. Use backoff with jitter in each replica, and let the shared limiter absorb the rest.

What this setup doesn’t do

A container that routes all its traffic through a proxy is a correctly plumbed container. That’s all it is.

It doesn’t make the scraper polite. Request pacing, concurrency limits, backoff on errors, caching and respect for a target’s robots rules and terms are all decisions in the application. Putting the scraper in a well-isolated container changes none of them. If anything, containers make it easy to scale a scraper horizontally, and each replica needs to share one rate budget per target, not multiply it.

It doesn’t hide what the client is. The target still sees the HTTP client’s headers, its TLS handshake and, for browsers, everything the page’s scripts collect. The piece on what your scraper leaks besides its IP covers those. A proxy decides the address, and that’s one signal among many.

It doesn’t guarantee that the proxy exit is the right one for the job. A container on the wrong proxy type or the wrong country is correctly routed and still wrong.

And it doesn’t make it acceptable to fetch things you shouldn’t. The target’s rules and the law about personal data apply the same whether the request leaves a laptop or a hardened container.

What the setup does do is remove a category of silent failure. When the proxy is the only way out, a misconfiguration becomes an error you notice on day one instead of a leak you discover a month later. For scrapers running unattended on servers, that’s worth the extra network definition.

If you want the related pieces on proxy environment variables, DNS inside tunnels and where to run a scraper in the first place, Proxy Scraping has the full set of guides from the operator side.

Get new guides and videos first — join the Telegram channel.

proxies
Need proxies that survive the block wall?

Singapore Mobile Proxy runs real 4G/5G mobile IPs on rotating SIMs — the carrier-grade addresses most of these targets still trust.

see plans →
read on
More scraping guides

The rest of the field manual: target-site playbooks, library walkthroughs, provider reviews, and anti-bot troubleshooting.

browse all guides →