How to scrape Home Depot at scale in 2026 with proxies that work
Home Depot is one of the more useful retail targets if you track pricing, stock, or catalog changes. It is also one of the more annoying ones. Prices and availability depend on the store you pick, the site sits behind bot protection, and a scraper that works for 50 pages will often fall over at 5,000. I run scrapers like this from Singapore, and most of the failures I see come from the wrong proxy type or a sloppy session setup, not from the parsing code.
This tutorial is for operators and small data teams who need product, price, and store-level stock data from homedepot.com on a repeatable schedule. You should be comfortable with Python and the command line. I do not cover logins or anything behind an account. This is public product pages only.
By the end you will have a working pipeline: residential US proxies, store-pinned sessions, a fetch loop with retries, a parser that reads the structured data on the page, and a storage layer that tells you when something silently breaks. This is not legal advice. Read Home Depot’s terms and robots.txt yourself, and talk to a lawyer if your use case is commercial and sensitive.
What you need
- python 3.11 or newer, with
httpxorrequestsfor plain fetching andplaywrightfor pages that need a real browser - a pool of US residential proxies, ideally with sticky sessions and city or ZIP targeting. expect to pay roughly $4 to $10 per GB from the mainstream providers, and check current pricing pages because it moves
- optionally a few US mobile or ISP proxies as a fallback for the hardest pages
- a list of product URLs or SKUs (item IDs), plus the store IDs or ZIP codes you care about
- somewhere to store results: sqlite is fine to start, Postgres once you pass a few million rows
- a small budget for failures. plan to burn 20 to 40 percent of your first bandwidth estimate on retries and blocked requests while you tune
- about two hours for a first version
If you are new to proxy types, I covered the tradeoffs in residential vs datacenter vs mobile proxies. For Home Depot, datacenter IPs get flagged fast, so I do not bother with them as the main pool.
Step by step
1. Read the rules before you write code
Open homedepot.com/robots.txt and read which paths are disallowed. The robots exclusion standard is defined in RFC 9309, and it is a courtesy convention rather than access control, but it tells you what the site owner does not want crawled. Stay off disallowed paths and keep your request rate low.
Expected output: a short list of path prefixes you will not touch, saved in your repo as NOTES.md.
If it breaks: if robots.txt returns a block page instead of text, you are already flagged. Try from a clean residential IP and a normal browser user agent.
2. Test your proxy before you scale anything
Do not point 500 threads at a new proxy pool. Check the exit IP, the country, and the latency first.
curl -x "http://USER:[email protected]:7000" https://api.ipify.org
Run it ten times. With rotating sessions you should see different US IPs. With a sticky session you should see the same IP repeatedly. Then check that the IP resolves to the US city you asked for using any IP lookup you trust.
Expected output: ten lines of IPs, all US, none obviously datacenter ranges.
If it breaks: if you get 407, your credentials or gateway port are wrong. If the IPs are not US, your provider’s geo parameter is formatted differently, so check their docs for the exact username syntax.
3. Fetch one product page with plain HTTP
Start simple. Many product pages return server-rendered HTML with JSON-LD in it, which is far cheaper than running a browser. The requests proxy docs cover the proxy dictionary format.
import requests
PROXY = "http://USER:[email protected]:7000"
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
}
r = requests.get(
"https://www.homedepot.com/p/EXAMPLE/ITEMID",
headers=HEADERS,
proxies={"http": PROXY, "https": PROXY},
timeout=30,
)
print(r.status_code, len(r.text))
Replace the URL with a real product page. Expected output: status 200 and a body of several hundred KB.
If it breaks: a 403 or a tiny body (a few KB) means bot protection rejected the request. In my experience the TLS fingerprint of plain requests is often the cause, not the headers. Move to step 4 before burning more proxies on it.
4. Switch to a real browser for the pages that need it
When plain HTTP gets blocked, use Playwright with a real Chromium. Playwright’s network docs show how to set a proxy per browser or per context.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
ctx = browser.new_context(
proxy={"server": "http://gate.yourprovider.com:7000",
"username": "USER", "password": "PASS"},
locale="en-US",
timezone_id="America/Chicago",
)
page = ctx.new_page()
page.goto("https://www.homedepot.com/p/EXAMPLE/ITEMID", wait_until="domcontentloaded")
html = page.content()
print(len(html))
browser.close()
Match the timezone and locale to the proxy’s location. A Chicago IP with a Singapore timezone is an easy tell. If you need heavier fingerprint control, the tooling comparison on antidetectreview.org is where I would start.
Expected output: full rendered HTML with the price element present.
If it breaks: if you get a challenge page, slow down, use a fresh residential IP, and add a few seconds of idle time before reading the page. Do not retry the same blocked IP in a tight loop.
5. Pin the store and ZIP code
Home Depot prices and stock vary by store. If you do not set a store, you get whatever the IP’s location implies, and your data becomes inconsistent. Pick the store once in a browser session, then reuse that session’s cookies for every request on that exit IP.
The practical rule: one sticky proxy session, one store, one cookie jar. Do not mix them. I name sessions like store-0121-a in my scheduler so I can see which store each row came from.
Expected output: the same SKU returning a store-specific price and a pickup availability line that names your chosen store.
If it breaks: if the store resets to a default, your cookies are not persisting across requests. Save the Playwright storage state to disk and reload it with browser.new_context(storage_state="store-0121.json").
6. Parse the structured data, not the layout
CSS classes on retail sites change often. JSON-LD and embedded state objects change less. Look for <script type="application/ld+json"> first.
import json
from bs4 import BeautifulSoup
def parse_product(html):
soup = BeautifulSoup(html, "html.parser")
for tag in soup.find_all("script", type="application/ld+json"):
try:
data = json.loads(tag.string or "")
except json.JSONDecodeError:
continue
items = data if isinstance(data, list) else [data]
for item in items:
if item.get("@type") == "Product":
offer = item.get("offers", {})
if isinstance(offer, list):
offer = offer[0] if offer else {}
return {
"name": item.get("name"),
"sku": item.get("sku"),
"price": offer.get("price"),
"currency": offer.get("priceCurrency"),
"availability": offer.get("availability"),
}
return None
Expected output: a dict with name, SKU, price, and availability. Test it on 20 pages across different categories.
If it breaks: some categories, like appliances or made-to-order items, use different structures. Log the URLs where parse_product returns None and look at three of them by hand before changing the parser.
7. Add retries, backoff, and a block detector
At scale, the fetch loop matters more than the parser. Treat 403, 429, and suspiciously short responses as blocks. Retry on a new IP with exponential backoff and a cap.
import random, time
def fetch_with_retry(fetch, url, max_tries=4):
for attempt in range(max_tries):
status, body = fetch(url)
if status == 200 and len(body) > 50_000:
return body
time.sleep((2 ** attempt) + random.uniform(0, 1.5))
return None
The 50,000 byte threshold is my own rule of thumb for this site, not a documented number. Calibrate yours from real pages.
Expected output: a block rate you can measure. Log status, byte length, and proxy session ID for every request.
If it breaks: if more than about 10 percent of requests need a retry, the problem is upstream. Check your request rate, your IP quality, and your fingerprint, in that order.
8. Store results with timestamps and a run ID
Write every result to sqlite with sku, store_id, price, availability, fetched_at, and run_id. Keep the raw HTML for a small random sample, say 1 in 200 pages, so you can debug parser drift later without refetching.
Expected output: a table you can query for price changes between runs.
If it breaks: if prices look wrong, compare a stored raw page against the live site by hand. Nine times out of ten it is a store mismatch, not a parser bug.
9. Schedule it and alert on silence
Run on cron or a small queue worker. The most important alert is not an error, it is a drop in row count. If last Tuesday’s run returned 40,000 rows and today’s returned 9,000, something broke quietly. Send yourself a message when the row count falls more than 25 percent below the trailing average.
Expected output: a daily run and one alert path you actually read.
If it breaks: if the run finishes with no errors but low volume, you are probably being served empty or degraded pages. Check the byte-length histogram in your logs.
Common pitfalls
- using datacenter proxies for the main pool. they are cheap and they work for a day, then the block rate climbs and you pay for it in retries
- ignoring the store. a dataset where half the rows came from one store and half from a default location is close to useless for price tracking
- rotating the IP on every request while keeping the same cookies. a session that changes country-level identity mid-visit looks wrong. rotate between sessions, not inside them
- scraping at the speed your proxy plan allows rather than the speed the site tolerates. low and steady beats fast and blocked
- skipping the monitoring. most scraper failures are silent, and you find out when a customer or a spreadsheet tells you
If you also run multiple accounts or sessions in other workflows, the operational habits overlap a lot. multiaccountops.com has good write-ups on session hygiene.
Scaling this
Each jump from 10x to 1000x changes different things.
At 10x, say a few thousand pages a day, a single machine and one residential plan is fine. You will mostly fight parser drift and store mismatches.
At 100x, roughly hundreds of thousands of pages a day, bandwidth becomes the budget line. Browser fetches cost far more per page than plain HTTP, so push as much as possible down to the cheap path and use Playwright only as a fallback. Block pages count against your bandwidth too, so a high retry rate is a direct cost. You will also want a proper queue (Redis, or a Postgres-backed job table), per-store session pools, and dashboards for block rate by proxy session.
At 1000x, millions of pages a day, you are negotiating with proxy providers rather than buying a self-serve plan, and you are probably running your own session manager. Split the workload by store and category, stagger schedules so you do not hit the site in a daily spike, and consider whether you need every SKU every day. Most catalogs have a hot set that changes often and a long tail that barely moves. Refreshing the tail weekly cuts cost sharply without hurting the data you use.
At every level, I would rather have 80 percent coverage with clean store-pinned data than 100 percent coverage with noise.
Where to go next
- how to pick a residential proxy provider in 2026
- how to scrape Walmart product pages with rotating proxies
- the full list of tutorials is on the blog index
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-10-13.