How to scrape Indeed at scale in 2026 with proxies that work
Indeed is one of the harder job boards to scrape well. It’s not that the HTML is complicated, it’s that Indeed runs aggressive bot detection on top of Cloudflare, rate-limits by IP fast, and rotates its DOM structure often enough that a scraper written in January can be dead by March. I’ve run job-listing pipelines against Indeed for lead-gen and market-research clients, and the pattern that actually holds up isn’t a clever parser, it’s disciplined request hygiene: real residential IPs, sane concurrency, and a scraper that expects to get blocked and recovers instead of crashing.
This tutorial is for people who need structured job data from Indeed on an ongoing basis: recruiting tool builders, salary/market researchers, competitive intel teams, or anyone building a job aggregator. It’s not for a one-off pull of fifty listings, you can do that by hand. This is for when you need thousands of listings a day, repeatedly, without your IPs getting burned in an afternoon.
By the end you’ll have a working scraper that pulls structured job data (title, company, location, salary range where listed, description, posted date) through rotating residential proxies, with retry logic and pitfalls documented so you’re not debugging blind at 2am. I’m not going to pretend this is risk-free or that Indeed’s terms of service love it, more on that below.
what you need
- Python 3.11+ with
requests,playwright(orselenium), andbeautifulsoup4 - A residential or mobile proxy pool with rotation. Datacenter IPs get flagged almost immediately on Indeed’s search pages. I’ve had the best mileage with Decodo and Smartproxy-style rotating residential pools; budget $8-15 per GB depending on volume, or a flat plan around $75-350/month for sustained pulls
- A headless browser runtime (Playwright with Chromium) for pages that render job cards via JS, not every Indeed page is static HTML
- A queue/storage layer — even a simple SQLite table or Postgres instance to dedupe job IDs across runs, otherwise you re-scrape the same listings every cycle
- A user-agent rotation list — pair proxy rotation with UA rotation, matching one without the other is a fingerprint red flag
- Time budget: expect 2-4 hours to get a stable v1 pipeline running, plus ongoing maintenance whenever Indeed changes its markup (roughly every 6-10 weeks in my experience)
- Legal awareness: read Indeed’s Terms of Service before you start. This is not legal advice, but scraping public job listings for research/aggregation has held up in US courts (see the hiQ v. LinkedIn ruling below), while scraping behind a login wall or violating explicit ToS terms is a different risk profile entirely
step by step
1. Check the robots.txt and map what’s actually public
Before writing a line of scraper code, pull Indeed’s robots.txt. It tells you which paths are disallowed for crawlers and gives you a sense of what Indeed considers off-limits to automated agents. Job search result pages and individual job postings are generally public and indexed by Google, which is the data you want anyway.
Expected output: a list of disallowed paths, mostly account/application flows, not the public job search itself.
If it breaks: if you can’t reach the robots.txt at all, your network or proxy is already being blocked at the DNS/connection level, fix that before going further.
2. Set up a rotating residential proxy pool
Configure your proxy provider’s rotating gateway endpoint rather than a static IP list. Most residential providers give you a single gateway host/port and rotate the exit IP per request or per session automatically.
import requests
proxy_url = "http://user-session-rotate:[email protected]:10000"
proxies = {"http": proxy_url, "https": proxy_url}
resp = requests.get("https://httpbin.org/ip", proxies=proxies, timeout=15)
print(resp.json())
Expected output: a different IP address on each run, confirming rotation is live.
If it breaks: if you get the same IP repeatedly, you’re likely on a “sticky session” mode meant for login flows, switch to per-request rotation in your provider’s dashboard.
3. Build the search URL and paginate
Indeed search URLs follow a predictable pattern: https://www.indeed.com/jobs?q=<role>&l=<location>&start=<offset>. Pagination increments start by 10 or 15 depending on region.
base = "https://www.indeed.com/jobs"
params = {"q": "data engineer", "l": "Singapore", "start": 0}
resp = requests.get(base, params=params, proxies=proxies, headers=headers, timeout=15)
Expected output: HTTP 200 with HTML containing job card divs (class names like job_seen_beacon as of mid-2026, though these change).
If it breaks: a 403 or a CAPTCHA challenge page means your request headers or IP reputation triggered detection. Move to step 4.
4. Add real headers and match them to your proxy’s geography
A rotating proxy from Singapore with a User-Agent claiming to be a US Windows desktop is a mismatch signal. Keep headers consistent with the proxy’s exit region, and use a realistic header set including Accept-Language, Accept-Encoding, and a current Chrome User-Agent string.
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
"(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}
Expected output: fewer 403s and CAPTCHA redirects over a run of 100+ requests.
If it breaks: if CAPTCHAs persist, switch that request to a headless browser session (step 5) since Indeed increasingly gates search results behind JS challenges for suspicious traffic.
5. Fall back to Playwright for JS-rendered pages
Some job postings, especially detail pages with salary estimates, load content client-side. Use Playwright with the same proxy configuration.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(
proxy={"server": "http://gate.decodo.com:10000", "username": "user-session-rotate", "password": "password"}
)
page = browser.new_page()
page.goto("https://www.indeed.com/jobs?q=data+engineer&l=Singapore")
page.wait_for_selector(".job_seen_beacon", timeout=10000)
html = page.content()
browser.close()
Expected output: full rendered HTML including salary badges and expanded job snippets.
If it breaks: if wait_for_selector times out, Indeed likely served a bot-check interstitial instead, log the page title and screenshot it for debugging rather than retrying blind.
6. Parse job cards and dedupe by job ID
Extract the job ID from the card’s data-jk attribute, this is Indeed’s stable identifier and your dedupe key.
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "html.parser")
for card in soup.select(".job_seen_beacon"):
job_id = card.get("data-jk")
title = card.select_one("h2.jobTitle span")
company = card.select_one("[data-testid='company-name']")
print(job_id, title.text if title else None, company.text if company else None)
Expected output: a clean list of (job_id, title, company) tuples with no duplicates across pagination.
If it breaks: if selectors return None for everything, Indeed shipped a markup change, inspect the live page in a browser and update class names.
7. Throttle and randomize request timing
Fixed intervals are a fingerprint. Randomize delays between 3-9 seconds per request per worker, and cap concurrent workers so you’re not hammering one geographic proxy pool.
import random, time
time.sleep(random.uniform(3, 9))
Expected output: sustained scraping over hours without a block spike.
If it breaks: if blocks still climb, reduce concurrency before increasing proxy count, most bans I’ve seen come from request pattern, not just volume.
8. Store results and handle retries
Write to SQLite or Postgres with an upsert on job_id, and wrap each request in a retry with backoff for transient 429s and connection resets.
Expected output: a growing, deduplicated dataset that survives interrupted runs.
If it breaks: if retries stack up on the same job repeatedly, mark it failed after 3 attempts and move on, one stuck listing shouldn’t stall the whole queue.
common pitfalls
- Using datacenter proxies to save money. They’re cheap for a reason, Indeed’s bot detection flags known datacenter ASN ranges almost immediately. Residential or mobile IPs cost more per GB but survive far longer.
- Scraping too fast on a single session. I’ve seen operators fire 50 concurrent requests through one proxy session and get the whole session banned within minutes. Spread load across many rotating sessions instead.
- Ignoring salary parsing edge cases. Salary text on Indeed varies wildly, hourly vs annual, ranges vs single figures, “estimated” vs employer-provided. Build your parser to handle
Nonegracefully rather than crashing the whole batch. - Not handling the CAPTCHA/interstitial case separately from a normal error. Treating a CAPTCHA page as a parse failure and retrying immediately just burns more proxy IPs on the same block. Detect the interstitial by title or a known selector and back off that IP pool specifically.
- Forgetting job postings expire and get reposted. The same role often reappears with a new job_id days later. If you’re tracking market trends, dedupe on a fuzzy match of title+company+location too, not just job_id.
scaling this
At 10x (a few thousand listings a day), a single machine with a rotating residential proxy plan and 3-5 concurrent Playwright workers is plenty. Your bottleneck is proxy bandwidth cost, not compute.
At 100x (tens of thousands a day across multiple search terms/regions), split by region and stagger schedules so you’re not spiking traffic to Indeed’s edge from one time zone all at once. This is also where you want a proper queue (Redis or SQS) instead of an in-memory list, and where session/IP pool segmentation starts mattering: dedicate separate proxy pools per search vertical so a block in one doesn’t cascade.
At 1000x, you’re running a distributed job across multiple machines or containers, likely need a dedicated proxy plan (not shared pool) sized in the hundreds of GB/month, and should build in automatic markup-change detection, alert yourself when parse success rate drops below a threshold (say 90%) rather than discovering it three days later when a client asks why the data went stale. At this scale, also revisit whether Indeed’s official channels (their employer API, or licensed data partners) might be cheaper than scraping infrastructure and maintenance overhead. For pure job-data aggregation, actual licensed feeds sometimes beat scraping once you account for engineer time.
If you’re managing this scraping infra across many isolated environments or accounts, the fingerprint-consistency techniques covered on antidetectreview.org’s blog are worth reading, the same browser-fingerprint matching problems show up whether you’re doing account management or scraping.
where to go next
Indeed shares a lot of anti-bot DNA with other recruiting sites. If you’re building a broader job-market pipeline, read how to scrape Glassdoor at scale in 2026 with proxies that work next, the salary-data parsing problem is nearly identical. For company and recruiter-side data to cross-reference against job postings, best proxies for scraping LinkedIn in 2026 covers a stricter but related target. And if you haven’t picked a proxy vendor yet, our Decodo review breaks down pricing and reliability for exactly this kind of job. Browse the full archive at /blog/ for more scraping targets and vendor comparisons.
One more thing worth reading before you scale this up: the EFF’s writeup on the hiQ v. LinkedIn appeals ruling is the clearest public summary of where US courts have landed on scraping publicly accessible web data. It’s not a green light for everything, and this isn’t legal advice, but it’s useful context before you build a business on top of scraped job listings.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-15.