How to scrape Yellow Pages at scale in 2026 with proxies that work
Yellow Pages is still one of the largest structured local business directories on the open web. Name, address, phone, category, sometimes a website link, all sitting in predictable HTML across millions of listings. That makes it a solid source for lead lists, market mapping, and competitor research. It also makes it a target that gets rate-limited fast if you point a script at it from a single IP and start hammering pages.
This is for operators building local business datasets: agencies doing lead gen for clients, people validating a market before launching a service in it, or anyone who needs structured directory data that isn’t sitting behind an expensive API. It’s not for reselling scraped data in bulk against a site’s terms of service, and I’m not going to pretend otherwise, so read the section on legal footing before you scale this past a few hundred rows.
By the end of this you’ll have a working Python scraper that pulls business name, phone, address, and category from Yellow Pages search results, rotates through proxies so you don’t get blocked after the first fifty requests, and writes clean output you can grow from a test run of a few hundred listings to a job pulling hundreds of thousands.
what you need
- Python 3.10 or newer
requests,beautifulsoup4, andlxml(all pip-installable)- A rotating proxy pool. Residential proxies from a provider like Decodo, Smartproxy, or Oxylabs work here, datacenter proxies get flagged faster on directory sites that watch for repeat traffic patterns
- Somewhere to store output: a CSV file for small runs, Postgres once you’re past a few thousand rows
- A few hours for the first working version, longer if you’re also building dedupe and monitoring
- Optional: Playwright, only if you end up needing JS-rendered detail pages beyond the search results
- Budget: expect somewhere in the $50 to $300 a month range for proxy bandwidth depending on volume, scale up from there once you know your success rate
step by step
1. Scope your target and check the rules
Decide exactly which categories and geographies you’re pulling before you write a line of code. Then check yellowpages.com’s robots.txt and their terms of service. Robots.txt doesn’t carry legal force by itself, it’s a signal of what the site operator wants automated agents to avoid, but ignoring it changes your risk profile if anyone ever asks why your traffic pattern looked like a bot.
Expected output: a written list of the URL paths you plan to hit and which ones the site’s own rules flag as off-limits.
If it breaks: if your target paths are disallowed, dial back to lower volume and longer delays, or talk to a lawyer if this is for a commercial product. This isn’t legal advice, the line between “allowed” and “risky” on scraping shifts by jurisdiction and by how a court reads cases like hiQ Labs v. LinkedIn, which is worth reading if you want the actual legal reasoning instead of forum takes.
2. Set up your environment
python -m venv ypscraper
source ypscraper/bin/activate # Windows: ypscraper\Scripts\activate
pip install requests beautifulsoup4 lxml
Expected output: a clean virtual environment with the three packages installed and importable.
If it breaks: version conflicts between lxml and your Python version are the usual culprit on fresh installs. Pin versions in a requirements.txt once your scraper is stable so a future pip install doesn’t silently upgrade something and break your selectors.
3. Map the search results URL pattern
Yellow Pages search URLs follow a predictable pattern: https://www.yellowpages.com/search?search_terms=plumbers&geo_location_terms=Austin%2C+TX, with &page=2 and so on for pagination. Run a search manually in the browser first and copy the resulting URL rather than guessing the query parameter names.
Expected output: loading page 1 in your browser (not your scraper yet) shows a results page with individual business cards.
If it breaks: if geo_location_terms returns zero results, the format is usually off, check the exact string the site’s own search box produces before assuming your code is wrong.
4. Write the parser
import requests
from bs4 import BeautifulSoup
HEADERS = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}
def fetch_search_page(term, location, page=1, proxy=None):
url = "https://www.yellowpages.com/search"
params = {"search_terms": term, "geo_location_terms": location, "page": page}
proxies = {"http": proxy, "https": proxy} if proxy else None
resp = requests.get(url, params=params, headers=HEADERS, proxies=proxies, timeout=15)
resp.raise_for_status()
return resp.text
def parse_listings(html):
soup = BeautifulSoup(html, "lxml")
results = []
for card in soup.select("div.result"):
name = card.select_one("a.business-name")
phone = card.select_one("div.phones")
address = card.select_one("div.street-address")
category = card.select_one("div.categories")
results.append({
"name": name.get_text(strip=True) if name else None,
"phone": phone.get_text(strip=True) if phone else None,
"address": address.get_text(strip=True) if address else None,
"category": category.get_text(strip=True) if category else None,
})
return results
Expected output: a list of dicts, one per business card on the page, with name/phone/address/category filled in where present.
If it breaks: directory sites revise their frontend markup periodically. Save a raw HTML sample to disk every time you run this so you can diff it against a failed parse instead of guessing what changed.
5. Add proxy rotation
A single IP making sequential requests to a directory site gets rate-limited or blocked within minutes. Rotate through a pool.
import random
import time
PROXY_POOL = [
"http://user:[email protected]:10001",
"http://user:[email protected]:10002",
"http://user:[email protected]:10003",
]
def fetch_with_retry(term, location, page, max_retries=3):
for attempt in range(max_retries):
proxy = random.choice(PROXY_POOL)
try:
html = fetch_search_page(term, location, page, proxy=proxy)
time.sleep(random.uniform(2, 5))
return html
except requests.exceptions.HTTPError as e:
if e.response.status_code in (429, 503):
time.sleep(2 ** attempt * 3)
continue
raise
raise RuntimeError(f"failed after {max_retries} retries: {term}/{location}/page{page}")
Expected output: a run of 100+ sequential page fetches with a success rate consistently above 95%.
If it breaks: sticky sessions can leak your real IP through cookie or TLS fingerprint mismatches even with a proxy set. Most rotating proxy vendors give you a single gateway endpoint that rotates on their end, use that instead of hand-managing a small static list once you’re past testing. If you’re chasing detection issues specifically, antidetectreview.org covers browser fingerprinting in more depth than I will here.
6. Throttle and retry deliberately
Don’t treat this like an API. Add randomized delays between requests and exponential backoff on 429 or 503 responses, both already in the snippet above.
Expected output: a stable success rate that doesn’t degrade the longer the job runs.
If it breaks: if your failure rate stays high even with a healthy proxy pool, you’re likely tripping a behavioral pattern, not just an IP-based block, meaning the requests look too uniform in timing or headers. Randomize delay windows further and vary the User-Agent string across a small realistic set instead of hardcoding one.
7. Handle detail pages only when you need to
Search result cards usually give you name, phone, address, and category. If you need the business’s own website URL or hours, you’ll need to follow through to the individual listing page, which doubles your request volume.
Expected output: enriched records with a website field populated for listings that have one.
If it breaks: detail page markup differs from the search card markup, so write a separate parse function rather than trying to reuse the same selectors.
8. Store and dedupe
Write to CSV for a first run, move to Postgres once volume climbs. Dedupe by the listing ID embedded in the business’s URL, not by name and address, since the same business often shows up under multiple categories with slightly different address formatting.
Expected output: a clean output file with one row per unique business.
If it breaks: if your row count looks inflated, check for the same phone number appearing multiple times, that’s the fastest signal of a dedupe key that’s too loose.
9. Monitor and log failures
Log every non-200 response with the URL and which proxy handled it. Assert on a minimum set of expected fields (at least name and one of phone/address) before writing a row, rather than writing partial garbage silently.
Expected output: a log file that tells you exactly where and why a run degraded.
If it breaks: silent failures are the expensive kind, you find out three days later that half your dataset is empty rows. Build the assertion in from day one.
common pitfalls
- No proxy rotation at all. Testing from your home IP feels fine until you get blocked mid-run and can’t tell if it’s your code or the block causing empty results.
- Ignoring last-page detection. Requesting page 40 when there are only 12 pages of results either loops forever or silently returns duplicate content. Check for an empty results container and stop.
- Skipping dedupe. Businesses listed under multiple categories inflate your lead count and make client-facing lists look sloppy.
- Hardcoding selectors without monitoring. Directory sites update their markup a few times a year. A scraper that worked in January can silently return empty fields by June if nobody’s watching for it.
- Going too fast because it “worked” in testing. A 20-request test run succeeding doesn’t mean a 20,000-request production run will, the block triggers usually kick in on volume and pattern, not the first request.
scaling this
10x (hundreds to low thousands of listings): a single machine, synchronous requests + BeautifulSoup, a small rotating proxy pool, CSV output. This is what the code above handles fine as-is.
100x (tens of thousands): move to async requests with httpx and asyncio, or switch to Scrapy if you want built-in retry and pipeline handling. Increase your proxy pool size and concurrency together, and move storage to Postgres so you’re not rewriting a giant CSV on every run. Start tracking block rate as a metric, not just success/fail counts.
1000x (hundreds of thousands and up): distribute the work across multiple worker processes with a job queue (Redis works fine for this), budget proxy spend scaling into hundreds of dollars a month rather than tens, and add automated QA sampling that flags when a random sample of scraped records starts coming back with empty fields, that’s your early warning for selector drift instead of finding out from a client complaint. At this scale it’s also worth pricing out a managed scraping API against the cost of babysitting your own infrastructure, sometimes the math favors paying someone else to handle the block-and-retry problem.
where to go next
Once the proxy and rotation logic here is working, the same pattern applies to other directory and local-search targets, worth reading best proxies for scraping local search results in 2026 next for a comparison of providers beyond the one used in the code above. If you want the full vendor breakdown behind the proxy pool in this tutorial, see the Decodo review. For the general index of scraping tutorials on this site, start at /blog/.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-17.