How to scrape Realtor.com at scale in 2026 with proxies that work
Realtor.com is one of the harder US real estate sites to scrape reliably. It runs on a Next.js frontend with server-rendered listing pages, layers in bot detection that fingerprints headless browsers, and rate-limits aggressively by IP the moment you go past a handful of requests per minute from the same address. If you’ve ever pointed a plain requests script at a search results page and gotten back a 403 or a page full of nothing but a loading skeleton, you’ve already met these defenses.
This tutorial is for people building a real pipeline, not a one-off scrape: real estate investors tracking price changes across zip codes, data teams feeding a comps model, or agencies building a listing aggregator. I’m going to walk through the actual mechanics of getting Realtor.com data out reliably at volume, including proxy selection, request shaping, and the failure modes you’ll hit once you’re running more than a trickle of traffic.
One thing up front: scraping a site’s publicly viewable data is generally treated differently under US law than bypassing a technical access control, but Realtor.com’s terms of use restrict automated data collection, and enforcement of those terms is a civil contract matter, separate from criminal computer-access law. This is not legal advice, consult a lawyer if you’re building something commercial on top of this data. What follows is the technical how-to.
what you need
- A scripting environment. Python 3.11+ with
playwright(for JS-rendered pages) andhttpxorrequests(for lighter API-style calls). Node with Playwright works equally well if that’s your stack. - A rotating residential or ISP proxy pool. Realtor.com blocks datacenter IP ranges within minutes of sustained traffic. Budget roughly $4-15 per GB for residential proxy plans from providers like Decodo, Bright Data, or Oxylabs, depending on volume tier. See our Decodo review if you’re picking a vendor.
- A headless browser runtime. Realtor.com’s search and listing pages hydrate via JavaScript; raw HTML fetches miss most of the listing data. Playwright with Chromium is the standard choice in 2026.
- A place to store output. Postgres or SQLite for structured listing records, plus S3 or local disk if you’re archiving photos.
- A CAPTCHA-handling plan. Either a solving service (2Captcha, CapSolver) budgeted at a few cents per solve, or a backoff strategy that avoids triggering CAPTCHAs in the first place by staying under rate thresholds.
- Time to tune. Expect a week of iteration before your success rate stabilizes above 95%. Anti-bot systems change their fingerprinting checks periodically, so this isn’t a set-and-forget script.
step by step
1. Map the data you actually need
Before writing code, define your schema: address, price, beds, baths, square footage, listing status (active/pending/sold), days on market, agent name, and photo URLs are the common fields. Realtor.com listing pages embed a large JSON object (__NEXT_DATA__) in the page source that contains most of this in structured form, which is far more reliable to parse than scraping rendered HTML with CSS selectors.
Expected output: a field list and a target JSON schema you’ll map scraped data into.
If it breaks: if you’re not sure which fields are available, open a listing page in a browser, view source, and search for __NEXT_DATA__ to see the full payload structure before you start scraping at volume.
2. Set up your environment and proxy pool
Install Playwright and a proxy-aware HTTP client:
pip install playwright httpx
playwright install chromium
Configure your proxy pool credentials as environment variables rather than hardcoding them:
export PROXY_HOST="gate.yourprovider.com"
export PROXY_PORT="7000"
export PROXY_USER="youruser"
export PROXY_PASS="yourpass"
Test that a single proxy request resolves correctly before touching Realtor.com:
curl -x http://$PROXY_USER:$PROXY_PASS@$PROXY_HOST:$PROXY_PORT https://ifconfig.me
Expected output: the curl command returns an IP address that isn’t your own, confirming the proxy tunnel works.
If it breaks: a timeout usually means the proxy port or auth format is wrong, check your provider’s docs for whether they expect user:pass in the URL or a separate header. A 407 response means the IP making the request isn’t whitelisted if you’re on IP-auth instead of user/pass auth.
3. Build the rendering layer with Playwright
Realtor.com’s listing pages need JS execution to populate data. A minimal fetch-and-parse function looks like this:
from playwright.sync_api import sync_playwright
import json
def fetch_listing(url, proxy):
with sync_playwright() as p:
browser = p.chromium.launch(
proxy={"server": f"http://{proxy['host']}:{proxy['port']}",
"username": proxy["user"], "password": proxy["pass"]},
headless=True,
)
page = browser.new_page(
user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
)
page.goto(url, wait_until="networkidle", timeout=30000)
html = page.content()
browser.close()
return html
Parse the __NEXT_DATA__ script tag out of the returned HTML with a JSON parser rather than trying to scrape rendered DOM elements, it’s more stable across frontend redesigns.
Expected output: a full HTML payload per listing URL containing embedded JSON with price, address, and property details.
If it breaks: if page.content() returns a near-empty shell, the JS hydration didn’t finish, try wait_until="load" combined with an explicit page.wait_for_selector() on a known listing element instead of relying purely on networkidle.
4. Handle search and pagination
Realtor.com search results are paginated and filterable by location, price range, and property type via URL query parameters. Iterate through zip codes or city slugs rather than trying to paginate a single massive nationwide query, this keeps individual request sessions smaller and easier to distribute across proxy sessions.
Expected output: a queue of listing URLs generated per search page, typically 40-50 results per page of a Realtor.com search.
If it breaks: if pagination stops returning new results before you expect it to, check whether the site capped results at a max page depth for that search (common past ~20 pages), narrow your filters further (smaller price bands, single zip codes) to get under the cap.
5. Rotate identities: proxies, user agents, and headers together
Don’t rotate just the IP. Realtor.com’s bot detection correlates IP, user agent, TLS fingerprint, and browser automation signals together. Rotate proxy sessions per N requests (sticky sessions of 5-10 requests per IP tend to work better than a fresh IP every single request, which itself looks anomalous), and vary user agent strings and viewport sizes across a realistic pool of real browser signatures.
Expected output: a request pattern that looks like distinct organic users rather than one script cycling IPs.
If it breaks: if you’re getting blocked despite rotating IPs, the issue is usually a static Playwright fingerprint. Add playwright-stealth or equivalent patches to mask automation-detectable properties like navigator.webdriver.
6. Rate-limit deliberately
Add randomized delays between requests, 3-8 seconds is a reasonable starting range per session, not per proxy pool. Going faster than that per IP is the single fastest way to get flagged, even with good fingerprint hygiene.
import random, time
def polite_delay():
time.sleep(random.uniform(3, 8))
Expected output: a steady, sustainable request rate with a low block rate over hours of continuous scraping.
If it breaks: if your success rate degrades over a session even with delays, your proxy provider’s IP pool may be too small or already flagged by Realtor.com, check pool size and rotation health with your provider before assuming the code is at fault.
7. Parse, deduplicate, and store
Parse the extracted JSON into your schema, deduplicate on Realtor.com’s internal listing ID (not the URL, which can change), and upsert into your database. Track a last_seen timestamp per listing so you can detect price changes and delistings over time.
Expected output: a growing, deduplicated table of listings with historical price tracking.
If it breaks: if you’re seeing duplicate rows, confirm you’re keying on the listing ID field inside the JSON payload rather than the URL slug, Realtor.com URLs can include tracking parameters that make identical listings look like different rows.
8. Monitor block rates and retry with backoff
Log every response code and CAPTCHA occurrence per proxy session. Build exponential backoff for retries, and pull a session out of rotation for a cooldown period if it’s throwing repeated 403s.
Expected output: a dashboard or log showing block rate trending near zero, with automatic recovery when it spikes.
If it breaks: if block rates spike suddenly across your whole pool, Realtor.com likely shipped a detection update. Check your Playwright and stealth-plugin versions are current, this is the most common cause of a sudden across-the-board failure.
common pitfalls
- Scraping too fast per proxy. The most common mistake I see is treating proxy rotation as a substitute for rate limiting instead of a complement to it. A large pool at high speed still gets flagged if per-IP request rate is unrealistic.
- Ignoring the mobile/API endpoints. Some operators miss that Realtor.com’s app-facing API endpoints sometimes return cleaner structured data with less rendering overhead than scraping the public web pages, at the cost of being a less stable target since it’s undocumented.
- Not handling listing status changes. Listings go pending or sold and can disappear from search results entirely. If your pipeline only tracks “found” listings without periodic re-checks, your dataset silently goes stale.
- Skipping fingerprint hygiene. Rotating IPs while leaving default Playwright/Selenium automation flags exposed is a fast way to get blocked regardless of proxy quality. If you’re evaluating antidetect browser tooling for this, antidetectreview.org’s blog covers fingerprint-masking tools in more depth than I will here.
- Storing raw HTML instead of structured JSON. Pulling data from rendered DOM selectors breaks every time Realtor.com ships a frontend update. Parsing the embedded JSON payload is materially more durable.
scaling this
At 10x (a few thousand listings a day), a single machine running Playwright with a modest residential proxy plan (10-20GB/month) and simple sequential logic is enough. Concurrency of 3-5 browser contexts keeps you well under detection thresholds.
At 100x (tens of thousands of listings a day), you need a proper job queue (Celery, RQ, or similar) distributing work across multiple worker processes, each with its own sticky proxy session pool. You’ll also want to switch from full browser rendering to hitting Realtor.com’s internal JSON endpoints directly where possible, since spinning up a headless browser per request doesn’t scale cost-effectively at this volume. Proxy spend becomes a real line item, budget accordingly.
At 1000x (hundreds of thousands to millions of records), you’re running a distributed scraping infrastructure: multiple proxy vendors for redundancy, geographically distributed workers, a dedicated CAPTCHA-solving budget, and monitoring that pages you when block rates cross a threshold. At this scale it’s also worth evaluating whether a commercial real estate data API (Realtor.com has partner data feeds for some use cases) is cheaper than maintaining scraping infrastructure. Self-scraping at this volume is an ongoing engineering commitment, not a script you write once.
where to go next
If you’re building a broader real estate or business data pipeline, see our guide on scraping Crunchbase at scale for company and funding data that often pairs well with property records. For proxy vendor selection specifically, the Decodo review breaks down pricing and pool size in more detail than fits here. And if you’re scraping other listing-heavy sites, our Booking.com scraping tutorial covers similar pagination and anti-bot patterns you’ll recognize. For the full archive, browse the blog index.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-16.