← all guides

How to scrape Tripadvisor at scale in 2026 with proxies that work

Tripadvisor is one of the harder travel sites to pull data from once you go past a few hundred pages. A script that works on 20 restaurant pages from your laptop will start returning captcha walls or empty shells somewhere around page 200 to 500, because the site fingerprints datacenter ranges and watches request rhythm per IP. I’ve run into this on hotel and restaurant monitoring jobs from Singapore, and the fix is almost never “more threads”. It’s better IP quality, slower pacing per identity, and a pipeline that doesn’t re-fetch what it already has.

This guide is for operators who need listing data (name, rating, review count, address, price level) and sometimes review text across a few thousand to a few hundred thousand pages. If you only need a small amount of data, check the official API first, because it’s cheaper than any proxy setup.

By the end you’ll have a working Playwright scraper behind rotating residential or mobile proxies, with sticky sessions, retries, a disk cache and a way to measure success rate per proxy. I’ll also say where I think the whole thing stops being worth it.

what you need

  • python 3.11 or newer with playwright and httpx installed
  • a proxy source: residential or mobile, with country targeting and sticky sessions. datacenter IPs get flagged fast on this target in my experience
  • a list of target URLs, or a seed (a city page like a “restaurants in Singapore” index) to crawl from
  • a place to store raw HTML: local disk is fine up to a few hundred thousand pages, S3-compatible storage beyond that
  • sqlite or postgres for parsed output
  • a budget. residential bandwidth is billed per GB and mobile is usually billed per port or per GB, depending on the provider. a rendered Tripadvisor page can pull a lot of weight if you let images and scripts load, so blocking those matters for cost. check your provider’s current price sheet
  • an honest read of the rules. Tripadvisor publishes a robots.txt and terms of use, and scraping can raise contract and data protection questions, especially for reviewer names. this is not legal advice. if the data feeds a commercial product, talk to a lawyer

One more option before you build anything: Tripadvisor runs an official Content API that returns location details, ratings and a limited set of reviews. If it covers your fields, use it and skip the rest of this article.

step by step

1. check what you can get without a browser

Start with plain HTTP and see what comes back. Many listing pages embed structured data in a JSON-LD block, which is far cheaper to parse than the rendered DOM.

curl -s -A "Mozilla/5.0" "https://www.tripadvisor.com/Restaurant_Review-g294265-d1234567-Reviews-Example.html" \
  | grep -o '<script type="application/ld+json">.*</script>' | head -c 600

Expected output: either a JSON-LD snippet with aggregateRating, or nothing and a captcha or 403 page in the raw body.

If it breaks: a 403 from your own IP is normal and tells you the site is checking. move on to a browser plus proxy. don’t keep hammering from your home connection, you’ll just burn it.

2. pick the proxy type and test it before buying volume

Test three things on a small sample: success rate, median time per page, and bytes per page. Run 100 URLs through each candidate pool and log the status codes. In my testing, mobile and clean residential pools pass noticeably more often than datacenter, but your numbers will differ by country and provider, so measure instead of trusting my feel.

If you’re running your own modems, a 4G port with IP rotation on demand works well here because carrier-grade NAT puts you in a crowd of real users. If you’re buying, ask for sticky sessions of 5 to 30 minutes and country or city targeting.

If it breaks: if success is under roughly 70% on a 100 URL sample, change the pool or the country before you touch your code. the problem is almost always the IP, not the parser.

3. set up playwright with a proxy per context

Give each browser context its own proxy session so cookies and IP stay paired. Mixing one cookie jar across several IPs is one of the quickest ways to get flagged. The Playwright proxy docs cover the options.

import asyncio, random
from playwright.async_api import async_playwright

PROXY = {
    "server": "http://gate.example-proxy.com:8000",
    "username": "user-session-{sid}-country-sg",
    "password": "secret",
}

async def new_context(browser, sid):
    p = dict(PROXY, username=PROXY["username"].format(sid=sid))
    ctx = await browser.new_context(
        proxy=p,
        locale="en-SG",
        timezone_id="Asia/Singapore",
        viewport={"width": 1366, "height": 768},
    )
    # block heavy assets to save proxy bandwidth
    await ctx.route("**/*", lambda r: r.abort()
        if r.request.resource_type in ("image", "media", "font")
        else r.continue_())
    return ctx

Expected output: a context that loads a page and reports the proxy’s IP when you hit an IP echo service.

If it breaks: if the echo shows your real IP, your proxy credentials are being ignored. check the username format your provider expects for sticky sessions. also match locale and timezone to the proxy country, since a Singapore IP with a US timezone is an easy tell. if you need deeper fingerprint control, the antidetectreview.org blog compares browser tools for exactly this.

4. fetch one page and parse the structured data

import json, re

async def fetch(ctx, url):
    page = await ctx.new_page()
    resp = await page.goto(url, wait_until="domcontentloaded", timeout=45000)
    html = await page.content()
    await page.close()
    return resp.status, html

def parse_ldjson(html):
    out = []
    for m in re.finditer(r'<script type="application/ld\+json">(.*?)</script>', html, re.S):
        try:
            out.append(json.loads(m.group(1)))
        except json.JSONDecodeError:
            pass
    return out

Expected output: status 200 and a list with at least one object containing name, aggregateRating and address.

If it breaks: status 200 but no JSON-LD usually means you got a challenge page. save the HTML and open it. if it’s a challenge, rotate the session and slow down. if the layout changed, update your selectors, because Tripadvisor changes its markup often enough that I treat parsers as disposable.

5. cache every response to disk

This is the step that saves the most money. Store raw HTML keyed by a hash of the URL, and check the cache before every request. Parsing bugs then cost you CPU, not proxy bandwidth. I wrote about this in more detail in caching responses so a rerun does not rehit the site.

import hashlib, pathlib

CACHE = pathlib.Path("cache"); CACHE.mkdir(exist_ok=True)

def cache_path(url):
    return CACHE / (hashlib.sha1(url.encode()).hexdigest() + ".html")

def get_cached(url):
    p = cache_path(url)
    return p.read_text(encoding="utf-8") if p.exists() else None

Expected output: a second run over the same URLs makes zero network requests.

If it breaks: if the cache fills with challenge pages, you’ll keep “hitting” bad data forever. only write to cache when the status is 200 and the parser finds the fields you expect.

6. add pacing, sticky sessions and retries

Run a small pool of workers, each with its own context and sticky session, and add jitter between requests. I start at 2 to 4 concurrent contexts and 3 to 8 seconds between pages per context, then raise gradually while watching success rate.

async def worker(browser, urls, results):
    sid = random.randint(10**6, 10**7)
    ctx = await new_context(browser, sid)
    fails = 0
    for url in urls:
        if (html := get_cached(url)):
            results.append((url, 200, html)); continue
        status, html = await fetch(ctx, url)
        if status == 200 and parse_ldjson(html):
            cache_path(url).write_text(html, encoding="utf-8")
            results.append((url, status, html)); fails = 0
        else:
            fails += 1
            if fails >= 2:           # burn this identity, start a new one
                await ctx.close()
                sid = random.randint(10**6, 10**7)
                ctx = await new_context(browser, sid); fails = 0
        await asyncio.sleep(random.uniform(3, 8))
    await ctx.close()

Expected output: steady 200s, with a rotation after two bad responses in a row.

If it breaks: if you rotate constantly and still fail, the pool is burned for this target. stop and switch pool. rotating faster on a bad pool just spends bandwidth.

7. crawl review pages the careful way

Review text lives on paginated URLs. Fetch the first page of reviews for each location, read the total count, and only then queue the remaining pages. Don’t queue every page up front, because many locations have far fewer reviews than you’d guess and you’ll waste requests on empty pages.

If you collect reviewer names or profile links, you’re handling personal data. Drop what you don’t need. the EU’s GDPR text on EUR-Lex is the primary source if you’re unsure what counts.

Expected output: a queue sized to the real review counts.

If it breaks: duplicate reviews across pages mean the page offset is shifting between requests. dedupe on review id, not on position.

8. log success rate per proxy and per hour

Write one line per request to sqlite: timestamp, session id, country, status, bytes, seconds. Then look at it.

SELECT strftime('%H', ts) AS hr, COUNT(*) AS n,
       ROUND(AVG(status = 200) * 100, 1) AS ok_pct,
       ROUND(AVG(bytes) / 1024.0) AS avg_kb
FROM requests GROUP BY hr ORDER BY hr;

Expected output: a table where you can see which hours and which pools perform. you’ll often find a pool that looks fine overall but collapses at certain hours.

If it breaks: if avg_kb is huge, your asset blocking isn’t working. check the route handler, since that’s where most of your proxy bill leaks.

common pitfalls

  • running datacenter IPs and blaming the code. on this target I’ve had far better luck with residential and mobile, and swapping the pool fixed more “bugs” than any parser change did
  • sharing cookies across IPs. one context equals one identity equals one sticky session
  • loading images, fonts and video through a metered proxy. blocking them cut my per-page transfer sharply
  • caching failures. a cache full of challenge pages makes a rerun look successful while the data is garbage
  • scraping reviewer personal data you never use. collect the minimum, and keep a note of why you hold each field

scaling this

From 10x to 100x (a few thousand to tens of thousands of pages), the code barely changes. what changes is discipline: cache everything, log per-session success, and keep concurrency low per IP. one box with a handful of contexts is enough.

From 100x to 1000x (hundreds of thousands of pages), the bill and the failure modes shift. You’ll want a job queue (Redis or a plain postgres table with a status column), multiple workers across machines, and proxy pools split by country so each identity stays consistent. Bandwidth becomes your main cost, so measure kilobytes per successful page and treat any rise as a bug. You’ll also need monitoring that alerts when success rate falls below a threshold, because a silent drop at 3am will quietly fill a queue with failures. Managing many identities at this size is its own discipline, and multiaccountops.com covers the session and account hygiene side well.

Past that point, ask whether you should be scraping at all. Refresh frequency matters: if you need monthly data on a fixed set of locations, a slow crawl spread over weeks is cheaper and safer than a burst. If you need broad, high-volume data, a licensed data provider or the official API may cost less than the proxy bandwidth and engineering time.

where to go next

Standards context, if you want it: the Robots Exclusion Protocol is now RFC 9309, which is worth reading so you know what a robots.txt file does and doesn’t mean.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-10-18.

proxies
Need proxies that survive the block wall?

Singapore Mobile Proxy runs real 4G/5G mobile IPs on rotating SIMs — the carrier-grade addresses most of these targets still trust.

see plans →
read on
More scraping guides

The rest of the field manual: target-site playbooks, library walkthroughs, provider reviews, and anti-bot troubleshooting.

browse all guides →