How to scrape Trustpilot at scale in 2026 with proxies that work
trustpilot is one of the most requested scrape targets I see from operators. agencies want competitor review data, e-commerce teams want sentiment on suppliers, and reputation tools want to track star ratings over time. the pages are public and mostly server-rendered, so a first script works in ten minutes. then you scale to a few thousand business pages and the 403s and captchas start.
this guide is for people who already know basic Python and want a pipeline that survives volume. I run this kind of workload from Singapore, so my numbers come from a home and datacenter mix plus rented residential pools. the outcome is a script that pulls review pages for a list of businesses, rotates proxies sensibly, parses the JSON Trustpilot embeds in each page, and writes clean rows to disk.
one honest caveat up front. I can’t promise a given success rate, because it shifts with Trustpilot’s bot rules and your proxy pool quality. what I can give you is the setup that has held up best for me and the failure modes to watch for.
what you need
- Python 3.11 or newer, with
curl_cffi(for browser-like TLS fingerprints) andselectolaxorbeautifulsoup4for parsing - Playwright for Python as a fallback when plain HTTP gets challenged, see the Playwright Python docs
- a rotating residential proxy plan, billed per GB. review pages are small, so bandwidth cost is modest, but check your provider’s current per-GB rate before you commit
- optionally a small set of mobile or ISP proxies for the pages that residential IPs keep failing on
- a list of Trustpilot business slugs, for example
www.trustpilot.com/review/example.comgives the slugexample.com - a place to store results. SQLite is fine up to millions of rows, Postgres after that
- an hour to read Trustpilot’s robots.txt and terms before you start
on legality: scraping public pages sits in a gray area that depends on your country, what you collect, and what you do with it. reviewer names are personal data under rules like GDPR and Singapore’s PDPA. this is not legal advice, so talk to a lawyer if the data feeds a commercial product. if you only need your own company’s reviews, the official Trustpilot developer docs describe the Business API, and that is the cleaner route.
step by step
1. check robots.txt and decide what you will fetch
open the robots file and read which paths are disallowed. the Robots Exclusion Protocol is standardized in RFC 9309, and it is not a legal shield or a permission slip, but ignoring it is a choice you should make knowingly. I stick to public business review pages and skip anything behind login, user profile pages, and search endpoints.
expected output: a short written list of URL patterns you will and won’t request.
if it breaks: if the file changed since you last looked, re-read it. Trustpilot has edited its rules before and your old assumptions may be stale.
2. set up the project and install dependencies
python -m venv .venv
.venv\Scripts\activate # on Linux/macOS: source .venv/bin/activate
pip install curl_cffi selectolax
pip install playwright && playwright install chromium
expected output: all packages install without errors, and playwright install downloads a Chromium build.
if it breaks: curl_cffi ships wheels for most platforms, but on an unusual Python build it may try to compile. use Python 3.11 or 3.12 from python.org and retry.
3. fetch one page with a browser-like fingerprint
plain requests gets blocked fast because its TLS handshake does not look like Chrome. curl_cffi can impersonate a real browser handshake.
from curl_cffi import requests
PROXY = "http://USER:[email protected]:7000"
def fetch(url: str) -> requests.Response:
return requests.get(
url,
impersonate="chrome",
proxies={"http": PROXY, "https": PROXY},
headers={"Accept-Language": "en-US,en;q=0.9"},
timeout=30,
)
r = fetch("https://www.trustpilot.com/review/example.com")
print(r.status_code, len(r.text))
expected output: 200 and a body of a few hundred KB.
if it breaks: a 403 or a body that mentions a challenge means the IP or fingerprint was flagged. test the same URL from your own home connection without a proxy. if that works, your proxy IP is the problem, not your code.
4. parse the embedded JSON instead of the HTML
Trustpilot is a Next.js site, and review pages have historically carried the page data in a <script id="__NEXT_DATA__"> tag. parsing that JSON is far more stable than chasing CSS classes, which change often.
import json
from selectolax.parser import HTMLParser
def parse_reviews(html: str) -> list[dict]:
tag = HTMLParser(html).css_first("script#__NEXT_DATA__")
if tag is None:
return []
data = json.loads(tag.text())
props = data["props"]["pageProps"]
out = []
for rv in props.get("reviews", []):
out.append({
"id": rv.get("id"),
"rating": rv.get("rating"),
"title": rv.get("title"),
"text": rv.get("text"),
"date": (rv.get("dates") or {}).get("publishedDate"),
"lang": rv.get("language"),
})
return out
I deliberately leave out reviewer names and profile links. you rarely need them for sentiment work, and dropping them cuts your personal data exposure.
expected output: a list of about 20 review dicts per page.
if it breaks: the JSON path changes now and then. print props.keys() and look for where the reviews list moved. if __NEXT_DATA__ is missing entirely, the page was probably a challenge page, so go back to step 3.
5. add pagination
review pages paginate with a page query parameter. loop until a page returns no reviews or you hit your own cap.
def scrape_business(slug: str, max_pages: int = 25) -> list[dict]:
rows = []
for page in range(1, max_pages + 1):
url = f"https://www.trustpilot.com/review/{slug}?page={page}&languages=all"
r = fetch(url)
if r.status_code != 200:
break
batch = parse_reviews(r.text)
if not batch:
break
rows.extend(batch)
return rows
expected output: all reviews up to your page cap for one business.
if it breaks: if you only get English reviews, confirm the languages=all parameter is still honored. if page 2 returns the same reviews as page 1, check that your proxy is not serving you a cached response.
6. rotate proxies per business, not per request
most residential providers give you a single gateway that rotates the exit IP on each connection, or a sticky session you control with a username suffix. for Trustpilot I get better results with sticky sessions. one business means one IP for its whole page run, then a fresh IP for the next business. a single IP hopping between 25 pages looks less human than one IP reading them in order.
import random, string
def sticky_proxy(base_user: str, password: str) -> str:
sid = "".join(random.choices(string.ascii_lowercase + string.digits, k=8))
# session syntax differs by provider, check your docs
return f"http://{base_user}-session-{sid}:{password}@gate.yourprovider.com:7000"
expected output: each business is fetched through a different exit IP, visible if you log the IP from a test endpoint.
if it breaks: session syntax is provider specific. if every request returns the same IP or a different one each time, you have the wrong username format. read your provider’s sticky session docs. for more on this topic see how to rotate proxies in Python.
7. add retries, backoff and a circuit breaker
treat 403, 429 and challenge bodies as signals. retry on a new IP with exponential backoff, and stop hammering when a whole batch fails.
import time
def fetch_with_retry(url: str, tries: int = 4):
for attempt in range(tries):
r = fetch(url)
if r.status_code == 200 and "__NEXT_DATA__" in r.text:
return r
time.sleep(2 ** attempt + random.random())
return None
add a counter: if more than 30 percent of the last 50 requests failed, pause the worker for several minutes. burning through a pool that is already flagged just costs you bandwidth.
expected output: failures cluster in logs and then recover instead of snowballing.
if it breaks: if retries never succeed, your whole pool may be burned for this target. switch the pool or provider type, see the pitfalls section.
8. fall back to a real browser for stubborn pages
when HTTP-only fetches get challenged, a headless Chromium through the same proxy often passes.
from playwright.sync_api import sync_playwright
def fetch_browser(url: str, proxy_server: str, user: str, pw: str) -> str:
with sync_playwright() as p:
b = p.chromium.launch(proxy={"server": proxy_server, "username": user, "password": pw})
page = b.new_page()
page.goto(url, wait_until="domcontentloaded")
html = page.content()
b.close()
return html
expected output: full page HTML including the __NEXT_DATA__ script.
if it breaks: browsers are 10 to 20 times heavier than HTTP calls, so only use this as a fallback. if even Chromium is blocked, the fingerprint of your browser profile may be the cause. antidetect browsers address that, and antidetectreview.org compares the main options.
9. store results and dedupe
write to SQLite with the review id as the primary key so reruns update rather than duplicate.
import sqlite3
db = sqlite3.connect("trustpilot.db")
db.execute("""CREATE TABLE IF NOT EXISTS reviews(
id TEXT PRIMARY KEY, slug TEXT, rating INT, title TEXT,
text TEXT, date TEXT, lang TEXT)""")
def save(slug, rows):
db.executemany(
"INSERT OR REPLACE INTO reviews VALUES(?,?,?,?,?,?,?)",
[(r["id"], slug, r["rating"], r["title"], r["text"], r["date"], r["lang"]) for r in rows],
)
db.commit()
expected output: row counts that match what you see on the live page for a spot-checked business.
if it breaks: if your count is lower than the displayed total, some reviews may be filtered or on pages beyond your cap. raise max_pages and re-check.
common pitfalls
- using datacenter proxies for everything: Trustpilot sits behind bot protection, and datacenter ranges are the first thing flagged. use residential for the bulk and keep datacenter for testing your parsing only.
- rotating the IP on every single request: this breaks the natural pattern of one visitor paging through reviews. use sticky sessions per business.
- parsing HTML classes: class names on modern React sites are generated and change. the embedded JSON survives redesigns better, though not forever.
- ignoring the bandwidth bill: a retry loop against a flagged pool can eat gigabytes of paid residential traffic with nothing to show. add the circuit breaker before you scale, not after.
- storing everything you can: reviewer names, avatars and profile URLs add legal exposure and no analytic value for most use cases.
scaling this
at 10x (a few hundred businesses), one machine and one residential plan is enough. run 5 to 10 worker threads, log status codes per run, and watch success rate. a plain SQLite file handles the output.
at 100x (tens of thousands of business pages), move to a queue such as Redis plus several worker processes. track success rate per proxy source and per hour. this is where I start mixing in a second proxy provider, because pools differ by region and by time of day. add jitter between requests and cap concurrency per exit country. if you run many accounts or sessions in parallel, the operational side matters as much as the code, and multiaccountops.com covers that workflow.
at 1000x (hundreds of thousands of pages, regular refreshes), the problem changes from “can I fetch” to “should I refetch”. do incremental scraping: fetch only the newest page of each business first, and only paginate deeper if you see unseen review IDs. that cuts requests by a large factor. at this size you also want a proper proxy health dashboard, and you will probably pay for a premium residential tier or mobile proxies for the hardest slice of targets. budget a person-day a week for maintenance, because selectors, JSON paths and bot rules drift.
where to go next
- residential vs mobile proxies for scraping: when the extra cost of mobile IPs is worth it
- how to rotate proxies in Python: sticky sessions, pools and health checks in more detail
- the full proxyscraping.org article index: other target-specific scraping guides
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-10-19.