How to scrape Kayak at scale in 2026 with proxies that work
Kayak is one of the harder travel sites to collect from. Prices are rendered by JavaScript, results load in waves while a search “completes”, and the bot defenses punish anything that looks like a datacenter box hammering search URLs. If you have tried a plain requests loop from a VPS, you already know how that ends: a captcha wall within a few dozen searches, then a block page that never goes away.
This tutorial is for operators who need fare or hotel price snapshots on a schedule. Think price trackers, route monitoring for a small travel business, or research datasets. I run proxy infrastructure out of Singapore, so the examples lean on routes I actually care about (SIN to BKK, SIN to HND), but nothing here is region specific.
By the end you will have a working Playwright setup that routes through a rotating proxy pool, paces itself, detects blocks, and writes clean rows to disk. I will also be honest about where it breaks and what changes when you go from hundreds of searches a day to hundreds of thousands. This is not legal advice. Kayak’s terms of use restrict automated access, so read them and talk to a lawyer before you build a business on this data.
what you need
- a Linux box or home server with Python 3.11+ (a 4 GB RAM VPS runs about 4 to 6 concurrent browser contexts comfortably)
- Playwright for Python, installed with
pip install playwrightandplaywright install chromium - a proxy source: mobile (4G/5G) or clean residential IPs with sticky sessions. mobile is the most forgiving on Kayak in my experience, residential is the cheaper middle ground
- budget: residential bandwidth typically runs a few dollars per GB from the big providers, and a Kayak search page with assets can pull 2 to 5 MB if you do not block images and fonts. mobile ports usually cost more per port but include far more bandwidth, so do the math on your own volume
- a place to store results: SQLite is enough up to a few million rows, Postgres after that
- a list of searches: origin, destination, date pairs, passenger count
- 2 hours for the first working run, a weekend to make it stable
I will not recommend a specific vendor here because the right pick depends on your volume and geography. I cover that comparison in best mobile proxies for travel scraping.
step by step
1. check robots.txt and decide what you actually need
Before writing code, read what the site asks of crawlers. The Robots Exclusion Protocol is standardised in RFC 9309, and Kayak publishes its rules at kayak.com/robots.txt. A robots file is not a licence and not a legal shield either way, but it tells you which paths the site considers off limits, and ignoring it is the fastest way to get your whole range flagged.
Then be strict about scope. Do you need every result on a page, or only the cheapest fare and the top five? Fewer fields means fewer requests and less bandwidth.
Expected output: a one-line spec, for example “cheapest and best fare per route per day, 90 days out, refreshed daily”.
If it breaks: if you cannot state the spec in one line, you will overscrape. Cut it down until you can.
2. build the search URL list
Kayak flight URLs are predictable. A one-way or return search looks like this:
https://www.kayak.com/flights/SIN-BKK/2026-12-10/2026-12-17/1adults
Generate them from a CSV instead of hand-writing:
import csv, itertools, datetime as dt
def build_urls(routes, start, days_out, trip_len=7):
for (o, d), offset in itertools.product(routes, range(days_out)):
dep = start + dt.timedelta(days=offset)
ret = dep + dt.timedelta(days=trip_len)
yield f"https://www.kayak.com/flights/{o}-{d}/{dep:%Y-%m-%d}/{ret:%Y-%m-%d}/1adults"
routes = [("SIN", "BKK"), ("SIN", "HND")]
urls = list(build_urls(routes, dt.date(2026, 11, 1), 90))
print(len(urls)) # 180
Expected output: 180 for two routes and 90 departure days.
If it breaks: if a URL redirects to the homepage, the date is in the past or the airport code is wrong. Test three by hand in a normal browser first.
3. set up Playwright with a proxy
Playwright takes proxy settings per browser launch or per context, documented in the Playwright network guide. Per context is what you want, because it lets one browser process serve many identities.
from playwright.sync_api import sync_playwright
PROXY = {
"server": "http://gate.example-proxy.com:8000",
"username": "user-session-abc123",
"password": "secret",
}
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
ctx = browser.new_context(
proxy=PROXY,
locale="en-US",
timezone_id="Asia/Singapore",
viewport={"width": 1366, "height": 850},
)
page = ctx.new_page()
page.goto("https://api.ipify.org?format=json")
print(page.inner_text("body"))
browser.close()
Expected output: a JSON body with the proxy’s IP, not your server’s.
If it breaks: a 407 means bad credentials or your IP is not whitelisted at the provider. A timeout usually means the gateway port is wrong. Check with curl -x before blaming Playwright.
4. match the timezone and locale to the exit IP
This is the mistake that quietly burns the most sessions. If your exit IP is in Thailand but the browser reports a Singapore timezone and a US locale, you have handed the site a mismatch signal for free. Pick the timezone and locale from the proxy’s country. I wrote up the full checklist in timezone and locale mismatch in a proxied browser.
Expected output: your context settings change per proxy country, not per script.
If it breaks: open https://browserleaks.com/ip through the same context and compare what it reports against what the IP geolocates to.
5. block the heavy assets and wait for results to finish
Kayak streams results in as the search progresses. Grabbing the DOM too early gives you a partial list. Block images, fonts and media to save proxy bandwidth, then wait for the page to settle.
def route_filter(route):
if route.request.resource_type in {"image", "font", "media"}:
return route.abort()
return route.continue_()
ctx.route("**/*", route_filter)
page.goto(url, wait_until="domcontentloaded", timeout=60000)
page.wait_for_timeout(12000) # let the search complete
Expected output: a page whose result list stops growing. Blocking assets typically cuts per-page bandwidth by more than half.
If it breaks: if blocking images triggers a challenge, allow images for that session only. Some defenses treat a zero-image browser as a signal.
6. extract the fields
Kayak changes its class names often, so do not hardcode a selector from a blog post, including this one. Open devtools, find a stable attribute on the result container, and keep selectors in one config object you can edit without touching logic.
SELECTORS = {
"card": "REPLACE_WITH_CURRENT_RESULT_CARD_SELECTOR",
"price": "REPLACE_WITH_CURRENT_PRICE_SELECTOR",
}
def extract(page):
rows = []
for card in page.query_selector_all(SELECTORS["card"])[:10]:
price = card.query_selector(SELECTORS["price"])
rows.append({"price_text": price.inner_text() if price else None})
return rows
Expected output: a list of up to 10 dicts per page, with a price string you then parse into a number and currency.
If it breaks: zero rows with a normal-looking page means the selectors drifted. Zero rows with a captcha means step 7.
7. detect blocks and rotate
Do not retry blindly on the same IP. Check for a block signal, drop that session, and take a fresh IP.
def blocked(page):
text = page.content().lower()
return any(s in text for s in ("captcha", "are you a human", "unusual traffic"))
An HTTP 429 means too many requests, and per MDN’s reference the server may send a Retry-After header. Respect it. When you see a block, log the IP, rotate the session id in your proxy username, wait a randomised 30 to 90 seconds, and requeue the URL.
Expected output: block rate under roughly 5 to 10 percent on good mobile or residential IPs. On datacenter IPs I routinely see most requests blocked.
If it breaks: if the rate sits above 30 percent even on fresh IPs, the problem is your browser fingerprint, not the proxy. Look at your headless flags and fonts, or test with a headed browser. antidetectreview.org covers the fingerprint side in detail.
8. pace yourself
Run a small number of workers, each with its own context and sticky IP, and add jitter between searches. Two to four seconds of random delay plus the 12 second settle wait keeps a worker at roughly three to four searches a minute. That is slow per worker, so you scale by adding workers and IPs, not by speeding up.
Expected output: steady throughput with no bursts.
If it breaks: bursts of failures at the top of the hour usually mean all your workers started in sync. Stagger start times.
9. store results with context
Write every row with the URL, timestamp, exit IP country and a raw price string. Keep the raw string, because parsing logic will change and you will want to reprocess.
import sqlite3
db = sqlite3.connect("kayak.db")
db.execute("""create table if not exists fares(
url text, ts text, country text, price_text text)""")
Expected output: one row per result per search, queryable by route and date.
If it breaks: duplicate rows after retries mean you need a unique key on url plus timestamp bucket.
common pitfalls
- using datacenter IPs and blaming the code. Kayak’s defenses score the IP’s network type first. A perfect script on a bad IP still fails
- scraping with one IP per worker forever. sticky sessions are good for a few searches, not for days. rotate on a schedule and on any block signal
- ignoring currency and point of sale. Kayak shows different prices and currencies depending on the site domain and the IP’s country. If your dataset mixes them, your comparisons are meaningless. Pin the domain and the exit country per route
- parsing prices at scrape time. store raw text, parse later
- no monitoring. selectors drift silently and you collect empty rows for a week. alert when rows per search drops below a threshold
scaling this
At 10x (a few thousand searches a day), a single VPS and one proxy account are fine. Your main job is logging block rate per exit IP so you can spot a bad subnet quickly.
At 100x (tens of thousands a day), proxy bandwidth becomes the largest cost line. Move to a job queue (Redis or Postgres with SKIP LOCKED), run workers on two or three machines, and be strict about asset blocking. Cache aggressively: a route that has not moved in 24 hours does not need a re-search every 6.
At 1000x (hundreds of thousands a day), you are in infrastructure territory. You want pools of mobile or residential IPs spread across several providers so one provider’s bad week does not stop you, per-route scheduling based on how volatile the fare is, and a team member watching dashboards. At this scale many operators decide a licensed data feed or an official partner API is cheaper than running the scraping fleet, so price that option honestly before you commit. I would not promise anyone a return from a scraped dataset, because fare data is only useful if you have a product or a decision that uses it.
If you run several accounts or identities across tools while doing this, the operational habits overlap with what multiaccountops.com writes about.
where to go next
- all tutorials on the blog for the full index
- best mobile proxies for travel scraping to choose a provider for the volume you settled on
- timezone and locale mismatch in a proxied browser to fix the quiet fingerprint leaks from step 4
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-10-14.