How to scrape Twitter Ads Library at scale in 2026 with proxies that work
Meta has the Ad Library API. TikTok has the Commercial Content Library. X does not have an equivalent public API for its ad transparency tool, so if you want structured data on who’s advertising on the platform, what the creative looks like, and how long a campaign has been running, you’re scraping a web UI, not calling an endpoint with a key. That’s the whole problem this tutorial solves.
This is written for competitive intelligence analysts, growth marketers tracking competitor creative, and ad tech teams building monitoring dashboards. I’m not going to pretend this is a one-click job. X runs bot detection on top of the ads transparency tool the same way it does on the main timeline, and a naive requests.get() loop gets you blocked inside a few dozen page loads.
By the end you’ll have a Playwright-based scraper that pulls ad creative and metadata per advertiser handle, routes traffic through a rotating proxy pool so you’re not burning a single IP into oblivion, and stores results in a way you can diff over time. I’ll also flag where this gets expensive and where operators get sloppy and torch their proxy budget for nothing.
what you need
- Node.js 20+ (or Python 3.11+ if you prefer Playwright’s Python bindings, code below is JS)
- Playwright with the Chromium browser installed (
npx playwright install chromium) - A residential or mobile proxy pool. Datacenter IPs get flagged almost immediately on x.com properties. I use Decodo for most of my rotating-residential work and have tested SOAX side by side, both work, pricing runs roughly $7-9/GB on pay-as-you-go residential tiers as of mid-2026, check current rates before committing
- A proxy budget. Realistically 50-150MB per 1,000 ad cards scraped once you account for images and retries, plan accordingly
- Storage. Postgres if you’re tracking this over time, SQLite or plain CSV is fine for a one-off pull
- A list of advertiser handles or domains you actually want to track, don’t scrape blind
- Patience for selector rot. X changes its frontend markup often enough that any scraper built against
data-testidattributes needs maintenance built into the plan, not treated as a bug
step by step
1. scope your target list before writing any code
Build a flat file (CSV or JSON) of the advertiser handles you’re tracking. If you’re doing competitive intel, this is your competitor set plus a handful of adjacent brands for benchmarking. If you’re doing market research, it’s a category sample.
Expected output: a targets.json file like ["nike", "adidas", "underarmour"].
If it breaks: it won’t, this is just data entry, but resist the urge to scrape “everything.” Untargeted scraping is what gets IPs burned fastest because you’re hitting new, never-cached pages constantly instead of re-visiting known-good paths.
2. set up Playwright with a realistic browser fingerprint
mkdir ads-scraper && cd ads-scraper
npm init -y
npm install playwright
npx playwright install chromium
Don’t run Playwright with its default fingerprint. Set a real user agent, a common viewport, and disable the automation flags that bot detection scripts check for.
// scraper.js
const { chromium } = require('playwright');
async function newStealthContext(browser) {
return browser.newContext({
userAgent:
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
'(KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36',
viewport: { width: 1366, height: 768 },
locale: 'en-US',
timezoneId: 'America/New_York',
});
}
Expected output: a headless Chromium instance that renders x.com pages without an immediate challenge wall.
If it breaks: if you’re getting an interstitial “verify you’re not a bot” page on the very first load with no proxy involved yet, your fingerprint is the problem, not your IP. Test with headless: false locally first so you can see what’s actually rendering.
3. wire up your proxy pool
Point Playwright at your proxy provider’s gateway. Most residential providers use sticky sessions keyed by a session ID in the username, so you keep the same exit IP for the length of one advertiser’s scrape, then rotate for the next.
function proxyForSession(sessionId) {
return {
server: 'http://gate.decodo.com:10001',
username: `user-sp12345-session-${sessionId}-sessTime-10`,
password: process.env.PROXY_PASSWORD,
};
}
Expected output: each advertiser you scrape exits from a different residential IP, and that IP stays consistent for the duration of one advertiser’s session so you don’t look like you’re teleporting between requests.
If it breaks: if every request is timing out, check whether your provider’s gateway is geo-restricted or you’ve exhausted a trial allowance. If it connects but every page is a challenge page, your proxy pool has a bad reputation score, this happens with cheaper shared pools, worth testing a small amount of traffic before buying bulk GB.
4. navigate to the ads transparency tool and search by handle
X’s ads transparency tool has moved paths twice since it launched in 2023, so I’m not going to hardcode a URL that’ll be stale by the time you read this. Grab the current path from your own browser’s address bar while signed into x.com (search “ads transparency” from account settings), and store it as a config value.
async function searchAdvertiser(page, handle, adsLibraryBaseUrl) {
await page.goto(`${adsLibraryBaseUrl}?q=${handle}`, { waitUntil: 'networkidle' });
await page.waitForTimeout(1500 + Math.random() * 2500);
}
Expected output: a results page listing active and recent ads for that handle, with creative previews.
If it breaks: if the search returns zero results for a handle you know is running ads, check whether the handle needs to be the verified advertiser name rather than the @handle, X’s search sometimes indexes by display name.
5. extract structured data from the ad cards
async function extractAds(page) {
return page.$$eval('[data-testid="adCard"]', (cards) =>
cards.map((c) => ({
text: c.querySelector('[data-testid="adText"]')?.innerText ?? null,
mediaUrl: c.querySelector('img, video')?.src ?? null,
scrapedAt: new Date().toISOString(),
}))
);
}
Expected output: an array of objects, one per ad card, with the creative text and media URL.
If it breaks: this is the step that breaks most often because data-testid values change with frontend deploys. Keep a fallback selector strategy (text content matching, aria-label matching) and alert yourself when extraction returns zero results on a handle that returned data yesterday, that’s your signal the markup shifted, not that the advertiser stopped running ads.
6. handle pagination
Ad libraries lazy-load. Scroll and wait rather than looking for a “next page” button.
async function scrollToLoadMore(page, maxScrolls = 8) {
let previousHeight = 0;
for (let i = 0; i < maxScrolls; i++) {
const height = await page.evaluate(() => document.body.scrollHeight);
if (height === previousHeight) break;
previousHeight = height;
await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));
await page.waitForTimeout(1000 + Math.random() * 1500);
}
}
Expected output: the full set of ads for that advertiser loaded into the DOM before you extract.
If it breaks: if the page height stops growing after one scroll on an advertiser you know has many ads, you may have hit a rate limit mid-session, back off and retry with a fresh proxy session rather than hammering the same IP.
7. store results and de-dupe across runs
Hash the ad text plus media URL to a fingerprint so re-running the scraper daily doesn’t create duplicate rows for ads that are still live.
const crypto = require('crypto');
function adFingerprint(ad) {
return crypto.createHash('sha256').update(`${ad.text}|${ad.mediaUrl}`).digest('hex');
}
Expected output: a table where you can query “what ads are new since yesterday” rather than a growing pile of duplicates.
If it breaks: if your de-dupe rate looks too high (near 100% “new” every run), your extraction is probably capturing a timestamp or session-specific query param inside what should be a stable field, strip those before hashing.
8. schedule it and monitor for silent failures
Cron or a queue like BullMQ works fine for this. The part people skip is monitoring: alert yourself when a scrape returns 0 ads for a handle that historically returns dozens, since that’s almost always a broken selector or a block, not a real data change.
Expected output: a daily or weekly run that either produces data or pages you when it doesn’t.
If it breaks: if you’re getting silent empty results with no errors thrown, check response status codes, not just DOM state, X sometimes serves a 200 with a soft-block page instead of an HTTP error.
common pitfalls
- Using datacenter proxies to save money. You’ll save money right up until the entire subnet gets flagged and every request returns a challenge page. Residential or mobile IPs cost more per GB but actually complete requests.
- Not rotating fingerprint alongside IP. Same browser fingerprint hitting the site from twelve different countries in an hour is a more obvious signal than a stable IP with a stable fingerprint. Keep them consistent per session.
- Scraping too fast with no jitter. Fixed-interval requests are trivially detectable. Randomize your delays.
- Ignoring pagination timing. Scrolling and extracting on the same tick before lazy-loaded content renders gives you incomplete data that looks complete, silent data loss is worse than a visible error.
- Treating this as legally settled. Scraping publicly viewable data has real precedent behind it (the hiQ Labs v. LinkedIn line of cases in the US), but X’s own automation rules restrict scripted access, and unauthorized access claims under US law get evaluated against the Computer Fraud and Abuse Act. This is not legal advice, if you’re scraping at real commercial scale, get an actual lawyer to review your specific use case before you build a business on it.
scaling this
10x (a few hundred advertisers, weekly cadence): one machine, one proxy provider account, sequential scraping is fine. Budget maybe 2-5GB of residential proxy traffic a month.
100x (thousands of advertisers, daily cadence): you need concurrency, which means multiple proxy sessions running in parallel rather than one sticky session at a time. Move to a job queue, run 5-10 browser contexts concurrently, each with its own session, and stagger start times so you’re not spiking your proxy provider’s gateway all at once. This is also where you should split by advertiser geography, since X’s ad transparency data differs for EU-facing advertisers versus the rest of the world under the EU Digital Services Act’s ad repository requirement (Article 39), which is the actual regulatory reason this transparency tool exists in its current global form at all.
1000x (tens of thousands of advertisers, near real-time): you’re now running infrastructure, not a script. Dedicated worker pool, headless browser farm instead of a laptop, an enterprise proxy contract instead of pay-as-you-go GB (worth calling Decodo or SOAX sales directly at this volume rather than using the self-serve dashboard), and a selector-monitoring system that pages someone the moment extraction yield drops, because at this scale a silent frontend change costs you days of bad data before anyone notices manually. If you’re also managing many browser profiles or ad accounts to cross-reference against scraped creative, that’s a separate operational problem worth reading up on at multiaccountops.com, which covers profile isolation at exactly this kind of scale.
where to go next
- If you’re building out a broader competitive intelligence pipeline, read how to scrape LinkedIn at scale in 2026 next, similar bot-detection problem, different data model.
- For the proxy layer itself, my Decodo review covers pricing tiers and session behavior in more depth than I could fit here.
- Browse the rest of the scraping tutorials at the blog index if you’re building out a full monitoring stack rather than a single scraper.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-17.