How to scrape X (Twitter) at scale in 2026 with proxies that work

X is one of the harder scraping targets I deal with. The official API has been priced out of reach for most side projects since the 2023 overhaul, and the site itself throttles, shadow-bans, and login-walls unauthenticated traffic within minutes of any unusual pattern. If you’ve tried pulling a competitor’s follower list or a hashtag’s worth of posts with a plain requests script and gotten nothing but empty JSON or a login redirect, you already know this.

This tutorial is for people who need X data on a recurring basis: growth teams tracking mentions, researchers building sentiment datasets, or devs feeding a monitoring dashboard. I’ll walk through the proxy and account setup that actually holds up against X’s detection, not just a one-off script that dies after 50 requests. I run scraping infrastructure daily for proxyscraping.org’s own testing, so this is what I’ve verified works in mid-2026, not theory.

By the end you’ll have a repeatable pipeline: rotating proxies matched to account sessions, a browser automation layer that survives X’s fingerprinting, and a rate-limit strategy that doesn’t burn through accounts in a day.

what you need

Residential or mobile proxies, not datacenter. X fingerprints datacenter ASN ranges aggressively and blocks them fast. Budget $4-8/GB on providers like Decodo or SOAX (see my Decodo review for current pricing).
A handful of aged X accounts, ideally 30+ days old with some organic activity. Fresh accounts get rate-limited within the first hour of automation.
Playwright or Puppeteer (not raw requests or httpx) since X’s timeline and search endpoints require a real browser context with JS execution for the GraphQL calls to authenticate properly.
An antidetect browser profile manager if you’re running more than 2-3 accounts concurrently, to keep canvas/WebGL fingerprints distinct per account. I cover fingerprint tooling in more depth on antidetectreview.org.
Storage: Postgres or even SQLite for anything under a few million rows. Don’t overbuild this early.
A cron or queue runner (I use a simple Node worker with a job queue, but cron + a lock file works fine at small scale).
Cost estimate for a 10-account, moderate-volume setup: roughly $150-300/month in proxy bandwidth plus whatever you pay for aged accounts if you’re not growing your own.

step by step

1. Decide your route: API tier vs browser scraping

X’s official X API still exists, and if your volume is low and budget allows, it’s the path of least legal and technical risk. The pricing tiers page has moved around a few times since the initial 2023 overhaul that TechCrunch covered in detail, but as of writing, the free tier caps you at 1,500 posts/month for write actions and read access requires a paid Basic or Pro tier starting in the hundreds of dollars monthly. For most people that’s a non-starter, which is why the rest of this guide covers browser-based scraping.

Expected output: a decision on which route fits your budget and volume. If it breaks: if you’re unsure, start with the API for a proof of concept on a small dataset, then move to browser scraping once you know exactly what fields you need.

2. Source proxies matched to your account count

Buy proxies in a 1:1 or 2:1 ratio with your accounts, not a shared pool. X ties session risk scoring to both IP and account, and mixing accounts across a rotating pool of thousands of IPs actually looks more suspicious than one account sticking to one sticky residential IP for its session.

# example sticky session proxy string (Decodo-style)
curl -x http://user-sessionid123:[email protected]:10000 https://x.com

Expected output: a 200 response with X’s HTML shell (not JSON, since the timeline renders client-side). If it breaks: a 403 or immediate redirect to a login/checkpoint page means the proxy’s IP is already flagged. Rotate to a fresh residential IP and check the provider’s dashboard for that IP’s abuse history.

3. Warm up your accounts before automating

Log into each account manually or via Playwright with human-like delays for a few days before you start scraping. Like some posts, follow a few accounts, check notifications. X’s trust score for an account is cumulative and new accounts doing nothing but scraping look synthetic almost immediately.

Expected output: accounts that don’t trigger a phone/email verification checkpoint on first automated login. If it breaks: if you get a “confirm it’s you” checkpoint, that account’s warmup wasn’t sufficient, back off automation for another week and reduce request frequency.

4. Build the Playwright collector with a persistent context

Use a persistent browser context per account so cookies and local storage survive between runs, exactly like a real user’s browser would.

const { chromium } = require('playwright');

async function scrapeProfile(username, proxy, userDataDir) {
  const context = await chromium.launchPersistentContext(userDataDir, {
    proxy: { server: proxy.server, username: proxy.user, password: proxy.pass },
    headless: true,
  });
  const page = await context.newPage();
  await page.goto(`https://x.com/${username}`, { waitUntil: 'networkidle' });
  await page.waitForSelector('[data-testid="tweet"]', { timeout: 15000 });

  const posts = await page.$$eval('[data-testid="tweet"]', nodes =>
    nodes.map(n => n.innerText)
  );

  await context.close();
  return posts;
}

Expected output: an array of visible post text blocks from the target profile’s timeline. If it breaks: a timeout on waitForSelector usually means you hit a login wall or a rate-limit interstitial. Screenshot the page (page.screenshot()) before closing the context to see what actually loaded.

5. Handle pagination and rate limits explicitly

X’s timeline loads via infinite scroll backed by GraphQL calls. Scroll incrementally and add jittered delays (2-5 seconds, randomized) between scroll events rather than one fast loop.

for (let i = 0; i < scrollCount; i++) {
  await page.mouse.wheel(0, 2000);
  await page.waitForTimeout(2000 + Math.random() * 3000);
}

Expected output: new posts loading into the DOM on each scroll, visible in your $$eval selector count going up. If it breaks: if the post count stalls, you’ve likely hit that session’s soft rate limit. Stop that account for 15-30 minutes rather than retrying immediately, retrying immediately is what gets accounts suspended.

6. Deduplicate and store incrementally

Write to your database as you go, keyed on post ID (extract it from the permalink href, not the visible text), so a crash mid-run doesn’t lose everything.

Expected output: a growing table with no duplicate post IDs across runs. If it breaks: if you see duplicate rows, check that you’re parsing the actual tweet ID from the URL rather than hashing the text, retweets and quote posts often have near-identical visible text.

7. Rotate accounts and monitor for suspensions daily

Run a small health-check script each morning that logs into each account and checks for a suspension banner or checkpoint before your main scrape jobs kick off.

Expected output: a daily list of healthy vs flagged accounts. If it breaks: if more than 20% of your accounts get flagged in a week, your request volume per account is too aggressive, dial it back before replacing accounts, since replacements just eat the same limit again.

common pitfalls

Using datacenter proxies to save money. They’re cheaper per GB but X blocks known datacenter ranges within minutes for anything beyond casual browsing. This is the single most common reason first attempts fail.
Scraping too fast on brand-new accounts. An account created and immediately put to work doing 500 profile visits an hour reads as a bot to any detection system, human or automated.
Sharing one proxy IP across many accounts. This creates a footprint that’s easy to cluster and ban in bulk, one flagged account can take down the others sharing that IP.
Ignoring X’s Terms of Service entirely. Automated data collection outside the official API is restricted under X’s Terms of Service, and while enforcement varies, understand you’re operating outside the documented rules. This isn’t legal advice, if your use case has legal exposure (commercial resale, personal data at scale), talk to a lawyer familiar with data scraping law in your jurisdiction.
No monitoring for silent failures. Selector changes on X’s frontend break scrapers quietly, they don’t error, they just return zero results. Add a sanity check that alerts you when a run returns suspiciously low post counts.

scaling this

At 10x (roughly 10-30 accounts), you can still run everything from one machine with a job queue like BullMQ, and manual account health checks are manageable once a day. Proxy cost is the main variable, expect $150-400/month.

At 100x (100-300 accounts), you need account and proxy inventory tracking in a database, not a spreadsheet. Automate the health-check step from Step 7 and build in automatic account retirement when flag rates spike. You’ll also want to distribute your Playwright workers across multiple VPS instances or containers, since a single machine running 100+ concurrent browser contexts will run out of memory fast. Consider looking at multi-account management practices from multiaccountops.com since account isolation at this scale becomes the core engineering problem, more than the scraping logic itself.

At 1000x (enterprise-scale monitoring), you’re now running a distributed system: proxy pools sourced from multiple vendors to avoid single-provider IP exhaustion, account pools in the thousands with automated warmup pipelines, and a dedicated ops process for replacing suspended accounts continuously. At this point the economics shift, you’re likely better served evaluating whether X’s official Enterprise API tier is actually cheaper than the account replacement and proxy costs of running browser scraping at this volume. Run the math before committing infrastructure.

where to go next

If you’re comparing proxy providers for this kind of workload, read my Decodo vs SOAX head-to-head for pricing and residential IP pool size differences.
For a similar scraping approach applied to a different high-friction target, see how to scrape LinkedIn at scale in 2026.
Browse the full tutorial index for more scrape-target guides as I publish them.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-17.