How to scrape YouTube at scale in 2026 with proxies that work

YouTube is the second-largest search engine on earth and the worst-documented one to pull data from at volume. The official YouTube Data API v3 covers metadata, but its default quota is 10,000 units a day, and a single search.list call costs 100 of those. Do the math on a competitor-tracking job across a few thousand channels and you burn the daily allowance before lunch. That’s why most operators end up mixing the API with direct page scraping, and direct page scraping is where IP-based blocking, consent walls, and region locks start costing you real time.

This tutorial is for people who need YouTube data on a recurring basis, not a one-off export: competitive research on channel growth, comment sentiment monitoring, SERP tracking for video search results, or building a dataset for an ML pipeline. I’m assuming you’re comfortable with Python and a terminal, and that you’re pulling public data (view counts, titles, descriptions, public comments) rather than anything gated behind a login you don’t own.

By the end you’ll have a setup that combines the official API for what it does well, a scraping fallback for what it doesn’t expose, and a proxy layer that keeps both running without getting your IPs burned. I’ll flag the legal and ToS boundary as I go, because it matters more here than on most targets.

what you need

• Python 3.10+ with requests, google-api-python-client, and yt-dlp installed
• A Google Cloud project with the YouTube Data API v3 enabled and an API key (free tier: 10,000 quota units/day)
• A residential or mobile proxy pool, not datacenter IPs. Datacenter ranges get flagged by YouTube’s anti-bot layer within minutes at any real volume. Budget roughly $4-8/GB on residential pools from providers like Decodo or SOAX, more if you need mobile carrier IPs for anything comment- or watch-page heavy
• A rotating session manager (proxy rotation logic, either self-built or via your proxy provider’s built-in gateway)
• A place to store output — Postgres or even a flat SQLite file is fine for most jobs; you’ll want dedupe on video IDs
• Time budget: expect a day to get quota math and proxy rotation right before your first clean 24-hour run

step by step

1. Decide what actually needs the API vs. what needs scraping

The API gives you clean structured data for search, channel stats, video metadata, and playlists, but it doesn’t expose full comment threads at any real depth, related-video recommendations, or the actual search ranking order a logged-out user sees. Map your data needs against the API’s resource list first. Anything the API can give you, take from the API. It’s faster, it’s sanctioned by Google’s own terms, and it doesn’t need a proxy at all for reasonable volumes.

If it breaks: if you’re not sure a field exists in the API response, check the part parameter you’re requesting. Half of “the API doesn’t have this” complaints are just missing a part=snippet,statistics,contentDetails combination.

2. Set up your API project and quota

Create a project in Google Cloud Console, enable “YouTube Data API v3,” and generate an API key restricted to that API. Test it:

curl "https://www.googleapis.com/youtube/v3/videos?part=snippet,statistics&id=dQw4w9WgXcQ&key=YOUR_API_KEY"

You should get back JSON with title, description, view count, and like count. Track your quota cost per call type; search.list is 100 units, videos.list is 1 unit, commentThreads.list is 1 unit. This asymmetry is the whole game: never use search.list when you can build the same list from playlistItems.list on a channel’s uploads playlist, which costs 1 unit instead of 100.

If it breaks: a 403 with quotaExceeded means you’re done for the day on that project. Spin up a second Cloud project with its own key rather than requesting a quota increase, which takes days and requires a use-case justification form.

3. Pull channel and video metadata at volume

from googleapiclient.discovery import build

youtube = build("youtube", "v3", developerKey="YOUR_API_KEY")

def get_uploads_playlist(channel_id):
    resp = youtube.channels().list(part="contentDetails", id=channel_id).execute()
    return resp["items"][0]["contentDetails"]["relatedPlaylists"]["uploads"]

def get_all_video_ids(playlist_id):
    ids, page_token = [], None
    while True:
        resp = youtube.playlistItems().list(
            part="contentDetails", playlistId=playlist_id,
            maxResults=50, pageToken=page_token
        ).execute()
        ids += [item["contentDetails"]["videoId"] for item in resp["items"]]
        page_token = resp.get("nextPageToken")
        if not page_token:
            return ids

This costs 1 unit per page of 50 videos, so a 2,000-video channel costs 40 units total. Run this across your channel list before touching anything that requires scraping.

If it breaks: an empty items array usually means the channel ID format is wrong (you need the UC... channel ID, not the @handle URL slug). Resolve handles first with channels().list(forHandle=...).

4. Set up your proxy pool for the scraping fallback

For anything the API doesn’t cover (full comment trees beyond what commentThreads returns, watch-page “up next” recommendations, search result ordering), you’ll hit YouTube’s public pages directly, and that’s where you need proxies. Configure a rotating gateway so each request gets a fresh residential IP:

import requests

proxies = {
    "http": "http://user-session-rotate:[email protected]:10000",
    "https": "http://user-session-rotate:[email protected]:10000",
}

resp = requests.get(
    "https://www.youtube.com/results?search_query=proxy+scraping",
    proxies=proxies,
    headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"},
    timeout=15,
)

Match your proxy’s geolocation to the user-agent’s implied locale, and to whatever region you’re trying to see results from, since YouTube personalizes search and recommendations by IP geography.

If it breaks: a 429 or a redirect to consent.youtube.com means the IP got flagged or you hit the EU cookie-consent wall. Rotate the session and, for the consent wall, pre-set a CONSENT=YES+ cookie on the request.

5. Use yt-dlp for video-level extraction

For metadata extraction at the individual video level, especially formats and subtitle availability, yt-dlp is more reliable than hand-rolled scraping because the maintainers patch it against YouTube’s player changes constantly:

yt-dlp --dump-json --skip-download --proxy "http://user:[email protected]:10000" \
  "https://www.youtube.com/watch?v=VIDEO_ID" > video_metadata.json

If it breaks: yt-dlp breaking after a YouTube player update is common; pip install -U yt-dlp before assuming your proxy or code is at fault. Check the project’s GitHub issues for open reports before filing your own.

6. Throttle and rotate deliberately

Don’t fire requests as fast as your proxy pool allows. Build in jitter, 2-6 seconds between requests per session, and cap concurrent sessions based on your proxy pool size, roughly one concurrent session per 50-100 IPs in the pool for sustained scraping.

If it breaks: if your success rate drops below 90% even with fresh IPs, you’re going too fast for that IP’s trust score. Slow the per-session request rate before adding more IPs.

7. Store and dedupe

Write to a database keyed on video ID and a fetch timestamp, not a flat file per run. You’ll re-scrape the same channels on a schedule, and you need to diff view counts over time, not just the latest snapshot.

If it breaks: if you’re seeing duplicate rows, check whether your pagination logic is re-requesting page 1 after a nextPageToken expires mid-run (they expire after roughly 24 hours).

8. Monitor block rates and rotate proxy vendors if needed

Log HTTP status codes and consent-wall redirects per proxy session. If one vendor’s IP block is getting flagged disproportionately, that’s a signal to diversify providers rather than push harder on the same pool.

If it breaks: a sudden spike in blocks across the board (not just one vendor) usually means YouTube shipped an anti-bot update. Check community scraping forums or the yt-dlp changelog before assuming your setup is broken.

common pitfalls

• Using search.list for everything. At 100 units a call it eats your daily quota in under two minutes of volume. Use playlist-based enumeration instead wherever the target is a known channel.
• Running scraping traffic on datacenter IPs. YouTube’s bot detection flags AWS/GCP/Azure ranges almost immediately. Residential or mobile IPs are non-negotiable for page scraping, even though they cost more.
• Ignoring the consent and cookie wall. EU-geo IPs without a pre-set consent cookie get redirected into a loop that looks like a block but is actually just an unhandled redirect.
• Not reading YouTube’s Terms of Service before building a commercial product on scraped data. YouTube’s ToS restricts automated access outside the API for certain uses, and robots.txt disallows crawling most page paths. I’m not a lawyer and this isn’t legal advice, but treat API-first as the compliant default and understand you’re taking on risk with any scraping that goes beyond it, especially for commercial redistribution of the data.
• Scaling proxy volume before fixing request patterns. Buying more IPs doesn’t fix a bot signature problem, it just gets you blocked slower.

scaling this

At 10x (a few hundred channels, daily refresh), the free API quota plus a small rotating proxy pool (a few hundred residential IPs) handles it fine. You’re mostly quota-constrained, not proxy-constrained.

At 100x (tens of thousands of videos, hourly-ish refresh on some subset), you need a second and third Google Cloud project for quota headroom, and your proxy pool needs to grow to a few thousand IPs with proper session stickiness so you’re not re-authenticating every request. This is also where you want a queue (Redis or SQS-style) between your scrapers and your database, since write contention becomes real.

At 1000x (hundreds of thousands of videos, near-real-time on your top targets), you’re running a distributed scraping cluster, not a script. Expect to need mobile proxy pools specifically for comment and watch-page data, dedicated infrastructure for job scheduling, and a serious monitoring layer that alerts on block-rate spikes per proxy subnet, not just per vendor. At this scale, the API quota stops mattering much because scraping volume dwarfs what quota covers, so your cost center shifts almost entirely to proxy bandwidth and IP quality.

where to go next

If you’re building a broader social scraping stack, the setup here overlaps heavily with how to scrape TikTok at scale in 2026 with proxies that work and how to scrape Instagram at scale in 2026 with proxies that work, since the proxy rotation and quota-management principles carry over almost directly. For the browser-fingerprinting side of avoiding detection when you go beyond API-only pulls, antidetectreview.org’s blog covers fingerprint spoofing tooling in more depth than fits here. And for the full article index on this site, start at /blog/.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-17.