How to scrape Reddit at scale in 2026 with proxies that work

Reddit is one of the most valuable text datasets on the internet, and also one of the more annoying platforms to pull at volume. Since the 2023 API pricing change, the free ride ended: Reddit started charging developers for API access, and the third-party tools that used to make this easy (Pushshift’s public search, most third-party apps) either shut down or got restricted. If you’re building a sentiment tool, training data pipeline, or a niche monitoring dashboard on subreddit activity, you now need a real plan instead of a weekend script.

This tutorial is for operators, not academics. I’ll walk through pulling posts and comments at scale using both the official Reddit API via PRAW and a fallback scraping approach for when API quotas don’t cut it, plus the proxy setup that keeps either method running without tripping rate limits or IP bans. This isn’t about circumventing Reddit’s terms to scrape private data or user PII, it’s about pulling public post and comment data reliably and at a volume that a single residential IP can’t sustain.

By the end you’ll have a working pipeline that can pull thousands of posts and comment threads a day without getting your requester IP or account flagged, and a sense of what infrastructure changes as you go from a hobby script to something running continuously across dozens of subreddits.

what you need

• Python 3.10+ with praw and requests installed (pip install praw requests)
• A Reddit account in good standing to register a developer app at reddit.com/prefs/apps — you need client_id and client_secret
• Reddit API access, either free-tier (rate-limited, fine for small pulls) or paid via Reddit’s Data API, which as of the 2023 pricing announcement runs $0.24 per 1,000 API calls for high-volume commercial use
• Rotating residential or mobile proxies if you’re doing HTML scraping instead of, or alongside, the API — I’ve had good results with SOAX and Decodo (formerly Smartproxy) for this specifically, see my Decodo review for pricing
• A place to store output — Postgres or even flat JSONL files if you’re just prototyping
• Budget: figure $50-150/month for residential proxy bandwidth at moderate scale (a few GB/day), plus whatever the API tier costs if you exceed the free quota
• Basic comfort with OAuth2 flows, since Reddit’s API requires an authenticated token, not just an API key

step by step

1. register a Reddit API app

Go to reddit.com/prefs/apps, click “create app,” and pick “script” as the type. This gives you a client_id (under the app name) and a client_secret. Set a redirect URI of http://localhost:8080 even though you won’t use it for a script-type app.

Expected output: a 14-character client ID and a 27-character secret string in your apps dashboard.

If it breaks: if the create button is greyed out, your account likely needs email verification first. Verify the email tied to your Reddit account and retry.

2. authenticate with PRAW

PRAW (Python Reddit API Wrapper) is the standard library here and it’s actively maintained. Install it and set up a basic client:

import praw

reddit = praw.Reddit(
    client_id="YOUR_CLIENT_ID",
    client_secret="YOUR_CLIENT_SECRET",
    user_agent="my-scraper/1.0 by u/yourusername",
    ratelimit_seconds=300,
)

print(reddit.read_only)  # should print True

Expected output: True printed to console, confirming the token exchange worked.

If it breaks: a 401 error almost always means a bad client_id/secret pair or a user_agent string that’s too generic (Reddit rejects the default praw user agent). Always set a custom, descriptive user_agent.

3. pull posts from a subreddit

subreddit = reddit.subreddit("dataisbeautiful")

for post in subreddit.new(limit=500):
    print(post.id, post.title, post.score, post.created_utc)

Expected output: up to 500 recent posts printed with id, title, score, and UTC timestamp.

If it breaks: PRAW silently caps at what Reddit’s listing endpoints allow, roughly 1000 items per listing regardless of limit. If you need more historical data than that, you’ll need to paginate by timestamp or switch to a search-based pull.

4. pull comment trees

submission = reddit.submission(id=post.id)
submission.comments.replace_more(limit=0)  # drop "load more comments" stubs

for comment in submission.comments.list():
    print(comment.id, comment.body[:80], comment.score)

Expected output: a flat list of top-level and nested comments for that post.

If it breaks: replace_more(limit=0) drops deeply nested threads to avoid extra API calls. If you actually need full depth, set a higher limit but expect it to burn through your rate limit fast, each “load more” click is its own API call.

5. respect rate limits and back off properly

Reddit’s OAuth API gives you roughly 100 queries per minute per OAuth client on the free tier (Reddit adjusts this periodically, check current limits in the API rules doc before building around a specific number). PRAW handles most of the backoff automatically via the ratelimit_seconds parameter, but for custom requests-based pulls you need to read the X-Ratelimit-Remaining and X-Ratelimit-Reset headers yourself and sleep accordingly.

import time

remaining = int(response.headers.get("x-ratelimit-remaining", 1))
reset = int(response.headers.get("x-ratelimit-reset", 1))

if remaining < 2:
    time.sleep(reset + 1)

Expected output: your script pauses automatically as it approaches the limit instead of getting a 429.

If it breaks: repeated 429s despite backoff usually means multiple scripts sharing one OAuth client. Reddit rate-limits per token, not per IP, for authenticated calls, so this is a code bug, not a proxy problem.

6. add proxy rotation for the scraping fallback

The API covers most cases, but if you’re pulling old.reddit.com HTML directly (for data the API doesn’t expose cleanly, like certain search result rankings) you’ll hit IP-based rate limiting fast on a single connection. This is where rotating proxies matter, not for the OAuth API calls themselves, but for any direct HTTP scraping you layer on top.

import requests

proxies = {
    "http": "http://user:[email protected]:9000",
    "https": "http://user:[email protected]:9000",
}

resp = requests.get(
    "https://old.reddit.com/r/dataisbeautiful/top/.json?t=month",
    proxies=proxies,
    headers={"User-Agent": "research-bot/1.0"},
    timeout=15,
)
print(resp.status_code)

Expected output: 200 with the JSON listing payload.

If it breaks: repeated 429 or 403 responses through the proxy usually mean the proxy IP is already flagged from prior traffic. Rotate to a fresh sticky session on your provider’s dashboard, most residential proxy panels let you force a new exit IP per session.

7. store and dedupe

Write straight to Postgres with a unique constraint on post/comment ID so re-running a pull doesn’t duplicate rows:

CREATE TABLE reddit_posts (
  id TEXT PRIMARY KEY,
  subreddit TEXT,
  title TEXT,
  score INT,
  created_utc TIMESTAMP,
  fetched_at TIMESTAMP DEFAULT now()
);

Expected output: INSERT ... ON CONFLICT (id) DO NOTHING runs clean on repeated pulls with zero duplicate rows.

If it breaks: if you’re seeing duplicate content anyway, check whether you’re pulling from both .new() and .hot() for the same subreddit without deduping on the application side first, both return overlapping posts.

8. schedule it

Wrap the pull in a cron job or a scheduled task (Task Scheduler on Windows, cron on Linux) running every 15-60 minutes depending on how fresh you need the data.

Expected output: a log file showing successful runs with row counts, no manual intervention needed.

If it breaks: if runs silently stop, check whether your OAuth token expired mid-cron. PRAW refreshes automatically, but raw requests-based auth flows need you to catch 401s and re-fetch a token.

common pitfalls

• Treating the free API tier like it’s unlimited. It isn’t, and Reddit does enforce cutoffs. Build backoff logic from day one instead of retrofitting it after your script gets throttled mid-run.
• Using a generic or missing user agent. Reddit’s API docs are explicit that identifiable user agents are required, and vague ones (python-requests/2.x) get rate-limited harder than descriptive ones.
• Scraping HTML instead of using the JSON endpoints. Appending .json to almost any Reddit URL gives you structured data without parsing HTML, and it’s far less likely to break when Reddit ships a frontend redesign.
• Running everything through one static datacenter IP. Datacenter IP ranges are widely known and get blocked faster than residential ranges, even for read-only public data. If you’re scaling past a few thousand requests a day outside the API, budget for residential or mobile proxies.
• Ignoring subreddit-level rules. Some subreddits explicitly disallow bots and automated data collection in their sidebar rules. Check before pulling, and don’t scrape private or quarantined communities, that’s a separate risk category entirely and outside what this guide covers.

scaling this

At 10x (a handful of subreddits, a few thousand posts/day), the free API tier plus PRAW’s built-in rate limiting is enough. No proxies needed if you’re staying inside API limits.

At 100x (dozens of subreddits, tens of thousands of items/day, comment trees included), you’ll likely exceed free API quotas and need to either pay Reddit’s per-call rate or supplement with scraping. This is where a rotating residential proxy plan earns its keep, expect $50-150/month in proxy bandwidth depending on provider and volume. I’ve covered proxy selection for scale scraping more broadly in my guide to scraping LinkedIn at scale, and a lot of the proxy-rotation logic carries over directly.

At 1000x (continuous multi-subreddit monitoring, real-time-ish comment tracking), you’re running a real pipeline: multiple OAuth clients to parallelize API pulls within Reddit’s per-client limits, a proxy pool sized for sustained concurrent connections rather than occasional bursts, and a queue system (Celery, or even a simple SQS-backed worker pool) so a single stalled request doesn’t block the whole run. At this scale, also plan for storage growth, comment trees compound fast, and for account-level risk if you’re running multiple Reddit accounts for higher combined quota. If multi-account management is part of your setup, that’s a distinct operational problem worth reading up on separately, the folks at multiaccountops.com cover account isolation and ban-avoidance in more depth than I will here.

where to go next

If proxies are new territory for you, start with my breakdown on best proxies for scraping local search results in 2026, which covers the residential-vs-datacenter tradeoff in more detail than fits here. For a head-to-head on the two providers I mention above, see Decodo vs Smartproxy 2026. And for more scraping tutorials across other platforms, browse the full article index.

This is not legal advice. Reddit’s terms and API rules change, and you’re responsible for reading and complying with the current Reddit Data API terms and each subreddit’s own rules before you scrape it.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-16.