How to scrape Hacker News at scale in 2026 with proxies that work

Hacker News looks like a simple orange page, but it’s one of the richest, cleanest datasets in tech. Story titles, points, comment trees, submission timestamps, user karma, all of it structured and mostly public. People build sentiment trackers on it, train models on the comment threads, monitor it for brand mentions, or just want a local mirror they can query without hammering Y Combinator’s servers every time they refresh a dashboard.

This is for builders who want more than the front page: someone pulling thousands of stories and comment trees a day, backfilling months of history, or running a standing monitor that checks Hacker News every few minutes. I’m going to walk through the official API, the Algolia search index that most scrapers ignore, where proxies actually help versus where they’re dead weight, and the mistakes that get people’s scrapers throttled or blocked.

By the end you’ll have a working pipeline that pulls live and historical Hacker News data, stores it somewhere queryable, and scales from a single script on your laptop to a fleet of workers pulling tens of thousands of items a day without getting your IP range flagged.

what you need

• Python 3.10+ (or Node, the logic ports directly) and requests or httpx
• A SQLite or Postgres database to store items, comments doesn’t fit nicely in flat files past a few thousand rows
• Familiarity with the official Hacker News API, a read-only Firebase-backed REST API, free, no key required
• Access to the Algolia HN Search API for full-text and historical queries the official API doesn’t support well
• A residential or rotating datacenter proxy plan if you’re going to scrape the HTML front-end or user profile pages at volume, not strictly needed for the JSON APIs but essential once you’re running many parallel workers against the web UI. I’ve used Decodo for this kind of job, budget $50 to $200/month depending on volume
• A job scheduler (cron, or a simple queue) if you’re running this continuously rather than as a one-off pull
• Basic respect for Hacker News’ guidelines and robots.txt, this is a small community-run site, not a corporate API with infinite headroom

step by step

1. Decide which data source actually fits your goal

Before writing a line of code, figure out whether you need live data (new stories as they post), historical data (everything since 2007), or full-text search (find every story mentioning “Rust” with 100+ points). Each maps to a different endpoint.

• Live/current: the official Firebase API
• Historical + full-text + filters: the Algolia HN Search API
• Anything not in either (vote counts over time, flagged status changes, user comment history beyond what’s indexed): the HTML pages themselves

Expected output: a one-line answer to “what am I actually pulling” that tells you which of the next three steps you need.

If it breaks: people usually try to force everything through the Firebase API because it’s the “official” one, then get stuck when they need date-range filtering it doesn’t support. Check Algolia first for anything historical.

2. Pull live items from the official Firebase API

The official API exposes story, comment, job, poll, and user objects as flat JSON over HTTPS, no authentication, no API key.

curl https://hacker-news.firebaseio.com/v0/topstories.json

That returns an array of up to 500 story IDs. Hydrate each one:

import requests

BASE = "https://hacker-news.firebaseio.com/v0"

def get_item(item_id):
    r = requests.get(f"{BASE}/item/{item_id}.json", timeout=10)
    r.raise_for_status()
    return r.json()

top_ids = requests.get(f"{BASE}/topstories.json").json()
items = [get_item(i) for i in top_ids[:30]]

Expected output: a list of dicts with title, by, score, time, descendants (comment count), kids (child comment IDs).

If it breaks: a 404 on an item ID usually means the item was deleted or is a job/poll type with different fields, don’t assume every ID resolves to a story. Wrap the hydrate call in a try/except and log skipped IDs rather than crashing the batch.

3. Walk comment trees recursively

Comments are just items with a parent field and a kids array. To get a full thread you recurse.

def get_comment_tree(item_id, depth=0, max_depth=10):
    item = get_item(item_id)
    if not item or depth > max_depth:
        return None
    item["replies"] = [
        get_comment_tree(kid, depth + 1, max_depth)
        for kid in item.get("kids", [])
    ]
    return item

Expected output: a nested tree for one story’s full discussion, deleted or dead comments show up with "deleted": true and no text, filter those out downstream.

If it breaks: deep threads (500+ comments) will make hundreds of sequential requests per story. Use asyncio with httpx.AsyncClient and a semaphore capped around 20-30 concurrent requests, going higher gets you soft-throttled.

4. Use the Algolia API for historical and filtered pulls

For anything with a date range, minimum score, or keyword match, skip the Firebase API entirely.

curl "https://hn.algolia.com/api/v1/search_by_date?query=rust&tags=story&numericFilters=points>100"

Expected output: paginated JSON with hits, each hit already flattened with author, points, created_at, objectID (maps back to the Firebase item ID if you need the comment tree too).

If it breaks: Algolia paginates at 1000 results per query by default and the page parameter maxes out around 50 pages deep for a given query, if you need more than that, narrow your numericFilters or date range and run multiple queries rather than fighting pagination limits.

5. Add proxies once you move to HTML scraping or high concurrency

The two APIs above are generous and don’t require proxies for normal use. Proxies become relevant in two cases: you’re scraping the HTML front-end for data the APIs don’t expose (user profile pages, hiring thread formatting, who's hiring replies before they’re indexed), or you’re running enough parallel workers that a single IP starts tripping rate limits.

import httpx

proxies = {
    "http://": "http://user:pass@proxy-host:port",
    "https://": "http://user:pass@proxy-host:port",
}
client = httpx.Client(proxies=proxies, timeout=15)
resp = client.get("https://news.ycombinator.com/user?id=someuser")

Expected output: normal 200 responses spread across rotating exit IPs instead of one IP taking the full request volume.

If it breaks: getting served the “you’re posting too fast” throttle page (it happens on the HTML side, not the API) means you need to slow down or rotate more aggressively, this isn’t a wall you brute-force through, it’s a signal to respect the site’s robots.txt and crawl-delay.

6. Store items in a database, not flat JSON files

Once you’re past a few hundred items, flat files become unmanageable for dedup and querying.

CREATE TABLE items (
    id INTEGER PRIMARY KEY,
    type TEXT,
    by TEXT,
    time INTEGER,
    title TEXT,
    score INTEGER,
    descendants INTEGER,
    raw JSON
);

Expected output: an upsert-friendly table where re-running your pull script updates scores and comment counts instead of duplicating rows.

If it breaks: if you’re seeing duplicate rows, you’re probably inserting instead of upserting, use INSERT OR REPLACE (SQLite) or ON CONFLICT DO UPDATE (Postgres) keyed on id.

7. Schedule incremental pulls instead of full re-crawls

Use updates.json from the Firebase API to get only items and profiles that changed since your last check, instead of re-pulling everything.

curl https://hacker-news.firebaseio.com/v0/updates.json

Expected output: {"items": [...], "profiles": [...]}, a small list you can hydrate and upsert on a 60-90 second cron.

If it breaks: if updates.json returns an empty or stale-looking list, check your polling interval, hitting it more than once every 30 seconds doesn’t get you fresher data, HN’s backend batches these updates on its own cadence.

8. Add monitoring so silent failures don’t rot your dataset

Log request counts, error rates, and a daily row count against expected volume (HN publishes roughly a few hundred new stories and several thousand comments a day).

Expected output: an alert if your daily ingested row count drops more than 30-40% versus a 7-day rolling average, that’s your signal something upstream changed or a proxy pool went stale.

If it breaks: if row counts silently drop to zero, check your proxy pool health before assuming the API changed, dead proxies fail closed far more often than HN’s API schema changes.

common pitfalls

• Treating this like a hostile scrape target. Hacker News runs on comparatively modest infrastructure for its traffic. Hammering it like you would a well-funded SaaS API gets you throttled fast and is just bad etiquette for a site that gives you a free, generous API.
• Ignoring dead and deleted flags. A meaningful chunk of comment trees include removed content, if you don’t filter it your sentiment analysis or training data gets skewed by empty or moderated text nodes.
• Recursing comment trees without a depth or concurrency cap. A viral thread can hit thousands of comments several levels deep, an uncapped recursive crawler will either blow your rate limit or your memory.
• Using proxies where they add no value. The Firebase and Algolia APIs don’t need proxy rotation for reasonable use, if you’re throwing residential proxies at topstories.json calls, you’re paying for infrastructure you don’t need.
• Not handling job and poll item types. Not every item is a story, pollopt and job types have different field shapes, code that assumes every item has title and url will throw on the ones that don’t.

scaling this

At 10x (a few thousand items a day), you don’t need proxies at all. A single script on a cron job hitting the Firebase and Algolia APIs directly, with basic backoff, handles this comfortably within both services’ informal fair-use expectations.

At 100x (tens of thousands of items a day, including full comment trees and some HTML fallback pulls), you want a small proxy pool, 5-10 rotating IPs, concurrency caps around 20-30 requests at a time, and a proper job queue instead of a single script so failed pulls retry without blocking the rest of the batch.

At 1000x (hundreds of thousands of items, near-real-time monitoring across the full site, plus deep historical backfill), you’re running a distributed worker pool, a larger rotating residential proxy plan, a real message queue (Redis or SQS) to distribute work, and you should be talking to HN’s maintainers or at minimum reading their guidelines closely, because at this volume you’re a meaningful fraction of the site’s total request load and unannounced heavy crawling is the kind of thing that gets IP ranges blocked outright. If you’re also managing scraper identities and browser fingerprints across a worker fleet at this scale, that’s the exact problem antidetectreview.org/blog covers in more depth than I will here.

where to go next

If Hacker News is one source in a broader pipeline, two follow-ups worth reading next: how to scrape GitHub at scale in 2026 with proxies that work covers the sibling developer-community dataset most HN scrapers eventually want too, and how to scrape Crunchbase at scale in 2026 with proxies that work is useful if you’re cross-referencing HN “Show HN” and “Launch HN” posts against funding data. For everything else we’ve published, check the blog index.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-15.