How to scrape Producthunt at scale in 2026 with proxies that work

Product Hunt sits on a pile of data that VCs, growth marketers, and indie hackers all want: daily launch rankings, maker profiles, upvote velocity, comment sentiment, topic tags, and the collections that surface which categories are heating up. The problem is that Product Hunt is a Next.js app behind Cloudflare, most of the interesting data doesn’t live in the official API, and a handful of makers have already gotten IP-banned trying to pull it with a bare requests script.

This is written for people doing this for a real reason: competitive intelligence on SaaS launches, building a trend-tracking tool, sourcing leads for outbound to makers who just launched, or feeding a research dataset. It is not written for anyone trying to fake upvotes or manipulate rankings, and I’m not going to help with that. Vote manipulation violates Product Hunt’s terms and can get accounts and IPs banned permanently.

By the end of this you’ll have a pipeline that pulls launch data through Product Hunt’s official GraphQL API where possible, falls back to a proxied headless-browser scrape for what the API doesn’t expose, and survives rotation, rate limits, and the occasional layout change without you babysitting it every day.

what you need

A Product Hunt account and a registered API application (free, done through their developer settings) for OAuth access to the GraphQL API v2
Python 3.11+ or Node 20+ (examples below use Python)
Playwright for the pages the API won’t give you (comment threads, some maker profile fields, historical launches beyond the API’s window)
A proxy pool that isn’t shared garbage. Datacenter IPs get flagged fast on Cloudflare-protected sites like this one. I run residential or ISP proxies for the scrape leg, our own Decodo review and SOAX comparison cover what’s worth paying for
A place to store output: Postgres, SQLite for small runs, or just parquet files if you’re doing one-off analysis
Budget for proxy bandwidth. Cost scales with pages fetched, not rows extracted, so the API-first approach matters for keeping this cheap
A scheduler (cron, GitHub Actions, or a simple systemd timer) if you want this running daily instead of manually

step by step

1. Decide what actually needs the API vs the browser

Before writing any code, map your fields against what the GraphQL API exposes: post title, tagline, votes count, maker info, topics, and comments are all in the official schema. What’s not reliably there: full comment reply threads, some maker social links, and anything from Product Hunt’s older archive pages that predate the API’s coverage.

Expected output: a short spec, literally a spreadsheet column list, marking each field “API” or “scrape.”

If it breaks: if you’re not sure a field exists in the schema, query it in Product Hunt’s GraphiQL explorer (linked from their docs) before you write a single line of scraper code. Guessing field names against the live API burns your token’s rate limit for nothing.

2. Register your API application and get a token

Go to Product Hunt’s developer settings, create an application, and grab your client ID and secret. Use the client-credentials OAuth flow for read-only public data, you don’t need user-level auth unless you’re posting or voting on someone’s behalf (don’t do that).

curl -X POST https://api.producthunt.com/v2/oauth/token \
  -H "Content-Type: application/json" \
  -d '{
    "client_id": "YOUR_CLIENT_ID",
    "client_secret": "YOUR_CLIENT_SECRET",
    "grant_type": "client_credentials"
  }'

Expected output: a JSON blob with an access_token you attach as a bearer token on every GraphQL request afterward.

If it breaks: a 401 usually means you copied the secret wrong or your app got deactivated for inactivity. Regenerate it in the dashboard. A 403 on specific queries means you’re asking for a field that needs user-level scope, not client-credentials scope.

3. Pull the daily leaderboard through GraphQL first

import requests

query = """
query DailyPosts($first: Int!, $after: String) {
  posts(order: VOTES, first: $first, after: $after) {
    edges {
      node {
        id
        name
        tagline
        votesCount
        website
        topics(first: 5) { edges { node { name } } }
        makers { name, headline, twitterUsername }
      }
    }
    pageInfo { hasNextPage, endCursor }
  }
}
"""

resp = requests.post(
    "https://api.producthunt.com/v2/api/graphql",
    json={"query": query, "variables": {"first": 20, "after": None}},
    headers={"Authorization": f"Bearer {access_token}"},
)
data = resp.json()

Expected output: paginated JSON with up to 20 posts per page, cursor-based pagination via endCursor.

If it breaks: a 429 means you’ve hit the complexity-based rate limit on your token. Back off and retry with exponential backoff, don’t just hammer it again immediately. Track your remaining quota from the response headers Product Hunt returns.

4. Set up your proxy pool for the browser leg

For the fields the API doesn’t cover, you’ll render pages with Playwright, and Cloudflare will fingerprint a naked datacenter IP fast. Route traffic through a residential or ISP proxy pool with sticky sessions per scrape job, so a single product page’s requests all come from one IP rather than round-robining mid-session, which looks more like a bot than a normal user browsing.

from playwright.sync_api import sync_playwright

proxy = {
    "server": "http://gate.smartproxy.com:10000",
    "username": "your-user-session-abc123",
    "password": "your-pass",
}

with sync_playwright() as p:
    browser = p.chromium.launch(proxy=proxy, headless=True)
    page = browser.new_page()
    page.goto("https://www.producthunt.com/posts/example-product")
    page.wait_for_selector("[data-test='comment']")
    comments_html = page.content()
    browser.close()

Expected output: rendered HTML including comment threads that never show up in a plain curl request because they’re client-side rendered.

If it breaks: if pages load but data is missing, the selector likely changed. If pages hang or return a Cloudflare challenge page, your proxy’s reputation is probably burned, rotate to a fresh IP from the pool and increase the delay between requests.

5. Respect robots.txt and throttle deliberately

Check https://www.producthunt.com/robots.txt before you scrape anything and build your crawl paths around it. The Robots Exclusion Protocol is now a formal IETF standard, and ignoring it isn’t just bad etiquette, it’s the fastest way to get your IP range permanently blacklisted at the CDN level.

Expected output: a crawl delay you actually honor, and a clear list of paths you won’t touch.

If it breaks: if your scraper is getting blocked despite following robots.txt, the issue is usually request rate, not path. Scrapy’s AutoThrottle is a good reference implementation for adaptive delay even if you’re not using Scrapy itself, borrow the logic.

6. Parse and normalize into a stable schema

Whatever comes back from the API or the browser, map it into one internal schema (post_id, name, tagline, votes, topics, maker_ids, launched_at, comment_count) before it hits storage. Don’t let downstream code deal with two different shapes depending on which path the data came from.

Expected output: a single normalized table or JSON schema regardless of source.

If it breaks: if fields are null intermittently, it’s almost always a partial page load in the browser leg, not the API. Add a retry with a longer wait_for_selector timeout before falling back to a null.

7. Store with dedup keys

Use post_id as your unique key and upsert rather than insert, since you’ll be re-pulling the same daily leaderboard repeatedly to track vote velocity over time.

INSERT INTO ph_posts (post_id, name, votes_count, scraped_at)
VALUES (%s, %s, %s, now())
ON CONFLICT (post_id) DO UPDATE
SET votes_count = EXCLUDED.votes_count, scraped_at = now();

Expected output: a growing time series per post instead of duplicate rows.

If it breaks: if vote counts look frozen, your job is probably hitting a cached CDN response. Vary a cache-busting query param or check your proxy isn’t serving a stale edge cache.

8. Schedule it and monitor for schema drift

Run this daily via cron or a GitHub Actions scheduled workflow. Add a cheap sanity check: if the number of posts returned drops to zero or field parsing fails on more than a few percent of rows, alert yourself instead of silently writing garbage.

Expected output: a daily run log with row counts and error rates.

If it breaks: sudden zero rows almost always means Product Hunt shipped a frontend change that broke your selectors, or Cloudflare started challenging your proxy pool’s ASN. Check both before assuming it’s your code.

common pitfalls

Skipping the API entirely. Some operators go straight to browser scraping because it feels more “complete,” then burn proxy bandwidth on data the API hands over for free. Always check the API docs first.
Using datacenter proxies on the browser leg. They work for a day or two, then every request starts hitting a Cloudflare challenge page. Residential or mobile IPs cost more per GB but survive far longer.
Not sticking sessions. Rotating IP per request mid-session on a single product page reads as automated traffic. Keep one IP per scrape session, rotate between sessions.
Ignoring rate limit headers. The GraphQL API tells you your remaining complexity budget. Operators who ignore it and just retry on 429 dig themselves into longer and longer backoff penalties.
No dedup key. Without upserting on post_id, your dataset balloons with duplicate rows every run and vote-velocity analysis becomes useless.

scaling this

At 10x (a few hundred posts a day, maybe tracking a couple hundred makers), the API alone covers most of your needs and a single proxy IP with reasonable throttling is fine. Cost is mostly your time.

At 100x, you’re pulling comment threads and maker profiles across thousands of posts, which pushes real volume onto the browser leg. You need a proper rotating residential pool, not a single sticky IP, and you should split your scrape jobs across worker processes so one slow page doesn’t stall the whole run. This is also the point where fingerprinting starts to matter, browser automation leaves detectable traces beyond just the IP, and it’s worth understanding antidetect browser setups if you’re running many concurrent sessions. antidetectreview.org covers the tooling for that specifically.

At 1000x, you’re not scraping Product Hunt anymore, you’re running a data pipeline that happens to include it. At this scale you need distributed job orchestration (Celery, Temporal, or similar), proxy pools sized in the tens of thousands of IPs, and you should be talking to Product Hunt about a data partnership or higher API access rather than fighting their bot defenses. Most sites, including Product Hunt, would rather grant a legitimate research or business partner higher API limits than have their infrastructure treated as an adversary. This is not legal advice, but at this volume it’s worth having a lawyer review your data use against Product Hunt’s terms of service before you scale further.

where to go next

If you’re building a broader startup-intelligence pipeline, pair this with scraping Crunchbase for funding and company data, and scraping G2 for the review-side signal on the same SaaS products launching on Product Hunt. Both use the same proxy-and-throttle playbook covered here. For the full backlog of scraping tutorials and proxy reviews, check the blog index.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-16.