How to scrape GitHub at scale in 2026 with proxies that work
GitHub looks like an easy scrape until you actually try to pull data at volume. The repo pages render fine in a browser, but the moment you script requests against them you hit authentication walls, aggressive secondary rate limits, and a Terms of Service that explicitly restricts scraping for things like spam and surveillance. Most people who come to this problem are trying to build a dataset: repo metadata for a market map, contributor activity for a hiring tool, star/fork trends for a trend-tracking dashboard, or code search results for a research project.
This is for you if you’re pulling GitHub data in the thousands-to-millions of records range and your current script keeps dying to 403s, secondary rate limit messages, or token bans. It’s not for anyone trying to scrape private repos, harvest email addresses for outreach, or build a surveillance tool on top of user activity, that’s a ToS violation and I’m not going to help you build it.
By the end of this you’ll have a working setup that combines GitHub’s own REST/GraphQL APIs (which you should use for 90% of this job) with proxy-backed HTTP requests for the parts the API doesn’t cover, plus the rate-limit handling and monitoring that keeps the whole thing running unattended for weeks.
what you need
- A GitHub account with a fine-grained or classic personal access token. Free tier gives you 5,000 API requests/hour authenticated, which is more than most people expect. github.com signup, $0.
- A pool of GitHub tokens if you need more than 5,000 req/hour. One token per throwaway or org-linked account, generated via Settings > Developer settings > Personal access tokens.
- A proxy provider for the HTML-only parts. I use Decodo (formerly Smartproxy) for this kind of work, residential IPs, roughly $7/GB on their pay-as-you-go tier. Datacenter IPs are cheaper but get flagged faster on github.com’s HTML endpoints. See my full Decodo review if you’re picking a provider.
- Python 3.11+ with
requests,PyGithubor raw HTTP, and a task queue (even a simpleasyncio.Queueworks up to a few hundred thousand records). - A place to store results. Postgres or even SQLite for anything under a few million rows. Don’t overbuild this.
- Budget: figure $50-150/month in proxy bandwidth for a mid-size crawl (500k-2M pages/records), plus whatever compute you’re already running the script on. GitHub’s own API is free within rate limits.
- Patience for GitHub’s abuse detection. It’s genuinely aggressive on anything that looks automated hitting search or the web UI. Budget time for backoff tuning, not just infrastructure.
step by step
1. Decide what actually needs the API vs. the website
Before writing a line of code, map your data need to the right source. Repo metadata, issues, pull requests, commits, stars, forks, topics, and user profiles are all available through the REST API or GraphQL API. Code search, the trending page, and rendered README content with certain embeds are not fully covered and require hitting the actual web pages.
Expected output: a short list splitting your fields into “API-covered” and “needs HTML.”
If it breaks: if you’re not sure whether a field exists in the API, check the GraphQL explorer at https://docs.github.com/en/graphql/overview/explorer before assuming you need to scrape HTML. Nine times out of ten the API has it under a name you didn’t expect (e.g., stargazerCount not stars).
2. Generate and rotate personal access tokens
Create a fine-grained token scoped to read-only public repo access. For volume work, generate one token per GitHub account you legitimately control (don’t create fake accounts, that violates GitHub’s terms and gets ranges banned).
curl -H "Authorization: Bearer YOUR_TOKEN" \
-H "Accept: application/vnd.github+json" \
https://api.github.com/repos/torvalds/linux
Expected output: a JSON blob with repo metadata and, in the response headers, x-ratelimit-remaining counting down from 5000.
If it breaks: a 401 means the token is malformed or revoked, regenerate it. A 403 with rate limit exceeded means you’re already capped, check x-ratelimit-reset for the Unix timestamp when it clears.
3. Build the core fetcher with pagination handling
GitHub’s REST API paginates at 100 items per page using Link headers, not page-number query params you can guess blindly.
import requests, time
def fetch_all(url, token):
headers = {"Authorization": f"Bearer {token}", "Accept": "application/vnd.github+json"}
results = []
while url:
r = requests.get(url, headers=headers, params={"per_page": 100})
if r.status_code == 403 and "rate limit" in r.text.lower():
reset = int(r.headers.get("x-ratelimit-reset", time.time() + 60))
time.sleep(max(reset - time.time(), 1))
continue
r.raise_for_status()
results.extend(r.json())
url = r.links.get("next", {}).get("url")
return results
Expected output: a complete list across all pages without manual offset math.
If it breaks: if r.links comes back empty on a response you know has more pages, check that you’re not stripping the Link header somewhere upstream (some corporate proxies do this).
4. Add proxy rotation for the HTML-only slice
For code search or trending pages, route requests through rotating residential proxies since these endpoints get hit with browser-fingerprint and IP-reputation checks that the API doesn’t apply.
import requests
proxies = {
"http": "http://user:[email protected]:10000",
"https": "http://user:[email protected]:10000",
}
r = requests.get("https://github.com/search?q=proxy+rotation&type=code",
proxies=proxies,
headers={"User-Agent": "Mozilla/5.0"})
Expected output: a 200 with rendered search results HTML.
If it breaks: repeated 429s or a login wall usually means the IP got flagged. Rotate to a fresh sticky session on your provider’s dashboard rather than hammering the same exit IP.
5. Respect and monitor rate limit headers
Every API response carries x-ratelimit-limit, x-ratelimit-remaining, and x-ratelimit-used. Log these on every call so you can see your burn rate before you hit a wall, not after.
Expected output: a running log or dashboard showing remaining quota per token.
If it breaks: if remaining hits zero mid-run and your backoff logic doesn’t kick in, you’ll start seeing 403 abuse detection messages, which carry longer, sometimes unpredictable cooldowns per GitHub’s own rate limit documentation.
6. Handle secondary rate limits separately
GitHub has a second, undocumented-threshold limiter for concurrent requests and rapid-fire writes/searches, separate from the hourly quota. This is the one that actually bans people.
Expected output: your crawler should back off exponentially (start at 1s, double up to a cap) whenever it sees a retry-after header, not just when the primary quota is exhausted.
If it breaks: if you’re getting secondary limit errors with quota remaining, you’re firing requests too concurrently. Cap concurrent connections per token to 2-3, not 20.
7. Deduplicate and store incrementally
Use ETags on GET requests, GitHub returns a 304 Not Modified if nothing changed, which doesn’t count against your rate limit the same way.
headers["If-None-Match"] = last_seen_etag
Expected output: a growing dataset with far fewer wasted calls on unchanged records.
If it breaks: if you’re not seeing 304s ever, confirm you’re storing and resending the exact ETag string from the prior response, not a hash of your own.
8. Consider GH Archive before you scrape historical activity
If what you actually want is historical event data (commits, PRs, issues, stars over time) rather than a live snapshot, GH Archive already ingests every public GitHub event and makes it queryable via Google BigQuery’s public dataset. This is legitimate, free up to BigQuery’s query allowance, and saves you from scraping something GitHub already publishes in bulk.
Expected output: a BigQuery query returning years of event history in seconds instead of weeks of crawling.
If it breaks: BigQuery queries against the full dataset can get expensive if you scan more than the free tier allows, filter by date partition first.
common pitfalls
- Scraping the web UI for data the API already has. This is the single biggest time-waster I see. Check the REST/GraphQL docs before writing a single HTML parser.
- Running one token across too many concurrent workers. Secondary rate limits trigger on concurrency, not just volume, and the ban can extend to the whole token, not just the offending request.
- Ignoring GitHub’s Acceptable Use Policy. GitHub’s own acceptable use policies restrict scraping for spam, surveillance, or reselling user data. Build your use case inside those lines. This isn’t legal advice, if you’re planning anything commercial with user data, get it reviewed.
- Using datacenter proxies for the HTML-only endpoints. They get flagged faster than residential IPs on anything github.com serves outside the API, and you’ll burn bandwidth on retries instead of results.
- Not handling token revocation gracefully. Tokens expire or get revoked (by you, by GitHub, by an org admin). A crawler that crashes hard on a 401 instead of rotating to the next token in the pool will silently stop for hours before anyone notices.
scaling this
At 10x (tens of thousands of records), a single token, a single machine, and no proxies at all will get you through in a day or two, respecting the hourly quota. You don’t need this article’s proxy setup yet.
At 100x (hundreds of thousands to low millions), you need a small pool of tokens (5-10), per-token concurrency caps, and a real job queue so a crashed worker doesn’t lose in-flight state. This is also where GH Archive starts looking more attractive than live crawling for anything historical, since BigQuery does the heavy lifting for you.
At 1000x (multi-million record crawls, ongoing monitoring pipelines), you’re running a proxy pool for the HTML slice, a token rotation service, persistent storage with dedup on ETags, and alerting on rate-limit burn rate so a runaway job doesn’t quietly exhaust your whole token pool overnight. At this scale, also look at whether GitHub’s GraphQL API can replace several REST calls with one query, since GraphQL’s point-based rate limit is often more efficient for deeply nested data like a repo’s issues plus comments plus reactions in one shot. If you’re also managing many GitHub accounts to spread token load, the account-hygiene practices covered on multiaccountops.com are worth reading before you scale account count, since GitHub links accounts by behavioral and network signals, not just by token.
where to go next
- How to scrape Crunchbase at scale in 2026 with proxies that work, similar API-first approach for structured company data.
- Best proxies for scraping LinkedIn in 2026, useful if your GitHub dataset needs cross-referencing against professional profiles.
- Browse the full blog index for more scraping tutorials and proxy provider reviews.
Written by Xavier Fok
disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-15.