How to scrape Crunchbase at scale in 2026 with proxies that work

Crunchbase sits on one of the better structured startup and funding datasets on the web. Company profiles, funding rounds, investor names, acquisition history, all cross-linked and reasonably clean. The catch is that Crunchbase makes almost none of that usable at real volume unless you pay for it. The free tier caps you at a handful of profile views a month, the Pro tier still throttles bulk access, and the moment you start hitting pages programmatically from a single IP, their Cloudflare setup flags it within minutes.

This tutorial is for people who need Crunchbase data across hundreds or thousands of companies at once: sales teams building account lists, VCs tracking funding activity in a sector, market researchers building comparables, or engineers feeding a data product. It’s not a guide for scraping employee contact details or building spam lists, and it’s not a workaround for Crunchbase’s paid API. If your use case is commercial and ongoing, at real scale the official Crunchbase API is usually the cheaper and safer option once you run the numbers on proxy bandwidth and engineering time.

By the end you’ll have a scraper that pulls company and funding-round data from public Crunchbase pages through rotating residential proxies, gets past the Cloudflare challenge, parses the JSON Crunchbase embeds on every page, and doesn’t burn your IP pool in the first hour.

what you need

• A rotating residential proxy plan. Datacenter IPs get blocked by Crunchbase’s Cloudflare setup almost immediately. Budget $4 to $9 per GB with a provider like Decodo or IPRoyal, see our Decodo review if you haven’t picked a vendor yet.
• Python 3.11+ with httpx and playwright installed, or an equivalent stack in Node.
• A headless browser environment for pages that gate content behind a JS challenge. playwright-stealth or a similar patch helps reduce bot fingerprinting.
• Somewhere to store output. Postgres is fine for a few hundred thousand rows, S3 plus parquet if you’re going past that.
• A free Crunchbase account at minimum, a Pro subscription (roughly $29 to $49/mo as of mid-2026) if you want to manually spot-check what your scraper pulled against the real UI.
• Ten minutes to actually read Crunchbase’s robots.txt and terms of use before you start, so you know which paths are disallowed and can scope your crawl to stay off them.
• If you’re juggling multiple scraper identities or browser profiles across projects, the session and fingerprint isolation writeups at multiaccountops.com/blog are worth a read. The same principles that keep social accounts from getting linked apply to keeping scraper sessions apart.

step by step

1. Pick a proxy pool and confirm it isn’t already burned

Action: sign up for a rotating residential proxy plan and fire 20-30 test requests at a handful of public Crunchbase organization pages (e.g. crunchbase.com/organization/openai) before you build anything else.

Expected output: mostly 200 responses, with an occasional Cloudflare interstitial page mixed in.

import httpx

proxy = "http://user:[email protected]:7000"
url = "https://www.crunchbase.com/organization/openai"
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}

with httpx.Client(proxy=proxy, timeout=20) as client:
    r = client.get(url, headers=headers)
    print(r.status_code, len(r.text))

If it breaks: if you’re getting blocked on request one or two, the pool is already flagged, common with cheap shared blocks resold as “residential” that are actually recycled datacenter ranges. Switch providers or request a fresh subnet before you write another line of code.

2. Set up a headless browser for the Cloudflare challenge

Action: install Playwright and a stealth plugin, then load a test page through it with your proxy attached rather than a bare httpx client.

pip install playwright playwright-stealth
playwright install chromium

from playwright.sync_api import sync_playwright
from playwright_stealth import stealth_sync

with sync_playwright() as p:
    browser = p.chromium.launch(proxy={"server": "http://gate.proxyvendor.com:7000",
                                        "username": "user", "password": "pass"})
    page = browser.new_page()
    stealth_sync(page)
    page.goto("https://www.crunchbase.com/organization/openai", timeout=30000)
    html = page.content()
    browser.close()

Expected output: full rendered HTML including the client-side data, not just a challenge page.

If it breaks: if you’re stuck on the interstitial for more than a couple seconds, add page.wait_for_load_state("networkidle") and slow your request rate. Persistent stalls usually mean the proxy IP itself has a bad reputation score with Cloudflare, not a bug in your script.

3. Build your target URL list

Action: assemble the list of organization slugs you want (from a CSV export, a sector search on Crunchbase itself, or a seed list of known competitors) and normalize them into full URLs.

Expected output: a clean text file or database table of https://www.crunchbase.com/organization/<slug> URLs, deduplicated.

If it breaks: slugs sometimes change when companies rebrand. If a URL 404s, search the company name on Crunchbase manually to grab the current slug rather than assuming your list is wrong.

4. Extract the embedded JSON, don’t parse the HTML

Crunchbase renders profile data client-side from a JSON payload embedded in a script tag on page load. Parsing the rendered HTML directly with CSS selectors is brittle, class names shift with every frontend deploy. Pulling from the embedded JSON blob is far more stable across redeploys.

import re, json

match = re.search(r'window\.__INITIAL_STATE__\s*=\s*(\{.*?\});', html, re.DOTALL)
if match:
    data = json.loads(match.group(1))

Expected output: a nested dict with company name, founding date, funding rounds, and investor names.

If it breaks: if the regex comes back empty, Crunchbase has likely changed the variable name or moved to a different hydration pattern. Open dev tools on a live page and search the page source for __INITIAL_STATE__ or application/json script tags to find the new pattern.

5. Rotate identity and throttle deliberately

Action: cap your request rate per proxy IP (2-4 requests per minute per session is a reasonable starting point) and rotate to a new IP on a sticky session timer, not on every single request.

Expected output: steady completion rate with block rate under 5%.

If it breaks: if blocks climb above 15-20%, you’re rotating too fast or too slow for your provider’s pool. Slow down first, most bans come from request velocity, not raw volume.

6. Parse and normalize the data

Action: map the fields you care about (name, HQ location, founding date, total funding, latest round, investors) into a flat schema you control, independent of Crunchbase’s internal field names.

Expected output: a consistent row per company regardless of which page layout variant it came from.

If it breaks: some fields are null for companies with sparse profiles, that’s a data-quality gap on Crunchbase’s end, not your parser’s fault. Handle nulls explicitly rather than crashing the pipeline.

7. Store and dedupe

Action: write to Postgres (or your store of choice) with an upsert keyed on the Crunchbase slug, so re-runs update stale rows instead of creating duplicates.

Expected output: idempotent runs, row count grows only with genuinely new companies.

If it breaks: duplicate rows usually mean you’re keying on an internal database ID that changes between scrapes instead of the stable slug.

8. Monitor your block rate over time

Action: log status codes and Cloudflare challenge occurrences per run, and chart the block rate daily.

Expected output: a flat or slowly declining block rate as your proxy vendor’s pool ages in gracefully.

If it breaks: a sudden spike usually means Crunchbase updated its bot detection rules or your proxy provider’s IP block got reported and burned as a whole. Rotate to a different provider or subnet rather than waiting it out.

common pitfalls

1. Using datacenter proxies to save money. They’re 5-10x cheaper per GB but get flagged by Cloudflare almost instantly on a site like Crunchbase. You end up spending more in wasted requests and retries than you’d have spent on residential IPs.
2. Ignoring robots.txt entirely. Even where scraping public pages sits in a legal gray area, as established in cases like hiQ Labs v. LinkedIn, deliberately crawling paths a site has marked disallowed is an easy way to turn a defensible position into an indefensible one if you ever need to explain your methodology to a lawyer or a client.
3. Parsing rendered HTML instead of the embedded JSON. You’ll rebuild your selectors every few weeks as Crunchbase ships frontend updates. The JSON payload changes far less often.
4. Treating the login wall as a bug rather than a rate limit. Crunchbase intentionally caps free profile views. Don’t burn engineering time trying to “fix” a wall that’s a deliberate business decision, work around it with proxy rotation and pacing instead.
5. Not distinguishing aggregate company data from personal data on individual employee profiles. Crunchbase also lists people. Scraping structured company and funding data is a different risk profile than harvesting names, titles, and photos of individuals for outreach lists. Keep those use cases separate in your own head and in your data retention policy.

scaling this

10x (a few thousand pages a day): one machine, one rotating residential proxy plan, a simple queue (even a Python list with retries works), Postgres for storage. Cost is mostly proxy bandwidth, expect low double-digit dollars a day depending on page size.

100x (tens of thousands a day): you need a real job queue (Celery, RQ, or similar), multiple concurrent workers, and a proxy plan with enough concurrent sessions to match. Sticky sessions become important here so you’re not re-authenticating or re-triggering Cloudflare challenges on every request. Budget scales roughly linearly with volume, so track cost per 1,000 successful pulls as your core efficiency metric, not just total spend.

1000x (hundreds of thousands a day and up): at this point, the economics usually flip. Proxy bandwidth costs, block-rate losses, and the engineering time spent babysitting a scraper against an actively defended target start to exceed what Crunchbase charges for enterprise API access. This is the point to seriously evaluate the Crunchbase API or a data licensing conversation with their team instead of scaling scraping infrastructure further. It’s also the point where legal exposure from ToS violations at volume becomes a real business risk, not a theoretical one, so loop in counsel before committing engineering budget to scraping infrastructure at this scale. This isn’t legal advice, it’s a practical observation that the math and the risk both tend to favor licensing once you’re past a certain volume.

where to go next

If you’re building out a broader scraping stack rather than just a Crunchbase pull, a few related guides on this site are worth reading next:

• How to scrape Airbnb at scale in 2026 with proxies that work for a similar walkthrough against a different Cloudflare-protected target.
• Best proxies for scraping LinkedIn in 2026 if you’re also pulling company or people data from LinkedIn, since the anti-bot posture is comparably aggressive.
• Decodo review 2026: honest pros, cons, and pricing for a deeper look at one of the residential proxy vendors referenced above.

You can browse the full archive on the blog index for more proxy comparisons and target-specific scraping guides as they go live.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-14.