How to scrape G2 at scale in 2026 with proxies that work

G2 is one of the better data sources in B2B software if you’re building a competitive intel feed, a pricing tracker, or a lead list sorted by who’s actively shopping for a category. Review counts, star ratings, “market presence” grid positions, pricing pages, and alternative-to comparisons are all sitting there in structured form. The catch is that G2 knows this too, and it has invested in bot detection to protect that data, which means a naive requests.get() loop dies after a few hundred pages.

This tutorial is for people who already run scraping infrastructure for other targets (Amazon, LinkedIn, Crunchbase) and want the G2-specific playbook: what breaks, what proxy type actually holds up, and how to structure the crawl so it survives past a weekend project. I’m not going to pretend this is trivial or that a $10/month scraper is going to hoover up the entire G2 catalog. It isn’t. But a properly built pipeline running on residential proxies will get you clean, structured data on a schedule, without your IPs getting burned every few hours.

I’ll cover prerequisites, an 8-step build, the mistakes I see people make repeatedly, and what actually changes as you scale from a hobby crawl to something closer to a production feed.

what you need

• A residential or mobile proxy pool. Datacenter IPs get flagged fast on G2. Providers like Decodo, Smartproxy, and Oxylabs all sell residential pools billed by the GB, typically in the mid-single-digit to low-teens dollar range per GB depending on volume tier, check each provider’s current pricing page since these move often.
• Python 3.11+ with playwright and httpx installed (pip install playwright httpx then playwright install chromium).
• A headless browser context manager. G2’s category and comparison pages render a meaningful chunk of content client-side, so a pure HTTP client will miss data on those page types.
• A datastore. SQLite is fine under a few hundred thousand rows; move to Postgres once you’re running multiple crawlers against it concurrently.
• A queue or scheduler. Cron is enough at small scale; something like a Redis-backed task queue once you’re running parallel workers.
• A realistic budget line. Between proxy bandwidth, compute, and your time debugging selector changes when G2 redesigns a page, budget this as an ongoing line item, not a one-time cost.
• A legal read on what you’re allowed to do with the output, especially if you plan to redistribute or resell scraped review text. This is not legal advice, talk to counsel if you’re doing anything commercial with the data.

step by step

1. Define exactly what you need before writing a line of code

Decide up front whether you need company profile pages, review text, star rating aggregates, category grids, or pricing pages. Each of these is a different template with different anti-bot exposure. Trying to build one crawler that hits all of them at once is how projects stall.

Expected output: a short spec, literally a text file, listing target URL patterns and the fields you need from each.

If it breaks: if you can’t articulate the field list in one sitting, you’re not ready to build yet, go back and manually inspect 10-15 target pages first.

2. Set up your proxy pool and confirm rotation actually works

Configure your proxy provider’s rotating endpoint (most residential providers give you a single gateway host that rotates the exit IP per request or per session). Test it with a simple script before touching G2 at all.

import httpx

proxy = "http://user:[email protected]:10001"
for _ in range(5):
    r = httpx.get("https://ifconfig.me/ip", proxy=proxy, timeout=10)
    print(r.text.strip())

Expected output: five different IP addresses printed, confirming rotation.

If it breaks: if you see the same IP repeated, check whether you’ve accidentally pinned a “sticky session” flag in the proxy username string, most providers use a session-<id> suffix for sticky sessions and you don’t want that here.

3. Set headers and TLS fingerprint to match a real browser

G2’s edge sits behind Cloudflare, you can confirm this yourself by checking response headers for cf-ray on any g2.com request. A bare Python HTTP client has a TLS handshake fingerprint that’s trivially distinguishable from Chrome, and it’s a common early flag. Route your requests through playwright’s Chromium so the TLS and header fingerprint match a real browser rather than trying to hand-spoof headers on a raw socket client.

Expected output: page loads return full HTML with review content present, not a challenge page.

If it breaks: if you’re getting a JS challenge or a 403, your fingerprint still doesn’t match, switch from raw HTTP entirely to a real browser context.

4. Build the crawler on Playwright, not requests

from playwright.sync_api import sync_playwright

def fetch(url, proxy):
    with sync_playwright() as p:
        browser = p.chromium.launch(proxy={"server": proxy})
        page = browser.new_page()
        page.goto(url, wait_until="networkidle", timeout=30000)
        html = page.content()
        browser.close()
        return html

Playwright’s official docs cover proxy configuration, stealth-adjacent context options, and waiting strategies in more depth than I can here.

Expected output: rendered HTML including review cards that only appear after JS execution.

If it breaks: if networkidle times out repeatedly, the page may be polling an analytics endpoint indefinitely, switch to wait_until="domcontentloaded" plus an explicit wait for your target CSS selector instead.

5. Throttle and add jitter, don’t hammer

Set a random delay between 3-9 seconds per request per identity (IP + browser context pair), and cap concurrent sessions well below what you think you can get away with. G2’s rate limiting responds to burst patterns more than raw volume.

Expected output: sustained crawl over hours without a spike in 403s or CAPTCHA challenges.

If it breaks: if you see challenges climbing over a session, you’re going too fast on that identity, rotate the proxy and widen the delay window before retrying that URL.

6. Parse into a fixed schema

Pull review text, rating, reviewer role/company size (G2 shows this as metadata on each review), review date, and product category into a fixed schema before storing. Don’t store raw HTML as your primary record, it makes downstream analysis painful and eats disk.

Expected output: one row per review with consistent columns, ready to load into SQLite or Postgres.

If it breaks: if fields are coming back empty on some pages, G2 likely A/B tests template variants, log the raw selector miss and inspect that specific page manually rather than assuming your whole parser is broken.

7. Dedupe and run incrementally

On repeat crawls, check review IDs (visible in the page’s embedded JSON, not just visible text) against what’s already in your store and skip unchanged records. This cuts your bandwidth bill dramatically after the first full pass.

Expected output: second and subsequent crawls of the same company touch a fraction of the pages the first one did.

If it breaks: if your dedupe key keeps producing false negatives, you’re probably keying off review text instead of the stable ID field, switch keys.

8. Schedule, log, and alert

Put the crawl behind cron or a lightweight task queue, log every non-200 response with the URL and proxy identity used, and alert yourself if the block rate crosses a threshold (I use 15%) over a rolling hour.

Expected output: a crawl that runs unattended and tells you when something’s degrading before you notice from missing data.

If it breaks: if block rate is spiking site-wide rather than on specific identities, G2 likely shipped a detection update, pause the crawl and re-check your fingerprint setup (step 3) before resuming.

common pitfalls

• Datacenter-only proxies. They’re cheap and they die fast on G2. Budget for residential from the start rather than “upgrading” after you’ve burned a week debugging phantom blocks.
• Rotating IP but not the browser fingerprint. A new IP with the same canvas/WebGL fingerprint and cookie jar is still recognizable as the same client to a sophisticated bot system.
• No jitter, fixed intervals. A crawler that hits every 2.000 seconds exactly is a pattern, not traffic.
• Ignoring robots.txt entirely. Check G2’s /robots.txt and respect disallowed paths where you can; the robots exclusion standard isn’t legally binding everywhere, but ignoring it outright raises both legal and reputational risk for no upside, since disallowed paths are rarely where the useful structured data lives anyway.
• Assuming public data scraping is automatically fine. The legal landscape here has shifted with cases like hiQ Labs v. LinkedIn, and the DOJ’s own guidance on the Computer Fraud and Abuse Act is worth reading if you’re scraping anything gated behind a login or a ToS you’ve agreed to. This isn’t legal advice, get a lawyer’s read if this is commercial.

scaling this

10x (a few thousand pages a week): one machine, one proxy pool, manual monitoring is fine. Cron plus SQLite covers it.

100x (tens of thousands of pages, multiple categories tracked daily): you need a real proxy management layer, session-to-identity mapping, and a task queue with retry logic. Move to Postgres. Expect to spend real time tuning delay windows per page template since review pages and pricing pages behave differently under load.

1000x (continuous crawl across most of G2’s catalog): this is where a single proxy provider’s pool often isn’t diverse enough, you’ll want to blend providers or move to a proxy management platform that pools multiple upstream networks. CAPTCHA-solving services become a line item, not an edge case. At this scale it’s worth directly asking G2 about a data partnership or API access rather than fighting their bot detection indefinitely, some SaaS review platforms do license data commercially and it’s a more durable path than a crawl that a detection update can break overnight.

If your crawl targets involve managing many separate accounts or browser identities rather than just IPs, that’s a different problem set, antidetectreview.org/blog covers antidetect browser tooling in more depth than fits here.

where to go next

• If you’re scraping B2B company data more broadly, our Crunchbase scraping tutorial covers a similar target with different anti-bot characteristics.
• Choosing a proxy provider is half the battle here, Decodo vs Smartproxy 2026 head to head compares the two most common picks for this kind of crawl.
• For more tutorials like this one, browse the blog index.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-15.