Honeypot links and hidden traps in page markup
The link nobody could see
A crawler that follows every link on every page will, sooner or later, follow one that no human visitor could have clicked. The link is in the HTML, so the crawler finds it. It isn’t on the screen, so no person ever does. The site owner put it there for exactly that reason. Any client that requests that URL has told the site something precise about itself: it reads raw markup and follows everything, without looking at what the page shows or what the site’s rules allow.
That’s a honeypot link. It’s one of the oldest and simplest ways a site separates indiscriminate bots from people and from well-behaved crawlers, and it’s still in use because it works.
This piece explains how these traps are built, the wider family of traps that sit alongside them, and why they catch the crawlers they catch. It’s written from the defensive side on purpose. The lesson for anyone running a scraper is about scope. A crawler with a disciplined scope, one that fetches what the sitemap lists, respects the robots rules and follows only the link patterns it has a reason to follow, never comes near these traps at all.
The related pieces on this site cover planning a crawl from the sitemap, soft blocks that return 200, and the general reasons scrapers get blocked. This one sits between them: it’s about what happens when crawl scope is loose, and how the site notices.
One framing point before going further. A site that plants a honeypot is stating a preference in the clearest way it can: it doesn’t want indiscriminate crawling. Treating that as a puzzle to solve is the wrong reading of the situation. Treating it as information about how the site wants to be crawled is the right one.
How a honeypot link is built
The idea is simple. Put a link in the page’s markup that a person using a browser won’t see or can’t reasonably click, and point it at a URL that nothing legitimate should ever request.
There are several common ways to hide the link from people. It can be styled so it’s not displayed at all. It can be positioned far off the visible area of the page. It can be given zero size, or text the same colour as the background. It can sit inside an element that’s hidden until some interaction that never happens. It can be marked so assistive technology skips it too, which matters because a well-built honeypot shouldn’t trap a screen reader user. Some sites generate the hidden link with slightly different markup on each page load, so it can’t be recognised by a fixed pattern.
The destination is usually a URL that exists only as a tripwire. Often it’s listed as disallowed in the site’s robots file, which means compliant crawlers are told explicitly not to fetch it. Sometimes it carries a unique token per page view, so that a request to it can be tied back to the exact page load, and therefore to the exact client, that discovered it.
When a request arrives at that URL, the site has a very strong signal. A human didn’t click an invisible link. A search engine crawler that respects robots didn’t fetch a disallowed path. What’s left is a client that parses markup and follows links without regard to visibility or rules.
The response to that signal varies by site. Some log it and watch. Some block the address, or the session, or the range. Some start serving different content to that client. Some feed the information to a bot management vendor, which may share it with the vendor’s other customers.
Why traps work so well for the site
From the site’s point of view, a honeypot has properties that most bot detection methods lack.
It has very few false positives. Ordinary visitors don’t see the link and don’t click it. Accessibility tools, if the honeypot is built with care, skip it. Well-behaved crawlers read the robots file and don’t request the path. So nearly everything that arrives at the trap URL is exactly what the site wanted to catch. That matters, because the costliest mistake for a site is blocking real customers, and a trap is one of the few mechanisms that almost never does.
It’s cheap. There’s no scoring model, no client-side script to maintain, no fingerprint database. It’s a link and a log line.
It catches behaviour, not identity. A trap doesn’t care what address a request comes from, what client it claims to be, or how convincing its headers look. It catches the one thing that distinguishes an indiscriminate crawler: the decision to request a URL that no reasonable reader of the page would request. Changing the address, the user agent or the TLS fingerprint doesn’t change that decision.
And it’s quiet. The crawler usually has no idea it tripped anything. The trap URL may return an ordinary-looking page, so from the crawler’s side everything looks normal until the consequences arrive, often on requests that have nothing to do with the trap.
These properties are why traps have outlasted many fancier techniques. Long-running community projects have used trap pages and hidden addresses for years to identify email harvesters and comment spammers, and the same logic carries straight over to web scraping.
The wider family of traps
Hidden links are the best-known kind, but a crawler with loose scope can walk into several related things, some deliberate and some accidental.
Robots-disallowed tripwires. A path listed as disallowed in robots, with no hidden link needed at all. The site simply watches for anyone requesting it. Crawlers that ignore robots find these through ordinary links, sitemaps of other sections, or guesswork, and identify themselves the moment they fetch.
Decoy content for non-compliant crawlers. Some sites and CDNs now serve generated, plausible-looking pages to crawlers that ignore their no-crawl signals, sending them through a maze of links that lead only to more generated pages. At least one large CDN has announced a feature along these lines aimed at AI crawlers that disregard opt-out directives. A crawler that wanders into it spends bandwidth and time collecting text that isn’t the site’s real content.
Tarpits. Endpoints that respond extremely slowly, holding connections open to waste a crawler’s resources. A crawler with high concurrency and no timeouts can have most of its workers stuck in one. The fix is ordinary engineering: a connect timeout, a read timeout and a total time limit per request, set to values that fit the target’s normal response times. A request that blows through them is a signal worth logging, whatever the cause.
Hidden form fields. Forms often include a field that people can’t see and shouldn’t fill in. Automated submitters that fill every field reveal themselves. For scraping, the lesson is simpler: a scraper has no business submitting forms on sites it’s reading, so this trap should never apply.
Accidental traps. Calendars that link to the next month forever. Faceted navigation where every combination of filters produces a new URL. Session identifiers embedded in links, making every page look new. Relative links that stack path segments. None of these is meant to catch anyone, but they catch unscoped crawlers just as effectively, and they cost the site capacity while doing it.
Why sitemap-led crawlers never touch them
A honeypot link has one defining feature: it exists only in the markup, for crawlers that read markup and follow everything. It isn’t in the sitemap. It isn’t in the site’s navigation as people see it. It isn’t anywhere a crawler would look if the crawler were trying to fetch what the site offers.
A sitemap is the site’s own list of the pages it wants discovered. It usually carries last-modified dates and sometimes priorities. The piece on planning a crawl from the sitemap covers how to turn one into a crawl plan. The relevant point here is that a site doesn’t put its tripwires in its sitemap, because the sitemap is for crawlers it welcomes. A crawl that starts from the sitemap and fetches those URLs is, by construction, fetching what the site has published for crawling.
The robots file works the same way from the other direction. It tells crawlers which paths not to fetch. Trap URLs are commonly listed there, because the whole point is that compliant crawlers never request them. A crawler that checks every URL against robots before fetching it skips those paths without needing to know they’re traps.
Put those two together and the trap problem mostly disappears. A crawler that fetches what the sitemap lists, and checks robots before every request, has no path to a honeypot. It doesn’t need to detect hidden links, analyse styles, or guess which URLs are suspicious. It simply never asks for anything outside the site’s published scope.
This is the core of the defensive lesson. Trap avoidance is a side effect of good scope. A crawler built to respect what the site offers and what it forbids doesn’t encounter the mechanisms built to catch crawlers that don’t.
Robots rules as the map of places not to go
Since the robots file carries so much of the weight here, it’s worth handling properly.
Fetch it before anything else, at the start of each crawl, and cache it. Refresh it on a schedule, daily is typical, since sites change their rules and a crawler running for weeks on a stale copy will drift out of compliance.
Parse it with a real parser, not a string search. The format has details that trip up hand-written checks: groups for specific user agents, the choice of which group applies to your crawler, wildcard and end-of-path patterns in many implementations, and precedence between allow and disallow rules. Python ships a basic parser in its standard library. More complete parsers exist in most languages that follow the current standard for the format more closely. Use one that matches how major crawlers interpret the file.
Check every URL before every fetch, including URLs discovered mid-crawl, redirect targets and URLs taken from the sitemap. Sitemaps occasionally list URLs that robots disallows, usually by mistake. When they conflict, robots wins.
Respect crawl-delay if present, and any rate guidance the site publishes elsewhere. Some crawlers ignore crawl-delay. A scraper that wants to stay on good terms with a site should honour it.
Pay attention to user-agent groups. If the file has specific rules for named crawlers and a default group for everyone else, a scraper that identifies itself with its own descriptive user agent falls under the default group. Borrowing a search engine’s user agent to get that engine’s more permissive rules is impersonation, and it’s also easy for sites to check.
Treat disallowed paths as places you’ve been asked not to go, whether or not they’re traps. That’s the whole point.
Following links only when there’s a reason
Some jobs can’t be done from the sitemap alone. The site might not have one, it might be incomplete, or the data might sit on pages the sitemap doesn’t list. Then the crawler has to discover URLs by following links. That’s where scope discipline matters most.
The unsafe pattern is “follow every link on the page within the same domain”. It’s the default in many simple crawlers, and it’s exactly the behaviour honeypots and accidental traps are built to catch.
The safer pattern is follow rules tied to page types. Decide which kinds of pages the job needs, such as category listings and product detail pages, plus pagination within those listings, and write rules that match only those URL patterns. From a listing page, follow links to product pages and to the next page of the listing. From a product page, follow nothing, or only links to its category. Everything else on the page is ignored.
Pattern rules should be specific. A rule that matches any URL containing a certain word will eventually match something it shouldn’t. A rule that matches a precise path structure with a numeric identifier is much harder to fool.
Stay on the target host unless the job explicitly requires otherwise. Links to other domains are outside scope by default.
Honour the site’s link hints where they’re relevant. Links marked nofollow are a signal to search engines about endorsement rather than a crawling rule, but a scraper that skips them loses little and avoids some trouble. Canonical tags tell you which version of a page the site considers primary, which helps avoid fetching duplicates.
Every URL a crawler fetches should have a reason that could be written down: this is a product page, found on this listing, matching this rule, allowed by robots. If a URL can’t be explained that way, the crawler shouldn’t fetch it.
Writing the scope down as data
Scope discipline is easier to keep when the scope is written down somewhere other than inside the crawler’s code. A small, explicit definition per target makes it reviewable, testable and hard to loosen by accident.
Something like this is enough for most jobs:
target: shop.example
seeds:
- sitemap: https://shop.example/sitemap.xml
allow:
- pattern: ^/category/[a-z0-9-]+/?$
follow: [product, category_page]
- pattern: ^/category/[a-z0-9-]+/\?page=\d{1,3}$
name: category_page
follow: [product, category_page]
- pattern: ^/product/\d+/[a-z0-9-]+/?$
name: product
follow: []
drop_params: [utm_source, utm_medium, utm_campaign, sessionid, sort]
limits:
max_pages: 20000
max_page_param: 200
robots: enforce
The crawler reads this and refuses anything that doesn’t match. Seeds come from the sitemap. Each allowed pattern names what it may lead to. Product pages lead nowhere. Parameters that create duplicates are dropped before the URL is even considered. Limits stop runaway growth. The robots check is on, and it isn’t the kind of setting someone should be able to switch off casually.
Writing scope this way has a second benefit. It can be tested offline. Feed a saved listing page into the link extractor and check which URLs it would follow. If the extractor picks up anything outside the declared patterns, the test fails before the crawler ever runs against the real site. The piece on recorded fixtures covers building that kind of test set.
It also makes review easy. When a colleague changes the scope, the diff shows exactly which new URLs the crawler may now fetch. A one-line change to a follow rule is far easier to question than a change buried in link-extraction code.
Infinite spaces and how to cap them
Accidental traps deserve their own treatment, because they catch careful crawlers too, and they’re far more common than deliberate honeypots.
Faceted navigation is the classic case. A category page with filters for size, colour, brand and price can generate a separate URL for every combination and every ordering of those filters. The number of possible URLs grows combinatorially, while the number of distinct products stays the same. A crawler that follows filter links will fetch the same products thousands of times.
Calendars and archives that link forward and backward in time without limit are another. So are search result pages that link to further searches, and session or tracking identifiers added to every link, which make each URL look new.
The defences are straightforward. Normalise URLs before deciding whether to fetch them: sort the query parameters, strip anything known to be a tracking or session parameter, and lowercase the parts the site treats as case-insensitive. Keep a set of URLs already fetched in this crawl, keyed on the normalised form. Ignore filter and sort parameters unless the job specifically needs them, and if it does, allow only the ones it needs, one at a time. Cap pagination depth to what the job needs. Cap total pages per crawl per section, so a runaway pattern stops itself.
Watch for signs of an infinite space in the crawl’s own statistics. A rising count of distinct URLs with a flat count of distinct records extracted is the clearest one. So is a single URL pattern accounting for most of the crawl’s requests.
These caps also protect the site. An unscoped crawl through faceted navigation can put a lot of uncached load on a site’s database, which is exactly the kind of cost that leads to blocks.
One shop, crawled two ways
A made-up example shows how the two approaches differ in practice.
Say the job is to collect product names and prices from a mid-sized shop. The shop has a sitemap listing its category and product pages, a robots file that disallows its cart, account pages, internal search and one oddly named path, and a footer containing, among other things, a link hidden from view that points to that oddly named path.
The first crawler starts at the homepage and follows every same-domain link it finds. On the first page it queues the category links, the footer links, the hidden link, the cart link, the account link, every filter combination on every category, and a link to the store locator that leads to hundreds of location pages. It checks robots only for the homepage, if at all. Within minutes it has requested the hidden path, the cart and the search results, and it’s deep into filter combinations that repeat the same products. The shop’s logs show a client that read the markup and followed everything, including the one link nobody could see.
The second crawler reads robots first, fetches the sitemap, and queues the category and product URLs it lists. From category pages, it follows only product links and the next page of the listing. It drops sort and filter parameters. It never sees the hidden link as a candidate, because the footer isn’t a page type in its scope, and the hidden path wouldn’t pass robots even if it were. It fetches each product once, at a polite pace, and stops.
Both crawlers collect the same products. The first one fetched several times as many URLs, put uncached load on the shop’s search and filter pages, and identified itself to the shop’s trap. The second one looks, from the shop’s side, like the kind of crawler the sitemap was published for.
What tripping a trap looks like from the inside
A crawler that trips a honeypot rarely finds out from the trap itself. The trap URL often returns something ordinary. The consequences show up elsewhere, and they can be confusing if nobody’s looking for them.
The most common pattern is a sudden change in how the site treats the crawler shortly after it fetched something unusual. Requests that were succeeding start getting challenges, block pages or soft blocks. If the site acted on an address, only that exit is affected at first. If it acted on a session, requests carrying that session’s cookies fail. If it shared the information with a bot management vendor, other sites using that vendor may start treating the same exit or client more strictly as well.
In a rotating pool, it can look like a spreading failure. The crawler trips the trap from one exit, that exit gets blocked, the rotation moves on, the crawler keeps following all links and trips it again from another exit, and so on. Within an hour, a large share of the pool is getting challenged on that site, and the logs show a gradual collapse rather than one clear event.
Decoy mazes look different. Success rates stay high, but the pages have no useful data, the count of distinct URLs climbs without limit, and the extracted records are empty, repetitive or nonsensical. Without content validation, it can run for a long time before anyone notices.
The way to catch all of this is in monitoring the crawl’s shape. Log every URL fetched with the rule that justified it. Alert on URLs fetched without a matching rule. Track the distribution of URL patterns and alert on new ones. Validate content on every page, and compare records extracted against pages fetched.
Why a proxy pool makes this worse, not better
It’s tempting to think that a large proxy pool protects a crawler from traps: if one address gets flagged, rotate to another. That misreads what the trap measured.
A honeypot identifies behaviour. The crawler requested a URL that only an indiscriminate crawler would request. Rotating the address doesn’t change the behaviour. The next exit follows the same links, reaches the same trap and gets flagged the same way. The pool spreads the problem, so that instead of one flagged address there are dozens, all carrying a mark on that site and possibly with that site’s bot management vendor.
That has costs beyond the one job. In a shared pool, those flagged addresses go on to serve other customers, who inherit a reputation they didn’t earn. The piece on what a shared proxy pool really shares describes this. On a dedicated pool, you’ve degraded your own capacity against that site and perhaps against others behind the same vendor.
There’s also the question of what continuing means. Once a site has signalled that it doesn’t want a particular behaviour, and the crawler has triggered that signal, continuing the same behaviour from new addresses is exactly what the site was trying to prevent. Rotating around a block is evasion, and this site won’t help anyone do it.
So the correct relationship between proxies and traps is that proxies are irrelevant to them. Scope is what prevents traps. A well-scoped crawler through a single exit won’t trip a honeypot. A badly scoped crawler through a million exits will trip it a million times.
There’s one more reason to care, beyond the single site. Bot management is a shared service for many of the sites that use it. When a vendor sees a client trip a trap on one customer’s site, it may raise that client’s risk score across every other customer it protects. Nobody outside the vendor knows exactly how those signals are combined or for how long they last, but the practical effect is that a scope mistake on one target can make unrelated jobs harder. That’s another reason to treat scope as a property of the whole operation, reviewed and tested, instead of a detail left to each crawler.
If a crawler has already tripped one
Sometimes the first sign of a scope problem is a block that turns out to trace back to a trap. The recovery steps are straightforward, if a little humbling.
Stop the crawl for that site. Don’t resume from fresh addresses. Whatever caused the problem is still in the code.
Find the URL. Search the crawl logs for requests to paths disallowed by robots, requests to URLs that didn’t match any follow rule, and requests to URLs that appear only once and carry odd tokens. The trap is usually easy to spot once you look, because nothing else on the site looks like it.
Work out how the crawler reached it. Was it following every link? Did a follow rule match too broadly? Did the robots check get skipped for redirect targets, or use a stale copy? Was the robots file fetched at all?
Fix the scope, not the symptom. Adding the trap URL to a blocklist fixes nothing, because the next trap will have a different URL. Tighten the follow rules, enforce robots on every request, and add the monitoring described above.
Then decide whether to resume at all. If the site has blocked the crawler, it has made its preference clear. For a job that matters, the next step may be to contact the site, explain what went wrong and what changed, and ask whether a slower, scoped crawl or an API would be acceptable. The piece on identifying a crawler honestly with a contact header covers why sites often respond better to that than to anonymous traffic.
What I can’t tell you
I can’t tell you which sites use honeypots, how they build them, or what they do when one fires. Sites don’t publish that, it changes, and the specifics are exactly the kind of detail that stops being useful the moment it’s written down.
I can’t tell you how to detect hidden links reliably, and I’m not going to try, because it’s the wrong problem. A crawler that relies on spotting traps is still following every other link it finds, and that’s the behaviour the site objected to in the first place.
I can’t tell you that a well-scoped crawler will never be blocked. Traps are one mechanism among many. Rate, client signals and the site’s own rules still apply.
What I can say is that the defence against every trap described here is the same short list. Start from the sitemap. Check robots before every request, with a real parser and a fresh copy. Follow only link patterns the job needs, and be able to explain every URL fetched. Normalise and cap to avoid infinite spaces. Monitor the crawl’s shape as well as its success rate. And when a site signals that it doesn’t want what the crawler is doing, stop and change the crawler rather than the address.
If you want the related pieces on sitemap planning, soft blocks and why scrapers get blocked, Proxy Scraping has the full set of guides from the operator side.
Get new guides and videos first — join the Telegram channel.