Price monitoring across regional storefronts
The job, stated plainly
A retailer sells the same range in several countries, each through its own storefront. You want to know how its public prices for the same products compare across those storefronts, and how they move over time. Maybe you’re the retailer’s competitor, a brand checking how resellers price its products, or an analyst studying regional pricing. The question is simple to say: is this item cheaper in one country than another, and by how much?
It’s a good example of a scraping job because every hard part of scraping shows up in it, just in a contained form. You need to reach each storefront as its local customers do, which is a geo-targeting problem. You need to know which product on one storefront is the same as a product on another, which is a matching problem. You need to turn a string like “1.299,00 €” into a number you can compare with “$1,149.99”, which is a parsing and normalisation problem. And you need to decide how often to look, which is a recrawl problem with a budget attached.
The pieces on this site about geo-targeting with proxies, picking a target country, and deciding how often to recrawl each cover one of those ideas in general. This piece doesn’t re-explain them. It puts them together for one concrete job, with the decisions in the order you’d actually make them.
Throughout, the scope is public prices: what any visitor to a storefront sees on a product page without logging in. Member prices, prices behind an account, prices that only appear in a cart, and anything involving personal data are out of scope, both because they bring in terms and privacy problems and because they rarely answer the question the job is asking.
Check whether a storefront needs an in-country exit at all
The instinct is to buy proxies in every country the retailer sells in, and send each storefront’s traffic from its own country. Sometimes that’s necessary. Often it isn’t, and finding out first saves money and complexity.
Regional storefronts are built in a few common ways. Some live on separate country domains. Some live under a path on one domain, like a country and language prefix. Some use one address for everything and decide the region from the visitor. And the way a storefront decides which prices to show can be independent of how it’s addressed.
So for each storefront, run a small test from a single ordinary connection. Load a product page on the country domain or path. Does it show local currency and local prices, or does it redirect you to your own country’s storefront? Is there a country selector, and does choosing a country set a cookie that sticks? Does the page say anything about delivery country, and does changing it change the price?
Many storefronts on separate domains or paths serve their own region’s prices to anyone, because they expect travellers, expatriates and cross-border shoppers. For those, the exit country barely matters, and you can reach them all from one location at a polite pace. Others redirect by IP or refuse to show local prices to foreign visitors. Those genuinely need an exit in the right country.
The piece on geo-targeting covers which signals usually win: the language header, sticky cookies, the IP lookup, account settings. For price monitoring, the output of this step is a short table, one row per storefront, saying whether it needs an in-country exit, a specific cookie or selector setting, or neither.
Matching exits to the storefronts that need them
For the storefronts that do key on location, the next decision is which exit to use and how precisely it needs to be placed.
Country is usually enough. Retail prices are almost always set per country or per region. City-level targeting is rarely needed for this job, and city data from IP lookups is unreliable anyway. If a storefront covers several countries, such as one storefront for a group of countries sharing a currency, the price may still differ by country because of tax, so check that too.
Pick the exit type on the evidence, not the reputation of retail as a hard target. The piece on targets where datacenter proxies are still enough applies here: a retailer’s storefront behind basic caching may serve a datacenter exit in the right country perfectly well. A retailer running strict bot management will treat traffic differently, and then the question becomes whether you should be scraping it at that volume at all, or whether there’s a sanctioned route such as an affiliate product feed or a partner API.
Verify each exit on the page itself, not with an IP checker. The geo-targeting piece makes this point in general. For price monitoring it’s concrete: an exit is in the right place when the storefront shows the right currency, the right tax labelling and the right delivery country. A checker site saying the address is in the right country tells you what one database thinks, not what this retailer thinks.
Keep each storefront’s session consistent. If a country selector sets a cookie, set it once per session and keep it. If the storefront reads the language header, send the language its local customers would send. Switching exits mid-session can leave you with a cookie from one country and an address from another, and the storefront’s response to that mix is unpredictable. Record the exit country with every observation, so a strange price can be traced to the conditions it was collected under.
Matching products across storefronts
This is the hard part, and it’s the part people underestimate. Collecting prices is easy. Knowing that two prices belong to the same product is where most of the work goes.
Start with identifiers. Many product pages carry structured data, usually schema.org markup in a JSON-LD block, that includes a SKU, a manufacturer part number, or a global trade item number such as an EAN or UPC. Global identifiers are the best match key, because they’re assigned to the physical product rather than by the retailer. When the same GTIN appears on two storefronts, you’re very likely looking at the same item.
The retailer’s own SKU is the next best thing, but retailers sometimes use different SKUs per region for the same product, or the same SKU for regionally different products. Treat it as strong evidence, not proof.
Regional variants are the trap. A kitchen appliance sold in the UK and Germany may be the same model with a different plug, and priced differently partly for that reason. An electronics product may ship with a different bundle or a different warranty. Clothing can be the same design sized differently. Decide up front what “the same product” means for your analysis, and whether a regional variant counts. For a pricing comparison, a variant usually counts as the same product with a note, but the note matters when someone reads the result.
When identifiers are missing, fall back on model numbers in titles or specifications, and treat those matches as lower confidence. Title similarity alone is unreliable across languages. Store a match confidence with each pairing, so the comparison can be filtered to high-confidence pairs.
Build the match table once, carefully, then maintain it. Products are added and discontinued, and a match that was right last quarter can break when a storefront reuses an identifier.
What “the price” actually means on a page
A product page rarely shows one price. It shows several, and picking the wrong one makes the whole comparison wrong.
There’s the current selling price, which is what a customer would pay today. There may be a crossed-out “was” price or a recommended retail price next to it. There may be a member or loyalty price that applies only to logged-in customers, shown to everyone as an incentive. There may be a unit price, such as per kilogram or per litre, which some countries require retailers to display. There may be a “from” price for a product with variants, which is the cheapest variant and not the one you care about. There may be a bundle or multi-buy price.
Decide which one the job tracks, and define it precisely. For most regional comparisons, it’s the current selling price of a specific variant for an ordinary visitor. The structured data on the page often carries exactly that, as an offer with a price and currency, which is more reliable than parsing visible text. Check it against the visible price on a few pages, because structured data can lag behind a sale, or reflect a different variant from the one on screen.
Variants need their own handling. If a product comes in sizes or colours with different prices, the variant is part of the product identity. The default variant on page load may differ between storefronts, so don’t assume the price you see first is comparable.
Record the context alongside the number: whether a sale was active, whether the price shown was a “from” price, whether a member price was displayed. A comparison that treats a German sale price as equivalent to a British regular price will produce a conclusion that isn’t true.
And record availability. A price for an item that’s out of stock may be stale or placeholder, and some storefronts keep showing the last price for discontinued items.
Parsing prices written the local way
Prices are written differently in almost every country, and a parser written for one storefront will quietly misread another.
The decimal separator is a comma in much of continental Europe and a full stop in the UK, the US and elsewhere. The thousands separator is the opposite, or a space, or a thin space, or a non-breaking space, or an apostrophe in Switzerland. So “1.299,00” and “1,299.00” are the same amount, and “1.299” might be one thousand two hundred and ninety-nine or one point two nine nine, depending on the storefront. Some currencies don’t use minor units at all in everyday display. The currency symbol can come before or after the number, with or without a space.
Don’t guess. For each storefront, define the format explicitly: decimal separator, grouping separator, symbol position, number of decimal places. Parse with those rules. Better still, prefer the machine-readable price in structured data, which is supposed to use a plain decimal format, and fall back to visible text only when that’s missing.
Test the parser against saved examples from every storefront, including edge cases: a price over a thousand, a price with no minor units, a price range, a sale price shown next to the original. The piece on recorded fixtures on this site explains how to build that test set once and reuse it. A price parser is the textbook case for fixtures, because the formats are stable and the bugs are subtle.
Add a sanity check on every observation. If a product’s price moves by a factor of ten or a hundred between two observations, it’s almost always a parsing error, a separator misread, or a currency mix-up, not a real price change. Flag it for review instead of writing it straight into the dataset.
Converting currencies without fooling yourself
Once prices are numbers, they’re still in different currencies. Comparing them means converting, and conversion has more choices in it than it first appears.
Always store the original amount and currency. Never store only a converted figure. Conversion rates change daily, and you’ll want to redo the conversion with a different rate or a different method later. The original observation is the fact; the converted figure is an analysis.
Choose a rate source and stick to it. Central banks publish daily reference rates, and the European Central Bank’s euro reference rates are a common choice for European comparisons. Commercial data providers publish more frequent rates. For a pricing study, a daily reference rate is usually fine, since retail prices don’t move minute to minute. Record which rate and which date was used for each conversion.
Convert at the observation date, not today. A price observed three months ago should be converted at the rate from that day, or you’re mixing exchange-rate movement into what looks like a pricing difference.
Remember what retailers actually do. Many set regional prices as round local numbers and change them infrequently. When the exchange rate moves, the converted gap between storefronts moves too, even though neither retailer changed anything. That’s a real effect worth reporting, but it’s an exchange-rate story, not a pricing decision. Showing both, the converted gap and whether either local price changed, keeps the two apart.
And be careful with rounding. Convert with full precision and round only for display.
Taxes, and why the sticker prices aren’t comparable
Tax is the single biggest reason naive regional price comparisons are wrong.
In the UK, the EU, Australia and many other places, retail prices shown to consumers include value added tax or goods and services tax. In the US and Canada, prices are usually shown before sales tax, which is added at checkout and varies by state, province or even city. So a US price and a German price, converted to the same currency, aren’t comparable until you deal with tax.
The usual approach is to compare prices excluding tax. For storefronts that show tax-inclusive prices, divide by one plus the applicable rate. For storefronts that show tax-exclusive prices, use them as they are. Record whether each observation was tax-inclusive, and what rate was removed.
The applicable rate is not always the standard rate. Many countries apply reduced rates to categories such as food, books, children’s clothing or medical goods. A comparison of grocery prices that strips the standard rate from a product taxed at a reduced rate will be wrong by several percent. Map each product category to the rate that applies in each country, and keep that mapping in one place, since rates change.
Storefronts sometimes say which rate they applied, in small print near the price or in structured data. Use that when it’s available, and check it against your mapping.
Shipping, duties and fees are a separate layer. Most regional price comparisons exclude them, because they depend on the delivery address and basket, and collecting them usually means going into a cart. Say explicitly that your comparison excludes them. If a storefront includes delivery in its displayed price, note that too.
Three storefronts, one product, worked by hand
A made-up example shows how much the normalisation steps change the answer. The prices and rates here are invented for illustration.
Say the same matched product, same variant, is listed at 119,00 € on a German storefront, £99.00 on a British one, and $109.99 on an American one. At a glance, the American price looks lowest and the German one highest, once you picture the currencies roughly.
Now apply the steps. The German storefront shows prices including VAT, and suppose the product is taxed at the standard German rate of 19 percent. Dividing 119.00 by 1.19 gives 100.00 euros excluding tax. The British storefront also includes VAT, at a standard rate of 20 percent, so 99.00 divided by 1.20 gives 82.50 pounds. The American storefront shows prices before sales tax, so 109.99 dollars is already the tax-exclusive figure.
Then convert all three to one currency at the observation date’s reference rate. Whatever those rates happen to be on that day, the comparison is now between 100.00 euros, 82.50 pounds and 109.99 dollars, all excluding tax. Depending on the rates, the American price can easily turn out to be the highest of the three, not the lowest. The ranking the sticker prices suggested was mostly a tax effect.
Now suppose the German storefront had a sale running and the British one didn’t. The comparison needs a flag saying so, or a reader will conclude the German storefront is structurally cheaper when it was simply on promotion that week.
And suppose the product were a book, taxed at a reduced rate in Germany and at zero in the UK. Stripping the standard rate would understate both tax-exclusive prices, by different amounts, and the gap would be wrong.
None of these steps is difficult. Skipping any one of them produces a clean-looking chart that says something untrue.
How often prices really need rechecking
It’s tempting to check prices every hour, because prices feel like fast-moving data. For most retailers and most products, they aren’t.
Regional prices at many retailers change on a schedule set by merchandising decisions: a weekly promotion cycle, a monthly review, seasonal sales, occasional repricing when costs or exchange rates move. Some categories and some retailers do use dynamic pricing that changes more often. You don’t know which you’re dealing with until you look.
So measure before you schedule. For the first few weeks, check a sample of products daily using the lightest method available, and record when each price actually changed. The piece on deciding how often to recrawl covers the signals a page gives you for free, such as last-modified headers, entity tags that support conditional requests, and sitemap lastmod dates. Where a storefront supports conditional requests, an unchanged page costs a tiny response.
Then tier the products. Items whose prices changed several times in the sample period get checked often. Items that didn’t change get checked less often. Items in categories affected by known sale events get a temporary boost around those dates.
Account for the storefronts separately. One region may run promotions weekly while another changes prices monthly, and the schedule should reflect each.
The result is a recheck schedule tied to how prices actually behave, which typically means far fewer requests than an hourly poll, with almost no loss of information. When a storefront starts changing prices more often, the change rate in your own data will show it, and the tiering can adapt.
Budgeting the job before it runs
Price monitoring across storefronts multiplies quickly. Products times storefronts times checks per week is the number of page fetches, and each fetch through a metered exit costs bandwidth.
Work it out before starting. Take the number of matched products you’ll track, the number of storefronts, and the average checks per week per product from the tiered schedule. Multiply by the average transfer size of a product page with images and other assets blocked. That gives weekly bandwidth per exit type. Compare it with the plan you’re considering, using its current pricing page rather than any figure quoted elsewhere.
Then look for ways to fetch less. Structured data on the page is usually all you need, so there’s no reason to render the page in a browser, which multiplies bandwidth. Some storefronts expose a lighter source for the same data: a product feed for comparison engines, a sitemap with lastmod dates, a public endpoint behind the site’s own search. Some retailers run affiliate programmes that provide full product and price feeds to partners, which is often the cleanest route of all, because it’s sanctioned and complete.
Storefronts that don’t need an in-country exit can go through the cheapest exit that serves them, or with no proxy at all at a polite pace. Only the storefronts that key on location need an exit in their country, and only their share of the traffic needs that exit’s price.
Cache what you fetch. If a product page was fetched for one purpose today, don’t fetch it again for another. The caching piece on this site covers keeping a rerun from refetching everything.
Politeness, terms and what to leave alone
A price monitor runs forever, on a schedule, against the same retailer. That makes politeness more important than in a one-off scrape, because any problem you cause repeats every day.
Read each storefront’s terms of service and robots rules. Terms can differ between regional storefronts of the same retailer, since they’re often run by different legal entities under different law. If the terms prohibit automated access or the robots file disallows the product pages, that’s a reason to look for a sanctioned source instead, and to take advice if the job matters commercially.
Set a rate limit per storefront, not per job. Each storefront may run on different infrastructure with different capacity, and a limit that’s fine for a large regional site may be heavy for a small one. Keep concurrency low, respect crawl delays, back off on 429s and 503s, and schedule heavy runs outside the storefront’s local peak hours where possible.
Stay out of carts, checkouts and accounts. Adding items to a cart to see a final price creates sessions and inventory holds on the retailer’s systems, and it’s a different kind of activity from reading a public page. If the comparison needs final landed prices, that’s a question for a different method, such as manual sampling or a partner arrangement.
Avoid personal data. Product pages sometimes include reviews with names or photos. A price monitor doesn’t need them, so don’t extract or store them.
Validating every observation
A price dataset fails quietly. Nobody notices a wrong number until someone makes a decision on it. So validation belongs in the pipeline, on every observation, before anything is written.
Check the currency. If a storefront should show euros and the observation came back in dollars, the geo-targeting didn’t take: the storefront probably redirected you to another region or showed a default. That’s one of the most useful signals in the whole job, because it catches exit and cookie problems immediately. Treat it as a failed fetch, not as a data point.
Check the tax labelling. A storefront that normally says prices include VAT and suddenly doesn’t may be showing a different region or a business customer view.
Check that the product matches. The page should carry the identifier you expected. If a URL now shows a different product, the storefront has reused or redirected it.
Check for soft blocks. A page that returns 200 but lacks the price block, the structured data or the product title may be a challenge page or a stripped template. The soft blocks piece on this site describes how those look. Count them per storefront and per exit, because a rising rate is a signal to slow down.
Check the size of changes. As mentioned, a tenfold jump is almost always an error. A moderate change during a known sale is expected. A moderate change outside one is worth a second look, not an automatic rejection.
Store the reason for every rejected observation. Over time, the rejection log tells you which storefronts and exits are healthy far better than success counts do.
Storing observations so the comparison stays honest
The storage design decides what questions you can answer later, so it’s worth getting right at the start.
Store observations, not conclusions. Each row should be one price seen on one storefront at one time: the matched product ID, the storefront, the observation timestamp, the original amount and currency as shown, whether it was tax-inclusive, the price type (regular, sale, from), the variant, availability, the source URL, the exit country and type used, and a hash or reference to the raw response. Conversion and tax removal happen in queries or derived tables, with the rate and date recorded.
Keep the raw response, or at least the extracted structured data, for a while. When a number looks wrong weeks later, you’ll want to see exactly what the page said.
Keep the match table and tax-rate table as separate, versioned reference data. When you fix a product match or a tax rate, you can recompute every comparison that used it.
When presenting results, compare like with like: same matched product, same variant, prices excluding tax, converted at the observation date’s rate, with sale status visible. Show the local prices alongside the converted ones, so readers can see whether a gap came from pricing or from exchange rates. And state what’s excluded: shipping, duties, member prices, anything behind a cart.
What I can’t tell you
I can’t tell you how any specific retailer decides regional prices, or how often it changes them. That’s exactly what the job is meant to find out, and the answer differs between retailers, categories and even storefronts of the same company.
I can’t tell you which storefronts will need an in-country exit. The test in the second section answers that for each one, and the answer can change when the retailer changes its platform.
I can’t tell you that a price monitor won’t be blocked. A polite schedule, a rate limit per storefront, and a sanctioned data source where one exists make it a job retailers have little reason to object to, but the retailer’s rules and technology decide.
What I can tell you is the order that keeps the job honest and cheap. Test each storefront before buying exits. Match products on real identifiers. Define which price you’re tracking. Parse each storefront’s format explicitly. Store originals and convert later. Strip tax with the right rate. Measure how often prices change before you schedule. Validate every observation.
If you want the underlying pieces on geo-targeting, choosing a target country and recrawl scheduling, Proxy Scraping has the full set of guides from the operator side.
Get new guides and videos first — join the Telegram channel.