I measured 30 Czech e-shops: 80% have no stance on AI crawlers

Articles about AI search keep repeating that "companies need to be ready", yet nobody says how the Czech market actually stands. So I measured it: on 22 August 2026 I checked thirty Czech e-shops for how their robots.txt treats AI crawlers, whether they publish an llms.txt, and what structured data their homepage offers. The methodology and aggregated data are below.

Quick answer: what the measurement showed

• Only 5 of 25 e-shops with a readable robots.txt mention any AI crawler in it. The remaining 80% have made no decision about AI at all — neither yes nor no.
• Only 2 of 25 fully block at least one AI crawler. Fear of AI barely shows up in actual configuration.
• llms.txt returned status 200 on 8 of 30 domains, but only 4 are a genuine text file. The rest serve an HTML application page — so the file effectively doesn't exist.
• 7 of 23 readable homepages carry no JSON-LD at all. The Product type appeared on 3, FAQPage on a single e-shop.
• The basics aren't a given: exactly one H1 on 61%, canonical on 48%, hreflang on 48%.
• 6 of 30 domains refused a plain HTTP client with a 403. Bot protection doesn't treat AI crawlers any differently than it treated me.

Methodology and public data

• Selection: 30 well-known Czech e-shop domains across sectors and sizes. This is a purposive, not random or representative sample — the numbers describe this sample, not the whole market.
• When and how: 22 August 2026, one HTTP request per file and domain, a standard browser User-Agent, a pause between domains.
• What was measured: robots.txt (rules for 11 AI agents), llms.txt (status and actual content type), homepage HTML (JSON-LD types, H1 count, meta description, canonical, Open Graph, hreflang).
• Limits: static HTML fetch without executing JavaScript, so JSON-LD injected by scripts may not be counted. Domains returning 403 are excluded from the relevant aggregates. This is a one-off snapshot, not long-term monitoring.
• Agents tracked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, Amazonbot.

Download the anonymised aggregated dataset (CSV). It contains 30 metrics with numerator, denominator and a method note; it contains no shop names and no mapping of values to specific domains.

Finding 1: almost nobody has made a decision about AI yet

The most frequently mentioned agent was GPTBot (3 of 25 domains have some rule for it), followed by ClaudeBot (4 mentions including partial restrictions) and CCBot (2 blocks). For the other agents, a mention is the exception.

More interesting than blocking is the opposite: some e-shops explicitly allow AI crawlers. That makes sense for a store that wants to appear in AI answers — only very few companies are deciding this consciously so far.

Finding 2: half the llms.txt files found aren't files

The llms.txt file (a text summary of a site aimed at language models) returned status 200 on eight domains. But when I checked the content type, only four actually returned text. The other four sent an HTML application page — the server said "fine", but the body was a website, not a file.

I know this bug intimately: I had it on my own site and found it the same day. It happens when the framework's routing layer intercepts the request for a static file and returns a page instead. Checking the HTTP status won't reveal it — you have to check the Content-Type.

Some honest context: public analyses have found no relationship between having an llms.txt and being cited in AI results, and Google has declined to support it. So the file is not a ticket anywhere. The problem here isn't that someone lacks it — it's that someone has it and it doesn't work.

Finding 3: structured data is missing where AI would need it most

Of the 23 readable homepages:

Seven e-shops have no structured data on the homepage. That's no disaster for classic search — Google copes without it. It becomes a problem once a language model assembles the answer: structured data is the most reliable way to tell it price, availability, ratings and company identity without it inferring them from prose.

Finding 4: the basics aren't a given, even for large stores

A missing canonical on half the sample is the most surprising number in the whole measurement. For an e-shop where the same product appears under filters, sorting and pagination, that's a direct route to duplicates in the index.

Finding 5: bot protection applies to AI too

Six of thirty domains refused a plain HTTP request with a 403. Operationally that's understandable — anti-scraping protection exists for a reason. It's worth knowing what it means, though: the same protection applies to AI crawlers that would otherwise use the content as the basis for an answer to a customer.

This isn't an argument for turning protection off. It's an argument for a conscious decision: either I want to be visible in AI answers and configure exceptions, or I don't and accept that as the price of security. The worst option is not knowing which one applies.

What this means for an e-shop owner

1. Open your robots.txt and decide. Not for the technicality, but because the default is a decision too — one somebody else made for you.
2. Check the Content-Type, not just the status. A file that returns HTML is invisible to machines, even if it "works" in a browser.
3. Add JSON-LD for product and organisation. Price, availability and identity are exactly what a model otherwise infers.
4. Fix canonical and H1 before anything AI-related. Without the basics nothing else takes hold.
5. Decide about bot protection deliberately. Protection and visibility are both legitimate choices; ignorance isn't.

If you want this checked and configured on your own site, that's exactly what I handle in SEO care and audits and, for stores, within a custom e-shop.

Frequently asked questions

How many Czech e-shops block AI crawlers?

In a sample of 30 domains (25 with a readable robots.txt), two e-shops fully block at least one AI crawler. Five have any rule for AI agents. The remaining twenty don't mention them at all, which in practice means access is allowed.

Is llms.txt worth having?

For now it's tidiness rather than return. Public analyses have found no relationship between having it and being cited in AI results, and Google declined to support it. If you do have it, though, it must be a genuine text file — in this measurement half the llms.txt files found were actually HTML pages.

How do I tell whether my llms.txt or robots.txt returns HTML?

Open the file and look at the server's Content-Type — it should be text/plain. A quick command-line check: curl -sI https://yourdomain.cz/llms.txt. Status 200 proves nothing on its own; an application page returns it too.

Which JSON-LD types should an e-shop have?

At minimum Organization (company identity), WebSite (the site as a whole), Product with price and availability on product pages, and BreadcrumbList for the navigation path. FAQPage where you genuinely answer questions. In the measured sample, only 3 of 23 e-shops had Product on the homepage.

Is a sample of 30 e-shops representative?

It isn't, and I don't claim it is. It's a purposive selection of well-known Czech e-shops across sectors. The numbers describe this sample — a good indication of the state of the market, but not a statistical estimate for all Czech e-shops. That's why the methodology and the aggregated dataset are published — the numbers can be verified and the measurement repeated with the same procedure on one's own sample (the dataset deliberately contains no domain names).

Sources and links

• Anonymised aggregated dataset (CSV) — 30 metrics from this measurement
• Previous measurement of 21 Czech e-shops — content, speed and site files, August 2026
• Google Search Central: robots.txt — how robot rules work
• Google crawlers overview — including Google-Extended