Articles about AI search keep repeating that "companies need to be ready", yet nobody says how the Czech market actually stands. So I measured it: on 22 August 2026 I checked thirty Czech e-shops for how their robots.txt treats AI crawlers, whether they publish an llms.txt, and what structured data their homepage offers. The methodology and aggregated data are below.
Quick answer: what the measurement showed
- Only 5 of 25 e-shops with a readable
robots.txtmention any AI crawler in it. The other 20 files have no explicit mention of tracked agents; general rules may still apply. - Only 2 of 25 fully block at least one AI crawler. These are observed agent-specific rules, not a complete access test for every crawler.
llms.txtreturned status 200 on 8 of 30 domains, but only 4 are a genuine text file. The rest serve an HTML application page, so the file effectively doesn't exist.- 7 of 23 readable homepages carry no JSON-LD at all. The
Producttype appeared on 3,FAQPageon a single e-shop. - The basics aren't a given: exactly one
H1on 61%, canonical on 48%, hreflang on 48%. - 6 of 30 domains refused a plain HTTP client with a 403. This request did not verify access by actual AI crawlers.
Methodology and public data
- Selection: 30 well-known Czech e-shop domains across sectors and sizes. This is a purposive, not random or representative sample, the numbers describe this sample, not the whole market.
- When and how: 22 August 2026, one HTTP request per file and domain, a standard browser User-Agent, a pause between domains.
- What was measured:
robots.txt(rules for 11 AI agents),llms.txt(status and actual content type), homepage HTML (JSON-LD types, H1 count, meta description, canonical, Open Graph, hreflang). - Limits: static HTML fetch without executing JavaScript, so JSON-LD injected by scripts may not be counted. Domains returning 403 are excluded from the relevant aggregates. This is a one-off snapshot, not long-term monitoring.
- Agents tracked: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, meta-externalagent, Amazonbot.
Download the anonymised aggregated dataset (CSV). It contains 30 metrics with numerator, denominator and a method note; it contains no shop names and no mapping of values to specific domains.
Worked example: Three different denominators
OWN MEASUREMENT · PURPOSIVE SAMPLE
| Check | Result |
|---|---|
| Selected domains | 30 |
| Readable robots.txt | 25 / 30 |
| Retrieved homepages | 23 / 30 |
| Explicit AI-agent mention | 5 / 25 robots.txt |
No explicit mention does not mean blocked. General rules also matter.
Finding 1: 5 of 25 files explicitly mention AI agents
The aggregated dataset reports explicit ClaudeBot mentions on 4 of 25 domains and GPTBot mentions on 3 of 25. It reports 2 complete blocks for CCBot. Mentions and complete blocks are different metrics; they do not form a shared ranking.
If there is no matching agent-specific group, RFC 9309 specifies the applicable User-agent: * group as the fallback where one exists. Agent names alone do not establish owner intent or overall site access. Explicitly allowing a bot also does not guarantee inclusion in an AI answer.
Finding 2: half the llms.txt files found aren't files
The llms.txt file (a text summary of a site aimed at language models) returned status 200 on eight domains. But when I checked the content type, only four actually returned text. The other four sent an HTML application page, the server said "fine", but the body was a website, not a file.
I know this bug intimately: I had it on my own site and found it the same day. It happens when the framework's routing layer intercepts the request for a static file and returns a page instead. Checking the HTTP status won't reveal it, you have to check the Content-Type.
| Check | Domains |
|---|---|
llms.txt returns 200 |
8 of 30 |
| of those, a genuine text file | 4 of 30 |
| of those, HTML instead of a file | 4 of 30 |
llms.txt is a voluntary proposal, not an AI citation requirement. Google states that its AI features do not require new AI text files. This test checked response status and content type; it did not measure whether any AI service used the file.
Finding 3: some initial homepage HTML lacks JSON-LD
Of the 23 readable homepages:
| JSON-LD type | E-shops |
|---|---|
| Organization | 11 |
| WebSite | 10 |
| Product | 3 |
| BreadcrumbList | 1 |
| FAQPage | 1 |
| No JSON-LD at all | 7 |
No JSON-LD was found in the initial HTML of seven homepages. This does not establish the absence of other structured-data formats or JavaScript-injected data, and says nothing about product pages. JSON-LD can describe visible content explicitly, but is not a universal AI citation requirement. A Product type on the homepage is not a general requirement for every shop either.
Worked example: One product, one price
FICTIONAL PRODUCT · EXAMPLE PRICE
Mismatch
- Page: lamp for CZK 2,490
- JSON-LD: price = 1990
- Currency: CZK
Corrected
- Page: lamp for CZK 2,490
- JSON-LD: price = 2490
- Currency: CZK
Valid JSON does not guarantee accurate data.
Finding 4: the basics aren't a given, even for large stores
| Element | Share (of 23) |
|---|---|
| Meta description | 91% |
| Open Graph title | 78% |
| Exactly one H1 | 61% |
| Canonical | 48% |
| hreflang | 48% |
No canonical was found in the initial HTML of about half the sample. This alone does not prove duplicate indexing; Google can select a canonical URL without that annotation. Product URLs, HTTP headers and Search Console state were not assessed here. Missing hreflang is not an error where there are no relevant language or regional variants.
Finding 5: an ordinary HTTP client received 403 responses
Six of thirty domains refused an ordinary HTTP request with a 403 response. The test did not use a verified AI crawler's network identity or examine its real visits in server logs. A CDN may distinguish IP addresses, verified bots and other request properties. This client's result cannot therefore be applied to actual AI crawlers.
To verify a particular search crawler, inspect robots.txt, CDN/hosting rules and verified visits. OpenAI separately specifies OAI-Searchbot permission and access for its published IP addresses. There is no need to disable site protection indiscriminately.
What this means for an e-shop owner
- Check specific and wildcard robots.txt groups. Distinguish training crawlers, search crawlers and user-triggered requests.
- Verify status, headers and content. A 200 response with HTML does not prove the intended text overview exists.
- Match structured data to the actual page. It must reflect visible, accurate information.
- Check indexing on real URLs. H1 counts or an absent homepage annotation are not a complete SEO audit.
- Verify crawler access separately. One ordinary HTTP request cannot establish the behaviour of every AI service.
If you want this checked and configured on your own site, that's exactly what I handle in SEO care and audits and, for stores, within a custom e-shop.
Frequently asked questions
How many Czech e-shops block AI crawlers?
In a sample of 30 domains (25 with a readable robots.txt), two e-shops fully block at least one AI crawler. Five have any rule for AI agents. The other twenty do not explicitly mention them. That does not establish allowed access: wildcard groups and hosting rules may apply.
Is llms.txt worth having?
It is an optional experiment, not a proven ticket into AI results. Google does not require such a new file for its AI features. If you offer one, verify its actual content; in this measurement, half the 200 responses at that path returned HTML instead of a text overview.
How do I tell whether my llms.txt or robots.txt returns HTML?
Check status, headers and body of the same GET request, for example curl -sS -D - https://yourdomain.cz/llms.txt. Expect a text overview, not an HTML page. Content-Type or status 200 alone does not establish correct content.
Which JSON-LD types should an e-shop have?
Choose types to fit the content: Organization for company identity, WebSite for the site and a suitable Product type on a specific product page; BreadcrumbList for the visible navigation path. This is not a mandatory bundle for every homepage or a guarantee of a rich result or AI citation. The table above only describes types found in initial HTML.
Is a sample of 30 e-shops representative?
It is not. This purposive selection describes only the sampled domains on the stated date. The aggregate CSV supports checking reported totals and denominators; without original domains and responses it does not permit independent reproduction of the identical sample. The same procedure can be applied to a new sample.
Sources and links
-
Anonymised aggregated dataset (CSV), 30 metrics from this measurement
-
Previous measurement of 21 Czech e-shops, content, speed and site files, August 2026
-
Google Search Central: robots.txt, how robot rules work
-
Google crawlers overview, including Google-Extended
Interpretation and technical sources checked on 2 October 2026. No new domain measurement was run; original counts and measurement date are preserved.