Updated
AI crawlers in robots.txt on Polish websites. We checked 1000 domains
Of 955 Polish domains that responded, 17 percent block at least one AI crawler, and under 3 percent block the bots that look up answers for users. Method and numbers.
In this article
In short
We checked 1000 random Polish domains from the Tranco list. 955 responded, and all percentages are based on them. 17 percent block at least one AI crawler in robots.txt, almost always the ones that collect data for model training. Under 3 percent block the bots that look up answers for a user during a chat. A sitemap can be found on 71 percent of domains, and HTTPS works on almost 95 percent.
When someone asks ChatGPT or Claude about a company, the tool often visits websites and reads them during the chat. Whether it may do so depends partly on the robots.txt file. We wanted to see what Polish websites tell AI crawlers in that file, so we studied 1000 random Polish domains.
How we checked robots.txt on Polish domains
The domains come from the Tranco list. It is a public list of the most popular domains in the world, built by researchers for studies and hardened against manipulation (tranco-list.eu). We used the list with ID 8P8NV, downloaded on 9 November 2026. Of its million domains, 9401 end in .pl. We drew 1000 of them with a fixed random seed, so the same sample can be repeated.
This is a list of popular domains, not a list of company websites. It contains shops, media, portals, public bodies, universities and small business sites. We did not split them by type, because we have no reliable method for that.
We checked each domain with one or two requests, one at a time, with a one second pause between domains.
- We fetched robots.txt over HTTPS. If HTTPS did not respond, we tried HTTP once.
- We read the rule for each AI crawler from that file using the same code our tool can AI see your website uses. What counts is whether the crawler may open the home page.
- We counted a sitemap as present when robots.txt gives its address. If it did not, we sent one request for sitemap.xml, and only when robots.txt allowed it.
That was at most three requests per domain and one and a half on average. The study took 28 minutes. We publish only totals, with no domain names.
How many domains responded
Of 1000 domains, 955 responded. The other 45 did not. 19 of them failed our DNS address check, almost always because the bare domain without www has no address. 26 did not answer within 10 seconds. All percentages below are based on the 955 domains that responded.
Which AI crawlers Polish domains block
800 domains, or 84 percent, have a robots.txt file. One in five, 199 domains to be exact, names at least one AI crawler from our list.
162 domains, or 17 percent, block at least one AI crawler. That includes 15 domains that close the home page to all crawlers at once.
| Crawler | Used for | Domains blocking | Percent |
|---|---|---|---|
| ClaudeBot | model training | 136 | 14.2 |
| GPTBot | model training | 133 | 13.9 |
| CCBot | Common Crawl dataset | 115 | 12.0 |
| Google-Extended | model training | 87 | 9.1 |
| Applebot-Extended | model training | 83 | 8.7 |
| ChatGPT-User | visits on a user’s request | 35 | 3.7 |
| PerplexityBot | search | 22 | 2.3 |
| OAI-SearchBot | search | 18 | 1.9 |
| Claude-SearchBot | search | 18 | 1.9 |
The numbers for each crawler include the domains that block all crawlers. Leaving those out, only 25 domains, or 2.6 percent, block the search bots or the bots that visit on a user’s request.
So Polish domains more often shut out crawlers that collect text for training and rarely the ones that look up answers for a user during a chat. That makes sense. Blocking training is a deliberate choice. Blocking search means an AI assistant will not read the site when someone asks about the company.
Sitemaps and HTTPS
497 domains, or 52 percent, give the sitemap address in robots.txt. Another 184, or 19 percent, have a sitemap.xml file at the usual address. In total, a sitemap can be found on 681 domains, or 71 percent. On 263 domains, or 28 percent, we found none either way. On 11 domains robots.txt did not allow us to ask, so we did not.
904 domains, or 94.7 percent, answered over HTTPS with a valid certificate.
What this study does not show
The Tranco list covers popular domains. Small businesses appear in it less often than large services, so the results do not describe all Polish websites.
We did not try the www version, so domains that only work under www are among those that did not respond. We only checked robots.txt and only the home page. A crawler can also be blocked by a server firewall or the hosting provider’s settings, which the file does not show. We looked for a sitemap in two places. If it lives elsewhere and is not named in robots.txt, we count it as missing, although Google may have received it another way. Some servers do not answer the HEAD request we used for sitemap.xml correctly, so some of those domains may also be counted as missing.
What to do on your own site
Open your website address with /robots.txt at the end and check that there is no block for AI search bots that you did not set on purpose. Our free tool can AI see your website does it faster. If you want to know where AI can help the business itself, not just the website, let us talk in the free AI audit.
What would you like to change in your business?
Leave your number. We will talk about your process and see whether we can help. The first call is free.
The service this text is about
WebsitesA website made for your company, not from a template. Before you pay, we show a working prototype.
Check it on your siteFree audit