AI crawlers in robots.txt on Polish websites. We checked 1000 domains

Of 955 Polish domains that responded, 17 percent block at least one AI crawler, and under 3 percent block the bots that look up answers for users. Method and numbers.

In this article
  1. How we checked robots.txt on Polish domains
  2. How many domains responded
  3. Which AI crawlers Polish domains block
  4. Sitemaps and HTTPS
  5. What this study does not show
  6. What to do on your own site

In short

We checked 1000 random Polish domains from the Tranco list. 955 responded, and all percentages are based on them. 17 percent block at least one AI crawler in robots.txt, almost always the ones that collect data for model training. Under 3 percent block the bots that look up answers for a user during a chat. A sitemap can be found on 71 percent of domains, and HTTPS works on almost 95 percent.

When someone asks ChatGPT or Claude about a company, the tool often visits websites and reads them during the chat. Whether it may do so depends partly on the robots.txt file. We wanted to see what Polish websites tell AI crawlers in that file, so we studied 1000 random Polish domains.

How we checked robots.txt on Polish domains

The domains come from the Tranco list. It is a public list of the most popular domains in the world, built by researchers for studies and hardened against manipulation (tranco-list.eu). We used the list with ID 8P8NV, downloaded on 9 November 2026. Of its million domains, 9401 end in .pl. We drew 1000 of them with a fixed random seed, so the same sample can be repeated.

This is a list of popular domains, not a list of company websites. It contains shops, media, portals, public bodies, universities and small business sites. We did not split them by type, because we have no reliable method for that.

We checked each domain with one or two requests, one at a time, with a one second pause between domains.

  1. We fetched robots.txt over HTTPS. If HTTPS did not respond, we tried HTTP once.
  2. We read the rule for each AI crawler from that file using the same code our tool can AI see your website uses. What counts is whether the crawler may open the home page.
  3. We counted a sitemap as present when robots.txt gives its address. If it did not, we sent one request for sitemap.xml, and only when robots.txt allowed it.

That was at most three requests per domain and one and a half on average. The study took 28 minutes. We publish only totals, with no domain names.

How many domains responded

Of 1000 domains, 955 responded. The other 45 did not. 19 of them failed our DNS address check, almost always because the bare domain without www has no address. 26 did not answer within 10 seconds. All percentages below are based on the 955 domains that responded.

Which AI crawlers Polish domains block

800 domains, or 84 percent, have a robots.txt file. One in five, 199 domains to be exact, names at least one AI crawler from our list.

162 domains, or 17 percent, block at least one AI crawler. That includes 15 domains that close the home page to all crawlers at once.

CrawlerUsed forDomains blockingPercent
ClaudeBotmodel training13614.2
GPTBotmodel training13313.9
CCBotCommon Crawl dataset11512.0
Google-Extendedmodel training879.1
Applebot-Extendedmodel training838.7
ChatGPT-Uservisits on a user’s request353.7
PerplexityBotsearch222.3
OAI-SearchBotsearch181.9
Claude-SearchBotsearch181.9

The numbers for each crawler include the domains that block all crawlers. Leaving those out, only 25 domains, or 2.6 percent, block the search bots or the bots that visit on a user’s request.

So Polish domains more often shut out crawlers that collect text for training and rarely the ones that look up answers for a user during a chat. That makes sense. Blocking training is a deliberate choice. Blocking search means an AI assistant will not read the site when someone asks about the company.

Sitemaps and HTTPS

497 domains, or 52 percent, give the sitemap address in robots.txt. Another 184, or 19 percent, have a sitemap.xml file at the usual address. In total, a sitemap can be found on 681 domains, or 71 percent. On 263 domains, or 28 percent, we found none either way. On 11 domains robots.txt did not allow us to ask, so we did not.

904 domains, or 94.7 percent, answered over HTTPS with a valid certificate.

What this study does not show

The Tranco list covers popular domains. Small businesses appear in it less often than large services, so the results do not describe all Polish websites.

We did not try the www version, so domains that only work under www are among those that did not respond. We only checked robots.txt and only the home page. A crawler can also be blocked by a server firewall or the hosting provider’s settings, which the file does not show. We looked for a sitemap in two places. If it lives elsewhere and is not named in robots.txt, we count it as missing, although Google may have received it another way. Some servers do not answer the HEAD request we used for sitemap.xml correctly, so some of those domains may also be counted as missing.

What to do on your own site

Open your website address with /robots.txt at the end and check that there is no block for AI search bots that you did not set on purpose. Our free tool can AI see your website does it faster. If you want to know where AI can help the business itself, not just the website, let us talk in the free AI audit.

What would you like to change in your business?

Leave your number. We will talk about your process and see whether we can help. The first call is free.

When is the best time to call?
Controller: Oleksandr Bondarenko (BondSoft)Privacy PolicyTerms of Service
Legal details

The data controller is Oleksandr Bondarenko, trading as Oleksandr Bondarenko Software Solutions (BondSoft), ul. Jana Heweliusza 11, 80-890 Gdańsk, Poland, NIP 5833576264, e-mail: office@bondsoft.pl.

We process your data solely to contact you about your enquiry and to present an offer — on the basis of your consent (Art. 6(1)(a) GDPR and Art. 398 of the Polish Electronic Communications Law) and our legitimate interest in handling the enquiry (Art. 6(1)(f) GDPR).

Providing the data is voluntary but necessary for us to reply. You may withdraw your consent at any time by writing to office@bondsoft.pl — this does not affect the lawfulness of processing before the withdrawal.

You have the right to access your data, rectify it, erase it, restrict its processing, transfer it, and lodge a complaint with the President of the Polish Personal Data Protection Office.

Full information: Privacy Policy.

Send us an email

The service this text is about

Websites

A website made for your company, not from a template. Before you pay, we show a working prototype.

Check it on your siteFree audit