How to appear in Claude search. Check which crawler you are allowing
Claude-SearchBot, Claude-User and ClaudeBot have different jobs. Check public page access, source accuracy and actual search citations.
In this article
In short
Anthropic separates search crawling, user-directed visits and model training. For search visibility, check Claude-SearchBot. Allowing access alone does not guarantee a citation or a recommendation for your business.
You find a ClaudeBot block in robots.txt and consider removing it to improve business visibility. First check what that agent does.
Know which agent you are changing
Anthropic's crawler documentation assigns different roles to its agents. Claude-SearchBot supports search. Claude-User visits websites for users. ClaudeBot collects content that may contribute to model training.
Make the search-access decision separately from the training decision. A name containing Claude does not tell you that both uses are covered.
Test the actual public page
Open the service page without logging in. Check whether its server response contains the description and whether a hosting firewall shows a verification screen.
Review both the general robots.txt group and any group naming Claude-SearchBot. A named group needs its own review. Do not replace a whole policy with an example from a tutorial.
Keep private documents and customer areas protected. A public service description does not require access to customer records.
Give a buyer enough information
Explain the problem, the work included and what gets handed over. For an integration, name the systems being connected. For an application, describe the process it supports.
Use real project results only. Explain the measure and period behind a percentage rather than adding an impressive number without evidence.
Check reading and discovery separately
With search enabled, first ask Claude about a supplied URL. In a fresh conversation, ask the customer's question without naming the company. Open cited sources and record the question, date and page address.
Keep attempts without a citation in the record too. Otherwise, your comparison will only cover successful answers.
Questions about Claude search
Does blocking ClaudeBot block search?
Anthropic assigns ClaudeBot to training. Review search access separately for Claude-SearchBot.
Does a pasted link prove discovery?
It shows what Claude can do with the address you supplied. It does not prove discovery without that hint.
What if the answer describes the company incorrectly?
Open the cited page and check the information behind the answer. Correct outdated service details and conflicting company descriptions at their source.
What should you send to your hosting provider?
Send the exact URL and the name Claude-SearchBot. Ask them to check robots.txt and firewall rules. For a particular visit, the request time and server response can help. Opening the page in your own browser is not enough.
Does a company mention mean a supplier recommendation?
Open the source and read the sentence containing the company name. It may be an example in a tutorial. Record service recommendations separately from mentions.
Compare the process with our guides to Perplexity and ChatGPT.
What would you like to change in your business?
Leave your number. We will talk about your process and see whether we can help. The first call is free.
The service this text is about
WebsitesA website made for your company, not from a template. Before you pay, we show a working prototype.
Check it on your siteFree audit