Before an AI assistant can recommend your hotel from your own website, it has to be allowed to read it. Many hotel sites block AI crawlers without the owner ever deciding to: a security plugin, a hosting firewall or a robots.txt line copied years ago. This check takes ten minutes and needs no technical knowledge beyond opening a web page.
Step 1: open your robots.txt
Type your address followed by /robots.txt, for example yourhotel.com/robots.txt. You will see a short text file. Look for two things:
- A line
User-agent: *followed byDisallow: /. This blocks every crawler, including Google. If you see it, call your web designer today. - Any of these names followed by
Disallow: /:OAI-SearchBot,ChatGPT-User,PerplexityBot,Claude-SearchBot,Googlebot.
If none of them appears, crawlers are allowed by default.
Step 2: know which crawler does what
The AI companies use different crawlers for different jobs. Blocking the wrong one costs you visibility; blocking the right one is a legitimate choice.
| Crawler | Company | What it does | If you block it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Reads pages for ChatGPT search answers | OpenAI says your pages won’t be shown in ChatGPT search answers |
| GPTBot | OpenAI | Collects content that may be used to train models | No effect on ChatGPT search |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT to | Single visits fail |
| PerplexityBot | Perplexity | Indexes pages for Perplexity answers | You are less likely to be cited |
| Claude-SearchBot | Anthropic | Indexes pages for Claude’s search results | You are less likely to be cited |
| Googlebot | Google Search, including AI Overviews and AI Mode | You disappear from Google entirely | |
| Google-Extended | Controls use of your content for training Gemini models | No effect on Search |
So a hotel that wants to appear in ChatGPT search but does not want its content used for training can allow OAI-SearchBot and disallow GPTBot. OpenAI says changes to robots.txt take about a day to reach its search systems.
Step 3: check the firewall, not just the file
robots.txt is a polite request. Firewalls and “bot protection” settings are a locked door, and they can block AI crawlers even when robots.txt allows them. Ask your web designer or hosting company one question:
Do any of our security settings block AI crawlers such as OAI-SearchBot, PerplexityBot or Claude-SearchBot?
If your site sits behind Cloudflare or a similar service, look at its bot settings as well. Some bot-protection options treat AI crawlers like any other bot and turn them away.
Step 4: Google’s new setting in Search Console
Since 31 August 2026, every site in Google Search Console has a Search generative AI setting (Settings → Search generative AI). It decides whether your content can appear in AI Overviews, AI Mode and generative features in Discover.
The default is to include your site. Check it once, and leave it on “include” unless you have a specific reason. Google notes that this setting does not affect AI training; that is controlled by Google-Extended.
Step 5: ask the assistants
Once the doors are open, find out what they say. Ask ChatGPT, Gemini and Perplexity the questions your guests ask:
- “Quiet family hotel in [your town] with parking”
- “Where to stay in [your town] for a long weekend in May”
- “Hotel in [your town] that allows dogs”
Note which hotels are named and which sources are cited. If you are missing, the cause is usually one of three things: missing structured data, a contradiction between your website and your listings, or no page that answers the question. Our free AI Visibility Check runs twenty such questions across four assistants and tells you which of the three applies to you.
Sources
- OpenAI: Overview of OpenAI crawlers
- OpenAI Help Center: ChatGPT search
- Google Search Console Help: Search generative AI control
Read on 4 November 2026.
