Your robots.txt may be hiding you from AI assistants. Here is how to check.

The most common reason a local business is invisible to an AI engine is not content, not reviews, and not competition. It is a text file that says “go away” to the engine's crawler — often written years ago, often by accident.

The crawlers that matter

AI providers read the web with their own bots, each with its own user-agent name. The ones that affect whether you appear in answers:

Provider	User-agent	What it is for
OpenAI	GPTBot	Training data
OpenAI	OAI-SearchBot	ChatGPT search results
OpenAI	ChatGPT-User	Fetches pages when a user asks
Perplexity	PerplexityBot, Perplexity-User	Index and live answers
Google	Googlebot, Google-Extended	Search and AI Overviews; Gemini training
Anthropic	ClaudeBot, Claude-SearchBot, Claude-User	Index and live answers for Claude
Microsoft	Bingbot	Bing and Copilot

Two things follow. First, blocking GPTBot alone does not block ChatGPT search — OAI-SearchBot is a separate agent. Second, blocking all of them does not remove you from the web; it removes you from the answers.

How businesses block them by accident

Open yourdomain.com/robots.txt in a browser. Look for these patterns:

A blanket rule. User-agent: * followed by Disallow: / blocks every crawler, including Google. It is rare on live sites, but it happens on sites that were once “under construction”.

A plugin's default. Some CMS security plugins and CDN “bot protection” settings block anything that looks like an AI crawler, by default, without telling you. The robots.txt looks fine; the block happens at the firewall. The only way to see this is to request a page with the crawler's user-agent and check the response.

A rule copied from a blog post. In 2023–2024 many articles recommended blocking GPTBot to “protect your content”. For a publisher that may make sense. For a dentist whose content is “we exist, here are our hours”, it means the model learns about the competitor instead.

A robots.txt that admits AI crawlers

This keeps your existing rules and explicitly allows the AI agents. Add the AI section below whatever you already have; do not delete your existing lines.

# Existing rules stay as they are, for example:
User-agent: *
Allow: /
Disallow: /wp-admin/

# AI crawlers — allow explicitly
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Bingbot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

If you use a CDN with bot protection (Cloudflare and similar), check its settings separately: allow verified AI crawlers, or add the user-agents above to the allow list. A robots.txt that says “allow” does nothing if the firewall returns an error page.

How to verify

From a terminal, request your home page as GPTBot and check the status code:

curl -A "GPTBot" -I https://yourdomain.com/

You want HTTP/2 200. A 403 or a challenge page means the firewall is blocking the crawler regardless of robots.txt. Repeat for OAI-SearchBot and PerplexityBot.

The second half of the check

Being crawlable is necessary, not sufficient. A page the crawler can fetch but cannot read — because every word is rendered by JavaScript after load — is still empty to the model. The same curl command with -s instead of -I, piped to grep, tells you whether your prices and phone number are in the raw HTML:

curl -s -A "GPTBot" https://yourdomain.com/ | grep -i "phone"

If nothing comes back, the content exists only after JavaScript runs, and the fix is on the developer's side: server-side rendering, or at least the key facts in the HTML.

Anserra's free scan runs both checks, for every crawler in the table above, and writes the robots.txt for you from what it finds. But the manual version is ten minutes, and it is worth doing once so you understand what the report is telling you.