What is robots.txt?
Updated:
robots.txt is the file that tells crawlers which parts of a site they may access; for AI visibility it decides whether crawlers such as GPTBot, ClaudeBot and PerplexityBot can read the site at all.
Blocking these crawlers, often by accident through a generic rule, is the single most common reason a business is invisible to an engine.
Related: AI crawler, llms.txt
See what AI says about your business today
One business, one-time check. No account, no card. Results in about 60 seconds.