SI Readiness by MCP-Tool
robots.txt lets AI crawlers in
Does robots.txt allow the crawlers behind AI answers to read the page?
Why it matters
Assistants that cite the web fetch through named agents. A blanket rule written for search engines years ago can quietly exclude every one of them, and nothing in your analytics will tell you.
How to fix it
Publish a robots.txt that names the AI agents you want and allows the paths you want cited. Blocking is a legitimate choice — but make it a decision, not an accident.
User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / Sitemap: https://example.com/sitemap.xml