Resources
AI Crawlers and you:
How to block or allow them using robots.txt
The age of AI is here, and it's actively reading, interpreting, and learning from your website. This presents every site owner with a critical choice: do you want to embrace AI discovery, or do you want to protect your content from being used in language model training?
Fortunately, you have control. Whether you want to optimize your site for AI visibility or block access entirely, the primary tool for the job is a simple text file: robots.txt. This guide will provide you with the user agents and templates you need to implement your AI strategy today.
Understanding Your Control Toolkit
What is robots.txt?
At its core, robots.txt is a public text file that lives on your website's server. It gives instructions to web crawlers — including AI bots — about which pages and files they are allowed or forbidden to request. It is the web's standard for crawler protocol. For a deep dive, refer to the official documentation from Google's Developer Center.
What about llms.txt and ai.txt?
These are newer, experimental files proposed to give more specific instructions to AI models. While robots.txt is the main "gatekeeper," you could use llms.txt to point AI models to content specifically optimized for them, creating a better-tailored experience. For now, the best practice is to ensure your robots.txt rules are clear, as it is the most widely respected standard.
Is this an all-or-nothing choice?
Not at all. You have granular control. You can choose to block a specific AI crawler while allowing others. You can also direct one bot to your main content while pointing another to a different, AI-optimized section of your site.
Important: Is this guaranteed to work?
robots.txt operates on an honor system. Think of it as a "No Trespassing" sign on your lawn, not a locked gate. Reputable crawlers (like those from Google, OpenAI, and other major tech companies) will respect your rules. However, less scrupulous bots may ignore them. It is your first and most important line of defense, but not a foolproof security measure.
Known AI & LLM Crawler User Agents
User agents for major AI crawlers and general-purpose bots known to feed data to LLMs. Linked sources provide official documentation from each provider.
Last Updated: April 17, 2026
Primary AI & LLM Crawlers
| User-agent |
|---|
| GPTBot |
| Google-Extended |
| ClaudeBot |
| Amazonbot |
| Applebot-Extended |
| meta-externalagent |
| cohere-ai |
| PerplexityBot |
| YouBot |
| Bytespider |
| Diffbot |
| CCBot |
General Bots That Can Feed LLMs
| User-agent |
|---|
| Bingbot |
| Googlebot |
| Applebot |
Our Own Crawler
robots.txt and never performs unsolicited crawling. PaceghostBot documentation → robots.txt Templates for AI Governance
Template 1: Block All Known AI Crawlers
Use this template if your goal is to prevent your content from being used by major AI models for training and summarisation.
# === BLOCK ALL MAJOR AI CRAWLERS ===
# Template provided by paceghost.io - The AI Readiness Platform
# Last updated: April 17, 2026
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: cohere-ai
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: YouBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Diffbot
Disallow: /
User-agent: CCBot
Disallow: /
# == Your Standard Rules ==
User-agent: *
Disallow: /admin/
Disallow: /private/
# == Sitemap Location ==
Sitemap: https://www.your-website.com/sitemap.xmlTemplate 2: Allow All Known AI Crawlers
Use this template if your goal is to maximise your visibility and ensure your content is discoverable by AI search and answer engines.
# === ALLOW ALL MAJOR AI CRAWLERS ===
# Template provided by paceghost.io - The AI Readiness Platform
# Last updated: April 17, 2026
# This template ensures AI crawlers can access your site while still protecting
# sensitive areas (like admin panels). By default, crawlers are allowed,
# so we only need to specify what to disallow.
# == Default Rules for All Crawlers (including AI) ==
User-agent: *
Disallow: /admin/
Disallow: /dashboard/
Disallow: /login/
# == Sitemap Location ==
Sitemap: https://www.your-website.com/sitemap.xmlWhat's Next?
Defining your rules is a critical first step. But the AI landscape changes daily. New crawlers appear constantly, and you're left wondering:
- Is my robots.txt file still effective?
- Are bots respecting my rules?
- How is AI actually interpreting my content?
That's where Paceghost comes in. We're building the toolkit to turn this manual checklist into an automated, effortless dashboard.
Stay in the loop
The AI crawler landscape moves fast.
We update this list as new bots emerge. Subscribe to get notified when we publish new resources, templates, and guides on AI readiness.