AI Crawler robots.txt Generator

Block GPTBot, ClaudeBot, Google-Extended and 18 other AI user agents while keeping ChatGPT, Claude and Perplexity search citations, then copy a ready robots.txt.

Crawler list as of September 2026.

Last updatedHow we build & check our tools
Results will appear here once you enter your values.

How the generator works

Start from one of three presets: Block training, allow AI search (the default), Block all AI bots or Allow all AI bots. Then override any of the 21 user-agent tokens individually or flip a whole group at once. The groups are model training (12 tokens), AI search and answers (4) and user-initiated fetches (5).

The output lists blocked bots in one group per category, each ending in Disallow: /. Allowed AI bots can be listed in their own group so your intent is documented, followed by an optional User-agent: * group. Paths you enter, such as /admin or /cart, are added as Disallow lines to the allowed groups, with a leading slash added if you forget it. A valid absolute Sitemap URL is appended at the end. Copy the result or download it as robots.txt.

Training bots versus AI search bots

Most AI companies now split crawling by purpose, which lets you opt out of training without vanishing from AI answers.

  • OpenAI: GPTBot collects training data, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User fetches a page when a user asks. OpenAI notes that sites blocking OAI-SearchBot will not appear in ChatGPT search answers.
  • Anthropic: ClaudeBot for training, Claude-SearchBot for search quality, Claude-User for user requests. Anthropic says its bots honor robots.txt and the Crawl-delay extension.
  • Google: Google-Extended is a control token, not a separate crawler. Blocking it opts content out of Gemini training and grounding and, per Google, does not affect inclusion or ranking in Google Search.
  • Apple and Amazon: Applebot-Extended controls use of Applebot data for Apple model training, and Amazon says Amazonbot data may be used to train its AI models.

Common mistakes

  • Treating robots.txt as security. RFC 9309 states the rules are not a form of access authorization. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Bytespider is widely reported to ignore them. Use a WAF or bot management rule for real enforcement.
  • Blocking Googlebot to stop Gemini. That removes you from Google Search. Block the Google-Extended token instead.
  • Duplicating User-agent: *. A crawler follows the most specific group that names it, and RFC 9309 merges groups with the same token. Keep one wildcard group and place the AI groups above it when merging with an existing file.
  • Uploading to a subfolder. The file only works at the root of each host, such as https://example.com/robots.txt. A separate file is needed for every subdomain.

Tips for maintaining your rules

Crawlers cache robots.txt, and RFC 9309 says they should not use a cached copy for more than 24 hours, so allow a day before judging whether a change worked. Check your server logs for the user-agent tokens afterwards.

Revisit the file every few months. New AI user agents appear often, and the generated file includes a comment with the list date (2026-09) as a reminder. Keep the legacy anthropic-ai token; it costs nothing.

If you rely on AI search traffic, the default preset is usually the right trade: training bots blocked, OAI-SearchBot, Claude-SearchBot, PerplexityBot and DuckAssistBot allowed. Remember that paths you disallow apply to those allowed bots too, so do not list pages you want cited.

Frequently Asked Questions

Common questions about the AI Crawler robots.txt Generator

No. GPTBot only collects content that may be used to train OpenAI models. ChatGPT search uses a separate crawler, OAI-SearchBot, and OpenAI says sites that block OAI-SearchBot will not appear in ChatGPT search answers. The default preset in this generator blocks GPTBot while allowing OAI-SearchBot, so you can opt out of training and still be cited.
From the same team

Stop paying per token — route AI requests to your own GPU

Wide Area AI is a local-first AI gateway: repeated requests hit an edge cache, the rest run free on your own hardware, and the cloud is only a failover. OpenAI-compatible endpoint, free tier.

Start routing — free
VRAM CalculatorCan I Run This AI?AI API Cost CalculatorGPU for Model

Sources & References

RFC 9309: Robots Exclusion Protocol

Robots Exclusion Protocol: group matching, caching, and that rules are not access authorization.
→Source: IETF (RFC Editor)

Overview of OpenAI Crawlers

GPTBot, OAI-SearchBot and ChatGPT-User tokens and their purposes.
→Source: OpenAI

Does Anthropic crawl data from the web, and how can site owners block the crawler?

ClaudeBot, Claude-SearchBot and Claude-User tokens and robots.txt support.
→Source: Anthropic

Google's common crawlers (Google-Extended)

Google-Extended token scope; no effect on Google Search ranking.
→Source: Google

Perplexity Crawlers

PerplexityBot and Perplexity-User purposes and robots.txt behavior.
→Source: Perplexity

Explore More Tools

Explore these related tools for your next step.

ai

LLM Token Usage Calculator

Calculate token usage and associated costs for LLM prompts.

Try it now
ai

AI Embedding Cost Calculator

Estimate the total token count and API cost to embed a document corpus using OpenAI, Cohere, or Voyage embedding models.

Try it now
ai

Neural Network Playground

Train a real neural network in your browser — no code, no installs. Pick a sample dataset (Titanic survival, vehicle classifier, handwritten digits) or upload your own CSV, watch the network learn with live visuals, then ask it questions and export the model as Python/Keras code.

Try it now
ai

LLM Token Counter

Count tokens in text for GPT, Claude, Gemini, and open models like Llama, Gemma, and Qwen — or search any model on Hugging Face. Estimate API costs, check context window fit, and optimize prompts

Try it now
ai

LLM VRAM Calculator

Calculate how much VRAM any LLM needs to run locally — pick a model (Llama, Gemma, Qwen, DeepSeek, or search Hugging Face), choose a quantization and context size, and see which GPUs it fits on, including multi-GPU setups.

Try it now
ai

LLM Inference Speed Calculator

Estimate LLM tokens per second from memory bandwidth, model size, quantization, and context window. Compare generation speed across GPUs (RTX 4090, 5090, A100, H100, Apple Silicon) and understand the memory-bandwidth bottleneck.

Try it now