How to Block GPTBot and ClaudeBot With robots.txt
The rule is two lines. Knowing whether it's actually working is the part most guides skip.
Blocking an AI crawler from training on your site is a robots.txt rule, the same mechanism that's controlled search-engine crawling since 1994. If you already know how to write a Disallow rule, you already know how to do this — the syntax isn't the hard part. The two things that actually go wrong are: not knowing there's more than one crawler per company, and having no way to confirm the rule is doing anything after you save the file.
Already have a robots.txt? Check your domain on AI Crawler Checker to see which crawlers it actually blocks — the rule itself is below.
The rule itself
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Each block is independent — a crawler only respects the group whose User-agent line matches its own name, plus the wildcard * group if it has no specific one. Add one block per crawler you want to exclude; there's no single line that blocks "all AI crawlers" at once, because there's no user-agent token that means that.
It's not just one bot per company
| Company | Crawler(s) | What it's for |
|---|---|---|
| OpenAI | GPTBot | Crawls for model training |
| OpenAI | ChatGPT-User | Fetches a page live when a user asks ChatGPT to browse it — a different purpose, and blocking GPTBot does not block this one |
| Anthropic | ClaudeBot | Crawls for model training |
Google-Extended | Opts out of Gemini/AI training use specifically — separate from Googlebot, which handles Search indexing and is unaffected by this rule |
The Google-Extended distinction trips up the most people: blocking it does not remove you from Google Search results, because Googlebot (search indexing) and Google-Extended (AI training data) are separate tokens that Google's crawlers check independently. Disallowing the wrong one — or assuming one rule covers both purposes — is the single most common mistake in a hand-written AI-crawler block list.
Confirming the rule actually works
A robots.txt rule is advisory — well-behaved crawlers check it before fetching a page and respect what they find, but nothing on your server enforces it, and a typo in the user-agent name (a stray space, wrong capitalization treated too strictly by a hand-rolled parser, or a rule placed in the wrong group) can silently do nothing. The syntax also has sharper edges than it looks: per the underlying spec, the longest matching path pattern wins when rules conflict, and equal-length ties resolve to allow — so a broad Disallow: / for one crawler can be quietly overridden by a more specific Allow: rule elsewhere in the same file, if you're not tracking pattern length.
AI Crawler Checker fetches your domain's live robots.txt and runs it through the actual RFC 9309 matching rules — longest-pattern-wins, tie goes to allow — against 15 AI crawlers including GPTBot, ChatGPT-User, ClaudeBot and Google-Extended, and shows you the exact verdict per bot plus the rule that produced it. The same tool also validates an llms.txt file at /llms, for the newer file some AI systems read for site summaries.
Related pages here about a rule that passes a quick check and still fails in practice: Discord embed character limits explained, where every field is under its own limit and the 6,000-character total still rejects the embed, and Secret Santa generator vs. a random wheel, where a shuffle sorted with a random comparator makes some pairings more likely than others.