llms.txt, robots.txt and AI Crawlers: A 30-Minute Setup Checklist

Three plain files decide what AI crawlers can read on your site. Here is the 30-minute setup, checked on November 11, 2026 against what OpenAI, Anthropic, Google and Perplexity say on their own pages.
llms.txt, robots.txt and AI Crawlers: A 30-Minute Setup Checklist

Three plain files decide what AI crawlers can read on your site. Here is the 30-minute setup, checked on November 11, 2026 against what OpenAI, Anthropic, Google and Perplexity say on their own pages.

Where the data comes from: the checklist’s own November 11, 2026 reread of the crawler pages of OpenAI, Anthropic, Google and Perplexity, and the sources named in each row.

Panel titled Which AI Crawlers to Allow: Search, allow: OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot (Perplexity); Training, your call: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), CCBot (Common Crawl), Bytespider (ByteDance)

Numbers in the post

Number What the post says Source Date
305% “Cloudflare’s July 2025 crawler report counted GPTBot requests up 305% between May 2024 and May 2025” Cloudflare, “From Googlebot to GPTBot: who’s crawling your site in 2025” May 2024 to May 2025 (report of July 2025)
80% / 18% / 2% “Cloudflare’s August 29, 2025 (last 12 months) breakdown of AI crawling said 80% was for training and 18% search and 2% user actions” Cloudflare, “The crawl-to-click gap” August 29, 2025 (last 12 months)
44.2% of 18,012 “in Kevin Indig’s Growth Memo analysis of 18,012 ChatGPT citations, published February 16, 2026, 44.2% came from the first 30% of the text” Kevin Indig, Growth Memo February 16, 2026
nearly 300,000 “SE Ranking even looked at nearly 300,000 domains and found no correlation (measurably) between having it and being cited more” SE Ranking Page dated September 2026
0 of 4 “Neither Anthropic nor Google mention it at all. And OpenAI and Perplexity only mention it to refer readers to their respective documentation index. So none of the 4 claim their crawlers read the file.” Search Shift check of the crawler pages of OpenAI, Anthropic, Google and Perplexity November 11, 2026
about 24 hours “OpenAI says that it can take about 24 hours for robots.txt changes to propagate to their search functionality” OpenAI crawler page Checked November 11, 2026 (page undated)
200 “Its robots.txt returned 200 and holds an Allow: / group for GPTBot, ClaudeBot and PerplexityBot. The homepage returned 200 to a GPTBot user-agent string, a ClaudeBot one and a plain browser one, and rctrl.com/llms.txt returned 200.” Search Shift check of rctrl.com November 11, 2026

Crawlers in the post and what it says to do with each

Crawler Company What the post says it does What the post says to do Source, date
OAI-SearchBot OpenAI “fetch pages so an AI product can show them in answers”; “sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers” Allow. “First three are search crawler friendly.” “block OAI-SearchBot to keep your pages from being in its answers” OpenAI crawler page, November 11, 2026
Claude-SearchBot Anthropic “fetch pages so an AI product can show them in answers” Allow. “First three are search crawler friendly.” Anthropic crawler page, November 11, 2026
PerplexityBot Perplexity “PerplexityBot is not used to train foundation models” Allow. “First three are search crawler friendly.” Perplexity crawler page, November 11, 2026
GPTBot OpenAI “OpenAI docs GPTBot as the training crawler in 2026”; “if you block GPTBot/ClaudeBot, it’s off limits for training” Disallow. “Last five are opting out of training.” “Block GPTBot and ChatGPT search can still be used” OpenAI crawler page, November 11, 2026
ClaudeBot Anthropic “if you block GPTBot/ClaudeBot, it’s off limits for training” Disallow. “Last five are opting out of training.” Anthropic crawler page, November 11, 2026
Google-Extended Google “a robots.txt token that does not have a user-agent string”; “used to control the training for future models of Gemini and also the grounding part”; “doesn’t impact Google Search” Disallow. “Definitely take a second look at Google-Extended before blocking because Google puts Gemini Apps grounding alongside training in what the token controls.” Google crawler page, November 11, 2026
CCBot Common Crawl “CCBot (open training data of Common Crawl)” Disallow. “Last five are opting out of training.” The post’s list (no source page named)
Bytespider ByteDance “Bytespider (ByteDance for TikTok/Doubao)”; “Bytespider respects robots.txt not always” Disallow. “Last five are opting out of training.” The post’s list (no source page named)
GoogleOther Google “generic crawler for one-off research crawls etc., rules for it don’t impact any specific product” Not in the post’s file. “Then check Googlebot (not GoogleOther).” Google crawler page, November 11, 2026
Googlebot Google “AI is incorporated into Search”; “So Googlebot robots.txt is the key” “Then check Googlebot (not GoogleOther).” Google AI features page, November 11, 2026
ChatGPT-User OpenAI “pulling one page as a result of being asked by a human”; “OpenAI’s crawler page makes it sound like they might ignore robots.txt stuff for ChatGPT-User” Not in the post’s file OpenAI crawler page, November 11, 2026
Claude-User Anthropic “pulling one page as a result of being asked by a human”; “If you turn off Claude-User, Anthropic claims that will prevent them pulling your stuff for the user’s question” Not in the post’s file Anthropic crawler page, November 11, 2026
Perplexity-User Perplexity “pulling one page as a result of being asked by a human”; “Perplexity claims they generally don’t respect robots.txt stuff for Perplexity-User” Not in the post’s file Perplexity crawler page, November 11, 2026

The post’s robots.txt file

# Allow AI search crawlers
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Opt out of model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

Write a comment