GPTBot and robots.txt: allow, block, and check
Last updated: August 16, 2026 · By Madhu Dadi
Direct answer
GPTBot is OpenAI’s training crawler. Disallow GPTBot means crawled pages should not be used to train OpenAI foundation models. OAI-SearchBot is a separate bot that surfaces sites in ChatGPT Search. The two robots.txt tags are independent: blocking GPTBot is not the citation switch. Allow or Disallow OAI-SearchBot if you want ChatGPT Search visibility. Verify the live robots.txt — CDNs can inject stricter rules.
Minimal allow example (marketing host)
User-agent: GPTBot Allow: / Disallow: /app Disallow: /login User-agent: OAI-SearchBot Allow: / Disallow: /app Disallow: /login
GPTBot and OAI-SearchBot are independent. Disallow GPTBot only to opt out of training — that is not the ChatGPT Search citation switch. Name-allow OAI-SearchBot, Claude-SearchBot, ChatGPT-User, and PerplexityBot for search-visibility. GPTBot, ClaudeBot, and Google-Extended are training/grounding tokens (Adticks allows them as product choice). Allow ≠ cited.
Keep authenticated dashboards and tokenized reports disallowed for every user-agent.
Common mistakes
Cloudflare or other managed “block AI scrapers” features that Disallow GPTBot or OAI-SearchBot while your marketing claims AI openness — a robots policy vs product-claim contradiction.
Treating Disallow GPTBot as the ChatGPT citation switch. ChatGPT Search uses OAI-SearchBot; the two robots.txt tags are independent.
Putting critical content behind client-only JavaScript with empty server HTML so bots fetch shells.
How to verify
Fetch https://yoursite.com/robots.txt with curl and as the GPTBot and OAI-SearchBot user-agents.
Run an Adticks technical audit to map AI bot access and render parity in one report.
Confirm sitemap.xml is reachable and lists only public canonical URLs.
FAQ
- Does blocking GPTBot stop ChatGPT from knowing my brand?
- Disallow GPTBot opts those URLs out of OpenAI training crawls. It does not by itself stop ChatGPT Search citations — that is OAI-SearchBot. Models may still use prior training data or third-party sources.
- Should I block all AI bots?
- Only if you intentionally opt out of training crawls and search-visibility. For most B2B SaaS marketing sites, name-allow search-visibility bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User) on public pages, and treat GPTBot, ClaudeBot, and Google-Extended as a training product choice. Allow ≠ cited.
Sources
Related guides
Next step
Run a free technical and AI-crawler readiness audit on your domain — no script install.
Claim free auditCompare plans on /pricing