GPTBot and robots.txt: allow, block, and check

Last updated: August 16, 2026 · By Madhu Dadi

Direct answer

GPTBot is OpenAI’s training crawler. Disallow GPTBot means crawled pages should not be used to train OpenAI foundation models. OAI-SearchBot is a separate bot that surfaces sites in ChatGPT Search. The two robots.txt tags are independent: blocking GPTBot is not the citation switch. Allow or Disallow OAI-SearchBot if you want ChatGPT Search visibility. Verify the live robots.txt — CDNs can inject stricter rules.

Minimal allow example (marketing host)

User-agent: GPTBot Allow: / Disallow: /app Disallow: /login User-agent: OAI-SearchBot Allow: / Disallow: /app Disallow: /login

GPTBot and OAI-SearchBot are independent. Disallow GPTBot only to opt out of training — that is not the ChatGPT Search citation switch. Name-allow OAI-SearchBot, Claude-SearchBot, ChatGPT-User, and PerplexityBot for search-visibility. GPTBot, ClaudeBot, and Google-Extended are training/grounding tokens (Adticks allows them as product choice). Allow ≠ cited.

Keep authenticated dashboards and tokenized reports disallowed for every user-agent.

Common mistakes

Cloudflare or other managed “block AI scrapers” features that Disallow GPTBot or OAI-SearchBot while your marketing claims AI openness — a robots policy vs product-claim contradiction.

Treating Disallow GPTBot as the ChatGPT citation switch. ChatGPT Search uses OAI-SearchBot; the two robots.txt tags are independent.

Putting critical content behind client-only JavaScript with empty server HTML so bots fetch shells.

How to verify

Fetch https://yoursite.com/robots.txt with curl and as the GPTBot and OAI-SearchBot user-agents.

Run an Adticks technical audit to map AI bot access and render parity in one report.

Confirm sitemap.xml is reachable and lists only public canonical URLs.

FAQ

Does blocking GPTBot stop ChatGPT from knowing my brand?
Disallow GPTBot opts those URLs out of OpenAI training crawls. It does not by itself stop ChatGPT Search citations — that is OAI-SearchBot. Models may still use prior training data or third-party sources.
Should I block all AI bots?
Only if you intentionally opt out of training crawls and search-visibility. For most B2B SaaS marketing sites, name-allow search-visibility bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User) on public pages, and treat GPTBot, ClaudeBot, and Google-Extended as a training product choice. Allow ≠ cited.

Sources

Related guides

Next step

Run a free technical and AI-crawler readiness audit on your domain — no script install.

Claim free auditCompare plans on /pricing