Skip to main content
Adticks
Why it mattersPlatformPricingGuidesFAQ
Log inClaim free audit
  1. Home
  2. /
  3. Guides
  4. /
  5. GPTBot and robots.txt: allow, block, and check

GPTBot and robots.txt: allow, block, and check

Last updated: September 3, 2026 · By Madhu Dadi

Direct answer

GPTBot is OpenAI’s training crawler. Disallow GPTBot means crawled pages should not be used to train OpenAI foundation models. OAI-SearchBot is a separate bot that surfaces sites in ChatGPT Search. The two robots.txt tags are independent: blocking GPTBot is not the citation switch. Allow or Disallow OAI-SearchBot if you want ChatGPT Search visibility. Verify the live robots.txt — CDNs can inject stricter rules.

Minimal allow example (marketing host)

User-agent: GPTBot Allow: / Disallow: /app Disallow: /login User-agent: OAI-SearchBot Allow: / Disallow: /app Disallow: /login

GPTBot and OAI-SearchBot are independent. Disallow GPTBot only to opt out of training — that is not the ChatGPT Search citation switch. Name-allow OAI-SearchBot, Claude-SearchBot, ChatGPT-User, and PerplexityBot for search-visibility. GPTBot, ClaudeBot, and Google-Extended are training/grounding tokens (Adticks allows them as product choice). Allow ≠ cited.

Keep authenticated dashboards and tokenized reports disallowed for every user-agent.

Common mistakes

Cloudflare or other managed “block AI scrapers” features that Disallow GPTBot or OAI-SearchBot while your marketing claims AI openness — a robots policy vs product-claim contradiction.

Treating Disallow GPTBot as the ChatGPT citation switch. ChatGPT Search uses OAI-SearchBot; the two robots.txt tags are independent.

Putting critical content behind client-only JavaScript with empty server HTML so bots fetch shells.

How to verify

Fetch https://yoursite.com/robots.txt with curl and as the GPTBot and OAI-SearchBot user-agents.

Run an Adticks technical audit to map AI bot access and render parity in one report.

Confirm sitemap.xml is reachable and lists only public canonical URLs.

FAQ

Does blocking GPTBot stop ChatGPT from knowing my brand?
Disallow GPTBot opts those URLs out of OpenAI training crawls. It does not by itself stop ChatGPT Search citations — that is OAI-SearchBot. Models may still use prior training data or third-party sources.
Should I block all AI bots?
Only if you intentionally opt out of training crawls and search-visibility. For most B2B SaaS marketing sites, name-allow search-visibility bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User) on public pages, and treat GPTBot, ClaudeBot, and Google-Extended as a training product choice. Allow ≠ cited.

Sources

  • OpenAI — GPTBot & bots docs
  • Cloudflare — Managed robots.txt

Related guides

  • What is AEO (Answer Engine Optimization)?
  • AEO vs SEO: what changes for your site
  • What is GEO (Generative Engine Optimization)?

Next step

Run a free technical and AI-crawler readiness audit on your domain — no script install.

Claim free auditCompare plans on /pricing

Footer

Adticks

SEO, AI visibility, and technical measurement — crawl, render, schema, and crawler access signals that affect whether engines can read your public pages.

Built by Madhu Dadi · Case study

Product

  • ·Create free account
  • ·Pricing
  • ·Open app
  • ·Features overview

Resources

  • ·Guides
  • ·What is AEO?
  • ·GPTBot robots.txt
  • ·About
  • ·Security
  • ·llms.txt

Connect

  • ·Privacy
  • ·Terms
  • ·Accessibility
  • ·Developer Blog
  • ·Deals
  • ·Udemy Enroller
  • ·TeleVault
  • ·Personal Website
  • ·Adticks Case Study
  • ·GitHub
  • ·X / Twitter
© 2026 Adticks. All rights reserved.
Privacy PolicyTerms of ServiceAccessibilitySecurity