GPTBot and robots.txt: allow, block, and check

Minimal allow example (marketing host)

User-agent: GPTBot Allow: / Disallow: /app Disallow: /login

Mirror the same pattern for ChatGPT-User, PerplexityBot, ClaudeBot, and Google-Extended if you want multi-engine citations.

Keep authenticated dashboards and tokenized reports disallowed for every user-agent.

Common mistakes

Cloudflare or other managed “block AI scrapers” features that Disallow GPTBot while your marketing claims AI openness — a direct AEO contradiction.

Allowing * but forgetting that some platforms honor bot-specific Disallow more strictly.

Putting critical content behind client-only JavaScript with empty server HTML so bots fetch shells.

How to verify

Fetch https://yoursite.com/robots.txt with curl and as GPTBot user-agent.

Run an Adticks technical audit to map AI bot access and render parity in one report.

Confirm sitemap.xml is reachable and lists only public canonical URLs.

FAQ

Does blocking GPTBot stop ChatGPT from knowing my brand?

It blocks fresh fetch for that bot. Models may still use prior training data or third-party sources, but you lose control over which pages get cited.

Should I block all AI bots?

Only if you intentionally opt out of AI citation and browsing. For most B2B SaaS marketing sites, allowing citation bots on public pages is the default AEO strategy.