Every week a client asks the same question: should I add an llms.txt file, and should I block AI from reading my site? Both questions deserve a clear, unhyped answer — because the wrong call costs you visibility in exactly the place where buyers now research before they ever reach your site.

What llms.txt actually is

llms.txt is a proposed standard: a single Markdown file at the root of your domain (/llms.txt) that lists your most valuable pages so an LLM can find and understand them quickly, without crawling clutter. Think of it as a curated table of contents written for machines — a short, plain-language map that says "here is what matters and here is where it lives."

The honest part: as of early 2026, no major AI provider has officially confirmed it reads llms.txt. Google, OpenAI, Anthropic and Perplexity have not endorsed it, and there is genuine debate about whether it will ever see real adoption. So treat it as low-cost insurance, not a ranking lever — a few hours to create, harmless if ignored, and potentially useful if the standard catches on. What it is not is a substitute for clean site structure, a sitemap, or strong content.

The major AI crawlers and their user-agents

Controlling AI access starts with knowing who is knocking. The user-agents that matter most:

• GPTBot — OpenAI, used to gather training data
• OAI-SearchBot — OpenAI, powers ChatGPT search results
• ClaudeBot — Anthropic, training and retrieval
• PerplexityBot — Perplexity, indexing for its answer engine
• Google-Extended — Google's toggle for Gemini and AI training (separate from normal Google indexing)
• Bytespider, Amazonbot, Applebot-Extended — other large players worth knowing

A key distinction: blocking Google-Extended does not remove you from normal Google Search. It only opts you out of AI training. Confusing the two is the most common mistake I see.

How to control access with robots.txt

This is the file crawlers genuinely respect. You set rules per user-agent in plain text at /robots.txt. To block AI training crawlers while keeping classic SEO intact:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

Normal search engines still welcome

User-agent: *
Allow: /
```

Remember: robots.txt is a polite request, not a wall. Reputable crawlers obey it; bad actors may not. For truly sensitive material, use authentication, not robots.txt.

Block or allow? The real trade-off

This is the strategic question. Blocking protects your content from being used to train models and scraped without credit. Allowing means your expertise can be quoted, cited and surfaced when someone asks ChatGPT or Perplexity about your industry — increasingly where consideration begins.

Weigh it like this:

• Lean toward blocking if your content is your paid product (courses, research, paywalled data) or you operate under strict licensing.
• Lean toward allowing if you want brand visibility, leads and authority — most service businesses, B2B, local providers and publishers funded by demand rather than the text itself.

There is no neutral default. Choosing to block is choosing to be absent from AI answers about your category — and your competitors will happily fill that silence.

Watch what crawlers actually do

A policy you never check is a guess. Once your rules are live, review your server logs or analytics for AI user-agents to confirm two things: that the crawlers you allowed are visiting, and that the ones you blocked have backed off. If a bot keeps hitting paths you disallowed, that tells you it is either misconfigured or not playing by the rules — useful signal either way. Revisit this quarterly, because crawler names and behavior change faster than most documentation.

A balanced policy for most businesses

For the majority of companies I advise, the pragmatic stance is:

1. Allow search and answer-engine crawlers (OAI-SearchBot, PerplexityBot, Googlebot) so you appear in AI-driven results.
2. Decide deliberately on pure training crawlers (GPTBot, Google-Extended) — block only if you have a real licensing or IP reason.
3. Publish a tidy llms.txt highlighting your cornerstone pages — cheap, future-proofing.
4. Keep content genuinely citable: clear answers near the top, structured headings, factual depth. This GEO/AEO work is what actually earns citations.

Visibility in AI answers is becoming as decisive as ranking on page one once was. Control the door deliberately — but for most, the smart move is to leave it open and make your content worth quoting.