How to set up robots.txt for Google and AI search (2026)

Allow the search bots you want to be found by, decide each training bot on its own, and keep every page you want read crawlable. A robots.txt AI crawler opt-out works one bot at a time, so give each bot its own group. Remember that robots.txt manages crawler traffic and is not a way to keep a page out of Google.

By Trendi · Updated 2 November 2026

Direct answer: Allow the search crawlers you want to appear through, opt out of training crawlers one by one, and keep pages crawlable so each bot can read your rules and your content.

Check your robots.txt now

What are AI crawlers?

AI crawlers are automated programs that fetch web pages for AI products. They fall into three kinds. Search and retrieval crawlers, such as OAI-SearchBot, PerplexityBot and Claude-SearchBot, collect pages so a product can surface and link sites in its search results. Training crawlers, such as GPTBot, ClaudeBot and CCBot, collect content that may be used to train models. Some names you will see in robots.txt, namely Google-Extended and Applebot-Extended, are controls and not crawlers. They are tokens you name in robots.txt to state a training preference. Google describes Google-Extended as a standalone product token, and Apple says Applebot-Extended does not crawl.

The third kind is user-triggered fetchers: ChatGPT-User, Claude-User and Perplexity-User. They visit a page because a person asked an assistant a question, not because a bot is sweeping the web. Because of that, owners treat robots.txt differently for them, as the table below shows. A robots.txt AI crawler opt-out is therefore a set of separate decisions: search, training and user-triggered access are not the same choice.

Search and retrieval bots: what blocking costs

Search and retrieval bots, per each owner's documentation
Bot or token	What it does	What blocking it costs
Googlebot Google	Crawls for Google Search, including Discover, all Google Search features, Google Images, Video mode and Google News.	Blocked pages can't be crawled for Google Search or its features, including AI Overviews. A blocked URL may still show without a snippet.
Bingbot Microsoft	Bing's standard crawler. robots.txt controls how Bing crawlers interact with a site.	Blocking it stops Bing's standard crawler from crawling the paths you disallow, so Bing cannot read that content.
OAI-SearchBot OpenAI	Surfaces sites in ChatGPT search features.	Sites opted out are not shown in ChatGPT search answers, though they can still appear as navigational links.
PerplexityBot Perplexity	Surfaces and links sites in Perplexity search results. It is not used to crawl content for AI foundation models.	Blocking it stops the crawler Perplexity uses to surface and link sites in its search results. It isn't a training opt-out.
Claude-SearchBot Anthropic	Improves the quality of Claude's search results.	Anthropic says disabling it may reduce your visibility in Claude's search results.
Applebot Apple	Powers search in Spotlight, Siri and Safari. Its data may also be used to train Apple foundation models.	Blocking it affects search in Spotlight, Siri and Safari. If robots.txt omits Applebot but names Googlebot, it follows Googlebot rules.

Training bots and controls: what blocking costs

Training bots and training controls, per each owner's documentation
Bot or token	What it does	What blocking it costs
GPTBot OpenAI	Crawls content that may be used to train OpenAI's generative AI foundation models.	Disallowing it indicates your content should not be used for training. It is independent of the OAI-SearchBot setting.
ClaudeBot Anthropic	Collects content that could contribute to model training.	Restricting it signals that future material from your site should be excluded from training data.
CCBot Common Crawl	Builds Common Crawl's open repository of web crawl data that anyone can access and analyse.	Blocking it keeps CCBot from adding your pages to Common Crawl's open repository. Some crawlers falsely identify as CCBot.
Bytespider Commonly attributed to ByteDance	We couldn't find first-party documentation for it, so we treat it as an undocumented crawler.	If you don't want it, block it by name.
Google-Extended Google	A standalone product token, not a separate crawler. It controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI.	Blocking it does not affect Google Search or AI Overviews. It is not a ranking signal. AI Overviews follow Googlebot rules.
Applebot-Extended Apple	Does not crawl. Disallowing it opts content out of training Apple's foundation models.	Pages that disallow it can still appear in Apple search results.

User-triggered fetchers

User-triggered fetchers, per each owner's documentation
Bot or token	What it does	What blocking it costs
ChatGPT-User OpenAI	Visits pages when a user asks ChatGPT or a Custom GPT something. It is not used to crawl the web automatically or to decide what appears in search.	OpenAI says robots.txt rules may not apply, because the user started the action.
Claude-User Anthropic	Fetches pages when a person asks Claude a question.	Anthropic says its bots honour robots.txt. Disabling it stops Claude retrieving your content for those user requests.
Perplexity-User Perplexity	Fetches pages when a user asks Perplexity a question. It is not used for web crawling or training.	Because a user requested the fetch, Perplexity says it generally ignores robots.txt rules.

What is GPTBot?

GPTBot is OpenAI's crawler for content that may be used to train its generative AI foundation models. According to OpenAI's bot documentation, disallowing GPTBot indicates that your content should not be used for training. It is separate from OAI-SearchBot, which is for search and surfaces sites in ChatGPT search features, and from ChatGPT-User, which visits a page when a user asks ChatGPT or a Custom GPT something. Each setting is independent, so you can allow OAI-SearchBot and disallow GPTBot. If both are allowed, OpenAI may use one crawl for both.

A clean example robots.txt

The example below allows the search bots and opts out of the training bots, with each bot in its own group. You can copy it and change the groups to match your own policy.

# Example: allow search crawlers, opt out of AI model training.
# Replace example.com with your domain. Each bot has its own group.

# Search and AI search crawlers
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Applebot
Allow: /

# Training crawlers and training controls
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Every other crawler
User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml

Each bot has its own group, so a change to one bot's rules cannot affect another's. Google says several user-agent lines in a row form a single group, so keep groups separate unless you want identical rules.
The Sitemap line points crawlers to your sitemap. It is not tied to any one group.
The * group covers crawlers you have not named. Google does not combine a specific group with the * group, so a named bot reads only its own rules.
Applebot-Extended is a control token, not a crawler. Disallowing it opts content out of Apple model training, and Apple says those pages can still appear in Apple search results.

Worked example: Trendi's live robots.txt

Trendi's own live robots.txt is a worked example of the opposite policy. We allow both search and training bots explicitly, by choice, because we want our public guides read widely. That is a policy choice for our site, not a recommendation for everyone.

Live file at https://trendibizz.com/robots.txt, shown verbatim on 2 November 2026:

# Trendi - trendibizz.com
# Allow major search + AI *search* crawlers. The main training crawlers are also
# allowed explicitly below; all other bots fall to the * group (Allow: /).

User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://trendibizz.com/sitemap.xml

# Optional site brief for tools that read it. Google Search ignores llms.txt
# for rankings / AI Overviews (see Search Central AI optimization guide).
# https://trendibizz.com/llms.txt

# SEO deploy 2026-10-26 - crawl files republished

The main search and training bots each have their own named group, so they don't depend on the * fallback. Bots that aren't named, such as CCBot, fall to the * group, which allows everything.
Applebot is named, so it does not fall back to Googlebot rules.
Comments in the file are ignored by crawlers. They are there for the people who maintain it.

Three parser traps

Three patterns trip up simple parsers. Here is how RFC 9309 and Google read each one, and our checker reads them the same way.

1. Stacked user-agent lines

User-agent: GPTBot
User-agent: CCBot
Disallow: /

Correct reading: This is one group that applies to both GPTBot and CCBot, so both are disallowed. The rule doesn't apply only to the last user-agent line. Google's robots.txt specification says several user-agent lines in a row form one group.

2. Duplicate groups for the same bot

User-agent: GPTBot
Disallow: /

User-agent: GPTBot
Allow: /blog/

Correct reading: Groups that match the same user-agent are combined, and RFC 9309 says they MUST be. Google also merges them into one group. GPTBot is then disallowed everywhere except paths under /blog/, because the longest matching path wins.

3. Allow: /$ with Disallow: /

User-agent: *
Disallow: /
Allow: /$

Correct reading: The homepage is allowed and every other URL is disallowed. For the homepage both rules match and Allow: /$ is the more specific one, so it wins. Every other URL matches only Disallow: /. Google's specification gives this exact case.

Checks before and after you publish

Save the file as UTF-8 plain text. LF line endings are recommended, and Google also accepts CR/LF. A byte order mark is not needed, since Google ignores it along with invalid lines.
Place the file at the root of each host and protocol. Keep it under 500 KiB, because content after that limit is ignored.
If you want a page kept out of search results, use noindex and leave the page crawlable. If robots.txt blocks the page, Google never sees the noindex, and Google does not support noindex inside robots.txt.
A disallowed page's content can't be indexed, but its URL may still be indexed and shown without a snippet.
Use the Search Console robots.txt report to see the files Google found, when they were last crawled, and any warnings or errors. It works for Domain properties and for URL-prefix properties without a path, shows versions from the last 30 days, and lets you request a recrawl. To test one URL, use the URL Inspection tool.
Allow time after a change. OpenAI says changes can take about 24 hours to reach its search systems, and Perplexity says changes can take up to 24 hours to apply.

When you have written your file, run it through the checker for robots.txt AI crawlers to see how each bot reads it. The checker lists 16 bots: 14 AI crawlers plus Googlebot and Bingbot.

FAQ

Is ChatGPT a web crawler?

ChatGPT itself is a product, and OpenAI uses separate bots with separate jobs. GPTBot crawls content that may be used for training, and OAI-SearchBot is for search. ChatGPT-User visits a page when a user asks ChatGPT or a Custom GPT something, and OpenAI says it is not used to crawl the web automatically.

Why block AI crawlers?

The usual reason is a wish to opt out of having your content used to train models. Training bots such as GPTBot, ClaudeBot and CCBot can be restricted without touching the search bots. Each block has a cost described by its owner, so decide bot by bot and not with a single blanket rule.

How does GPTBot work?

GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing it in robots.txt indicates that your content should not be used for training. Its setting is independent of OAI-SearchBot, and if both are allowed, OpenAI may use one crawl for both.

Does blocking Google-Extended remove my site from Google Search?

No. Google-Extended is a standalone product token that controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It does not affect a site's inclusion in Google Search and is not a ranking signal. AI Overviews are a Google Search feature, so they follow Googlebot rules and not Google-Extended.

Can robots.txt keep a page out of Google?

Not reliably. robots.txt manages crawler traffic, and a disallowed page's URL may still be indexed and shown without a snippet. To keep a page out of results, use noindex and leave the page crawlable so Google can see it.