AI crawlers now make up a meaningful share of bot traffic on most business websites, and your robots.txt is the policy document that decides what they get. Block them all and you disappear from AI answers. Allow everything and you hand over content with no say in the matter. The right move is a per-bot policy, and it takes about an hour to set up.
Key takeaways
- AI crawlers fall into two camps: training bots and retrieval bots. They deserve different policies.
- Blocking retrieval bots removes you from AI answers, which is where a growing share of buyers ask questions.
- Your server logs already show which bots visit and how often. Check before you write policy.
- Robots.txt is a request, not a lock. Reputable bots honor it; bad actors never did.
Training bots vs. retrieval bots: which AI crawlers do what?
The distinction that matters is what the crawler does with your pages. Training bots like GPTBot and ClaudeBot collect content to train future models. Retrieval bots like ChatGPT-User, OAI-SearchBot, and PerplexityBot fetch pages in real time to answer a specific question, often with a citation and a link back to you.
That difference changes the calculus. A citation from a retrieval bot is a referral channel. Training access is a donation to a model that may never send anyone back. Plenty of publishers allow retrieval and block training, and that split policy is legitimate. Google complicates it by using one crawler for search and AI features together, which its crawler documentation spells out, so blocking Googlebot was never really on the table.
Read your logs before you write your policy
Policy written without data is guessing. Pull thirty days of server logs and count requests by user agent. You’ll typically find a handful of AI bots you recognize, a few you don’t, and occasionally one hammering a faceted URL pattern that deserves a block for load reasons alone.
When we audit sites as part of our technical SEO work, the log review regularly surprises owners. Small business sites that “nobody visits” often log thousands of AI bot requests a month. Those bots are building a picture of your business whether you participate or not. The question is whether the picture is accurate, which is a structured data problem as much as a crawling one.
A sane default policy
Every business is different, but this default serves most companies that sell things:
- Allow retrieval bots. Citations in AI answers are distribution, and you want them.
- Decide on training bots deliberately. Publishers with content as the product often block; service businesses that want AI systems to know them often allow.
- Block any bot that ignores crawl-delay and hurts performance, whatever its purpose.
- Document the policy in robots.txt with comments so future-you knows why each rule exists.
Remember what robots.txt is: a published request that well-behaved crawlers follow, formalized in RFC 9309. It is not access control. Anything truly private belongs behind authentication, not behind a polite note.
FAQ
Does blocking GPTBot remove us from ChatGPT answers?
Not by itself. GPTBot gathers training data. ChatGPT’s browsing and search features use separate user agents, so you can block training and still appear in answers. Check each bot’s documentation before assuming one rule covers everything.
Do AI crawlers waste crawl budget?
They consume server resources, and on large sites with messy URL structures that can matter. For most sites under fifty thousand pages, the real crawl budget problems are duplicate URLs and redirect chains, not AI bots.
Can you set this up for us?
Yes. Log review, per-bot policy, and a documented robots.txt ship with every technical SEO engagement. Book a strategy call and bring your current robots.txt if you have one. We’ve seen some wild ones.