Your web host or Cloudflare dashboard has a new button that says something like "Block AI bots." Or you read that AI companies are copying websites to train their chatbots, and it felt wrong. Or a customer told you they found you through ChatGPT, and now you're wondering whether blocking would cut that off.

All three are reasonable reactions. The trouble is that "AI bots" covers several different programs doing different jobs, and blocking the wrong one costs you customers without protecting much.

The quick answer

For most small service businesses (a plumber, an accountant, an IT shop), the answer is block training if it bothers you, but keep the search bots allowed:

Leave Googlebot alone. Blocking it removes you from Google Search entirely, AI Overviews included.
Allow the AI search bots (OpenAI's OAI-SearchBot, PerplexityBot and similar) if you want to be named in AI answers.
Block the training bots (GPTBot, Google-Extended, ClaudeBot, CCBot) only if you'd rather your pages weren't used to train AI models. It won't hurt your Google rankings or keep you out of AI Overviews.
Check what your host or Cloudflare already does, because a default setting may have made this decision for you.

What "AI bots" actually do

A bot (or crawler) is a program that reads web pages automatically. AI companies run three kinds, and they matter differently to a small business:

Training bots collect pages that may be used to train future AI models. OpenAI's crawler documentation says blocking its GPTBot "indicates a site's content should not be used in training generative AI foundation models." OpenAI treats it separately from its search bot, so blocking it doesn't keep you out of ChatGPT search.
Search bots find pages so an AI tool can show and link them in answers. OpenAI says sites that block OAI-SearchBot "will not be shown in ChatGPT search answers." Perplexity says PerplexityBot "is designed to surface and link websites in search results" and isn't used to train AI models (Perplexity's crawler docs). These are the ones that send you customers.
User-triggered fetchers visit a page because a person asked the AI to look at it right now, like ChatGPT-User and Perplexity-User. Both companies say robots.txt may not apply to these, because a human started the request.

Google works differently. There's no separate Google AI search bot: Google's AI features documentation says AI Overviews and AI Mode use normal Google Search, so the regular Googlebot rules are the control. Google-Extended is a separate robots.txt setting that controls whether your content trains Gemini models. Google's crawler documentation says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." One catch: it also covers using your pages to ground answers in the Gemini app, so blocking it can leave you out of those answers.

We list every bot name and what it controls in ChatGPT or Google AI Doesn't Mention My Business: What to Check.

What blocking costs you, and what it gets you

What you give up by blocking search bots: AI tools answer questions like "who does IT support near me?" with a short list of businesses and links. Block the bot that builds that list and you're left off it. Your competitors aren't.

What you get by blocking training bots: a clear statement that you don't consent to your writing being used to train AI models. That's a legitimate choice, especially if your content is your product.

What you don't get: a guarantee. Google's robots.txt guide says the rules can't be enforced: "Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not." Cloudflare's documentation says the same: "robots.txt compliance is voluntary." It's a polite sign on the door, not a lock.

How to check what your site does now

Step 1: Read your robots.txt

Type your domain followed by /robots.txt into a browser, for example yourbusiness.com/robots.txt. It's a plain text file. Look for lines like these:

User-agent: GPTBot
Disallow: /

That pair means "GPTBot, stay out of the whole site." A Disallow: / under User-agent: * blocks every bot that follows the rules, including Google. If you see that, fix it today.

Step 2: Check Cloudflare (if you use it)

Cloudflare can write robots.txt rules for you and block bots at its network. As of October 2026, the setting lives on the Security settings page: filter by Bot traffic, then select Configure AI bot policies (Cloudflare's steps). There you choose separately what happens to training, search and agent bots.

If the managed robots.txt is on, Cloudflare adds its own block to the top of your file. Its example blocks training crawlers like GPTBot, ClaudeBot, Google-Extended and CCBot, and says search=yes, ai-train=no. That's the middle position from the quick answer, and fine for most businesses. Just make sure it's what you meant.

To see which AI bots are actually visiting, and which ignore your rules, Cloudflare's AI Crawl Control dashboard is available on every plan, including the free one. That's also where you turn a polite request into an actual block.

Step 3: Check Google's own switch

Google rolled out a per-site control in Search Console to every site in August 2026. Go to Settings > Search generative AI. Google's help page says the default is "Include my site's links and content in Search generative AI features." If someone switched it to Exclude, your site won't appear in AI Overviews or AI Mode, and Google says "You won't receive any traffic or impressions from these features." It doesn't affect your regular search rankings.

Step 4: Check plugins and firewalls

Security plugins, hosting firewalls and "bot protection" add-ons sometimes block AI crawlers by default. If your robots.txt looks fine but AI tools still can't see your site, this is the next place to look.

Content Signals: a newer way to say what you allow

You may see lines like this at the top of a robots.txt file:

Content-Signal: search=yes, ai-input=yes, ai-train=no

These are Content Signals, a format Cloudflare supports. They split permission into three uses: search (building a search index and showing links), ai-input (using your pages to write an AI answer right now) and ai-train (training models). Like the rest of robots.txt, they state your wishes rather than enforce them. Cloudflare notes that Google Search Console may flag the line as "Syntax not understood," and says it has seen no effect on crawling or SEO from that.

For what it's worth, our own site allows all three. We write to be found and quoted, so being read by AI tools is the point.

When blocking AI bots does make sense

Your content is the product: paid courses, members-only material, original research or photos you license.
Bots are straining a small hosting plan. Check your host's traffic stats first; Cloudflare's AI Crawl Control shows which crawlers are hitting you.
Pages you never wanted public at all. robots.txt won't protect those, since anyone can still open them. Put them behind a login.

Preventing surprises in a business

Give one person the job. Whoever manages the website owns robots.txt, the Cloudflare bot settings and the Search Console switch.
Re-check after any website rebuild, host move or new security plugin. Those are the moments a blanket "block everything" rule slips in.
Write the decision down: "We allow search bots, we block training bots," with the date. Next year you'll know it was deliberate.

When to call someone

Search Console shows pages blocked by robots.txt that should be in Google.
You need a real block (paywalled or licensed content), not a polite request.
Traffic or calls from AI tools dropped and you can't find a setting that changed.

If that's where you are, get in touch and we'll look at it with you.

Should My Small Business Block AI Bots From Its Website?

The quick answer

What "AI bots" actually do

What blocking costs you, and what it gets you