Is your site blocking AI assistants?

Paste your robots.txt file below. This tells you which AI bots can reach your site — and separates the ones that cite you in answers from the ones that train on your content. Most people who block AI block the wrong ones.

Prove it stays on this device

Load this page.
Turn off your Wi-Fi.
Paste and check — it still works.

Your robots.txt

Find it at yoursite.com/robots.txt. Open it in a browser, select all, copy, and paste it here. If that address shows nothing, you have no robots.txt — which usually means everything is allowed.

Paste the whole file Path to test (optional)

Result

Bot	Run by	Access	What it does

Your file, as the bots read it

Two kinds of bot, and why the difference matters

Citation bots fetch your pages so an assistant can quote and link to you right now. Block these and you cannot appear in AI answers at all.

Training bots collect content that may be used to train future models. Blocking these is a legitimate choice with no effect on whether you are cited today.

User-triggered fetchers act when a real person asks an assistant to open your page. Several operators state these may not follow robots.txt strictly, because a human initiated the request.

The common and expensive mistake is blocking all of them at once while meaning to block only training. That removes you from AI answers without anyone noticing.

robots.txt is not the only door

Your CDN or host may block AI crawlers at the network level, before the request ever reaches your site. Cloudflare, for example, has its own AI crawler controls, and in September 2026 it made Disallow AI Training the recommended default for sites that had previously used its blanket block.

If your robots.txt says allow and you still see no AI traffic, check your CDN settings too. A network-level block wins over anything written in this file.

Honest limits

This reads the file you paste. It does not fetch your site, so it cannot tell you what your server actually does, whether a bot obeys the file, or whether your CDN overrides it.

Compliance is voluntary. Some crawlers have been documented ignoring robots.txt, and any program can claim any name. Verify real traffic by IP address, not by the name in your logs.

Allowing every bot does not make you appear in AI answers. It only makes it possible. Published research finds the strongest correlations with AI citation are mentions of you elsewhere on the web, not anything in this file.

The bot list below is current as of 17 September 2026. New crawlers appear several times a year.