UtilForge
Free · runs on this device

Is your site blocking artificial intelligence assistants?

100% freeNo sign-upsNo trackingKeeps working with the internet off

Most websites keep a small public text file called robots.txt. It tells the programs that visit the site, called crawlers or bots, which pages they may read and which to leave alone. Artificial intelligence companies run crawlers too. Paste your file below to see which artificial intelligence crawlers it lets in, and whether it blocks the ones that quote you in answers or only the ones that train on your content. A common mistake is blocking the wrong ones.

The short answer. Possibly, and the file that decides is robots.txt. Paste yours here and this page lists each artificial intelligence crawler as allowed or blocked, in plain words. It reads only the text you paste. Your host or a service in front of your site can also block crawlers before robots.txt is read, so check those settings too.

  • Nothing is uploaded
  • Starts straight away
  • Keeps working with the internet off

How it works, in four steps

  1. Paste or choosePaste your website’s robots.txt file.
  2. The check runs on your own phone or computerWhat you add stays on your device, and the page keeps working with the internet off.
  3. Read what it found and what to do nextSee which artificial intelligence companies may read your pages, and what to change.
  4. What this check cannot doIt does not visit your website, so it cannot tell whether a crawler obeys the file or what your server really does.

Your robots.txt

Find it at yoursite.com/robots.txt. Open that address in a browser, select all, copy, and paste it here. If the address shows nothing, your site has no robots.txt, which usually means everything is allowed.

This is the part of a web address after the site name, called the path, such as /blog/. Leave it as / to test the whole site.

What are robots.txt and a crawler?

A crawler, also called a bot, is a program that visits web pages by itself, one after another, and reads them. Search engines use crawlers to build their search results. Artificial intelligence companies use them too, either to find pages an artificial intelligence assistant can quote or to collect text that may be used to train an artificial intelligence model.

The robots.txt file is a list of rules for those crawlers, kept at a fixed address on your site. Each rule names a crawler on a User-agent line, which is the name the crawler gives when it visits. A Disallow line then says which paths it should stay out of, and an Allow line says which it may visit.

The file is a request, not a lock. A crawler that follows the rules stays away, and the section on limits below says what that means in practice.

Three kinds of bot

Citation

Citation bots

They fetch your pages so an artificial intelligence assistant can quote you and link to you right now. Block these and you cannot appear in those artificial intelligence answers at all.

Training

Training bots

They collect content that may be used to train future artificial intelligence models. Blocking these is a legitimate choice, and it does not change whether you are cited today.

On request

User-triggered fetchers

They act when a real person asks an artificial intelligence assistant to open your page. Several operators state that these may not follow robots.txt strictly, because a person started the request.

A common and costly mistake is blocking all three kinds at once when the aim was to block only training. That removes you from artificial intelligence answers, and nothing tells you it has happened.

The list of bots follows the documentation published by OpenAI, Anthropic, Google, Apple, Perplexity and Meta. Amazon, Common Crawl and ByteDance are listed by name only. Companies add and rename bots, so check the operator’s own page before relying on a rule.

Check that it stays on this device

  1. Load this page.
  2. Turn off your Wi-Fi.
  3. Paste your file and check it. The result still appears.

robots.txt is not the only door

Your host (the company that stores your website) or your content delivery network (a service that sits in front of a site and passes its pages to visitors) may block artificial intelligence crawlers at the network level, before the request ever reaches your site. Cloudflare, for example, has its own artificial intelligence crawler controls. From 15 September 2026 it sorts artificial intelligence crawlers into three kinds (Search, Training and Agent), moves sites that had used its blanket Block artificial intelligence bots switch over to Disallow Artificial Intelligence Training, and sets starting choices for new sites. Its own announcement is “Have it both ways”.

If your robots.txt says allow and you still see no artificial intelligence visits, check your content delivery network settings too. A network-level block wins over anything written in this file.

What this tool cannot tell you

This tool reads the file you paste. It does not visit your site, so it cannot tell you what your server really does, whether a crawler obeys the file, or whether your content delivery network overrides it.

Obeying robots.txt is voluntary. Some crawlers have been documented ignoring it, and any program can use any crawler’s name. Check real visits by IP address (the number that identifies the computer making the request), not by the name shown in your logs.

Allowing every crawler does not put you in artificial intelligence answers. It only makes that possible. Published research finds that the strongest links with being cited by artificial intelligence are mentions of you elsewhere on the web, not anything in this file.

The bot list on this page is a starting point and not a complete register. New crawlers appear several times a year, so check the crawler page of each company before relying on a name.

Questions people ask

Who made this?

UtilForge makes it. UtilForge is a website of free cybersecurity tools. The About page explains why the tools were made, and the Contact page is the way to reach us.

Is this a scam site?

You do not have to take our word for it, because you can check. This page does not ask for payment, a sign-in or card details. It is set so that your browser stops the page sending anything off your phone or computer, and you can test that: let the page finish loading, switch the internet off, then use it. It keeps working, because it has nowhere to send anything. If you are still unsure about any website, do not type anything private into it. UtilForge also sells optional packs of tools through Gumroad, and nothing on this page needs one.

What happens when I close the page?

The robots.txt text you pasted and the results are gone. It saves nothing in your browser and puts nothing in the web address of the page. The only thing that can stay is text you chose to copy with a Copy button: your phone or computer keeps it on its clipboard, the place that holds copied text, until you copy something else.

Does it cost anything, and do I need to sign in?

It is free, and you can start straight away. UtilForge also sells optional packs of tools through Gumroad, and nothing on this page needs one.

How do I see which artificial intelligence crawlers my website allows?

Paste your robots.txt file, the small text file that tells automated visitors which parts of a site they may read. The page shows which artificial intelligence crawlers your site allows and which it blocks, and it separates the bots that cite you in answers from the ones that train on your content.

What are the three kinds of artificial intelligence bot?

Citation bots fetch your pages so an assistant can quote and link to you right now. Training bots collect content that may be used to train future models. User-triggered fetchers act when a real person asks an assistant to open your page, and several operators state these may not follow robots.txt strictly.

Does it look at my website?

No. It reads only the file you paste. It cannot tell you what your server actually does, whether a bot obeys the file, or whether your content delivery network overrides it.

My robots.txt allows artificial intelligence bots but I see no artificial intelligence traffic. Why?

Your content delivery network or host may block artificial intelligence crawlers at the network level, before the request reaches your site. A network-level block wins over anything written in the file, so check your content delivery network settings too.

Will allowing every bot get my site into artificial intelligence answers?

No. It only makes it possible. The page cites published research that finds the strongest links with artificial intelligence citation are mentions of you elsewhere on the web, not anything in this file.

Did this check help you?

Tell UtilForge what worked and what did not. Each button opens an email to lindsay@utilforge.org that you write and send yourself. UtilForge will only quote your words on this site if you say it may, and then with your first name and country only.

Yes, it helpedSomething was missing

The whole method in one book. “How to Become Safer Online” is a free large-print guide to the five steps most scams follow, and what to do if money or details have already gone. Read or print the free book.