Generator
Starting point for your kind of siteThe choices below are for common website builders, also called a CMS (content management system), the software a site is built with. Choose Blank to start with no paths.
A path is the part of a page address after the site name, such as /private/. The * means every crawler.
Training crawlers collect pages to teach artificial intelligence models, and they start on Block. Search crawlers and crawlers that visit when a person asks start on Allow. Untick a crawler to leave it out of the file.
Type the crawler's name (its user agent), choose Allow or Disallow, and type a path.
Checker
The checker reads the text above, including any edits you make by hand, and lists problems and warnings.
How it works and when to use it
This tool builds a robots.txt file from a starting point for your site type, any extra paths, and an Allow or Block choice for each 2026 artificial intelligence crawler. The text updates as you change options. Copy it, or download robots.txt, and place it at the top level of your domain so that it can be reached at https://yourdomain.com/robots.txt.
Use a generator when you launch a new site, change website builder, or want to opt out of artificial intelligence training crawls without touching Googlebot. The starting points are only a start. The checkout paths of a Shopify shop and the admin paths of WordPress differ from those of a Next.js site (one built with programming code rather than a ready-made template). Review every Disallow line before you upload.
A robots.txt file is not a password. Disallow only asks polite crawlers to skip a path. Private pages still need a login, a noindex tag or real access control. The checker looks for: wrong syntax, a missing Sitemap line, an empty Disallow (which allows everything), commands it does not know, repeated user agents, and Cascading Style Sheets or JavaScript paths that can stop Google from showing your page properly.
Good to know
The Sitemap line. A Sitemap: line is optional but recommended. It does not replace sending your sitemap through Google Search Console. Use the full Hypertext Transfer Protocol Secure address of your Extensible Markup Language sitemap or sitemap index. The checker warns if the line is missing.
The WordPress starting point. Choose WordPress to fill in Disallow /wp-admin/ and /wp-includes/ plus Allow /wp-admin/admin-ajax.php. Review /wp-includes/, because Google prefers access to the Cascading Style Sheets and JavaScript files there. Upload the file to the top level of your site, or paste the rules into an search engine optimisation plugin if WordPress is already serving a robots.txt.
An empty Disallow. Disallow: with no path allows everything for that user agent. Disallow: / blocks the whole site for that crawler. The checker warns about an empty Disallow so that you do not publish an allow-everything rule by accident.
Questions and answers
Who made this?
UtilForge makes it. UtilForge is a website of free cybersecurity tools and small calculators. The About page explains why the tools were made, and the Contact page is the way to reach us.
Is this a scam site?
Here are facts you can check instead of taking on trust. The tool itself does not ask for payment, an account or card details. The box near the bottom of the page offers the five website tools as one free file that opens offline. The page also carries a security setting that stops the browser letting it send anything off your device. To test it, open the page, switch your internet off and change the options: the file text and the checker keep working, because they run on your device. You do not have to take UtilForge's word for any of this.
What happens when I close the page?
The page does not save anything in your browser. It uses no cookies and no stored settings, and the page address does not change to hold what you choose, so there is no share link. When you close the page, your choices and any text you edited by hand are not kept by it. Only what you chose to take away stays: a file you downloaded stays in your downloads folder, and text you copied stays wherever you pasted it. The file stays on your device, so UtilForge holds no copy.
Does it cost anything, and do I need to sign in?
The tool is free and you can start straight away. Nothing on the page asks for a card, and nothing on it costs money. The five website tools are also available as one free file that opens offline.
Does robots.txt stop artificial intelligence companies using my website?
A robots.txt file asks crawlers to skip certain pages, and well-behaved crawlers follow it. It is voluntary, so it is not a lock: it does not hide a page from people, or from a crawler that ignores it. Blocking artificial intelligence training crawlers such as GPTBot, Google-Extended or Bytespider can limit how your content is used to train artificial intelligence models. Search-focused crawlers such as OAI-SearchBot or ChatGPT-User are optional: allow them if you want to be cited in artificial intelligence answers.
Will blocking artificial intelligence bots hurt my Google ranking?
Blocking training or artificial intelligence answer crawlers does not, by itself, change your Google Search ranking. Googlebot, the crawler for Google Search, is separate from Google-Extended, which controls whether your content may be used to train Gemini and related Google artificial intelligence products. This tool lists Google-Extended, not Googlebot. Do not block Googlebot, and do not block the Cascading Style Sheets and JavaScript files Google needs to show your page. The checker flags paths that commonly do this.
How do I block artificial intelligence training bots but keep Google Search?
Use separate rules. Leave Googlebot allowed, and Disallow training crawlers such as GPTBot or Google-Extended if that matches your policy. Community threads on r/SEO and r/TechSEO often treat Googlebot and Google-Extended as separate decisions. Check Google's current documents before you publish claims about what each one covers.
Will blocking GPTBot hurt my Google rankings?
Search crawlers are separate from many training crawlers. Blocking a training crawler is not, by itself, a Google ranking penalty. The community view is to avoid blocking Googlebot across a whole site unless you mean to leave Search.
Where should robots.txt live on WordPress?
At the top level of the site: https://yoursite.com/robots.txt. On WordPress the file is often virtual, which means the website builder or an search engine optimisation plugin serves it and there is no real file. Check your plugin settings so you do not keep two files that disagree.
Is robots.txt a security wall?
No. It is a voluntary request about crawling. Pages listed under Disallow can still be fetched by crawlers that ignore the file, or opened by people who know the address. Do not hide secrets behind robots.txt alone.