Tagorising
Tools

Free tool

robots.txt generator and tester

Create your site’s robots.txt in a minute, with a clear policy for AI crawlers, and check which URLs each crawler can fetch. We flag the mistakes that can take you out of Google.

Base

AI crawlers

Googlebot and Bingbot are never blocked: without them you would disappear from Google and Bing. Google’s AI answers (AI Overviews) cannot be blocked separately from search.

Own rules

Crawler "*" = all. Use "*" as a wildcard in paths and "$" for the end of the URL.

How to use it

  1. Choose the type of site, tick the areas not worth crawling and decide what to do with AI crawlers.
  2. Add your sitemap and, if needed, your own rules. Check the warnings: some mistakes remove pages from Google.
  3. Download the file and upload it to your domain root (yoursite.com/robots.txt). Then use "Test a URL" to check specific pages.

FAQ

Does robots.txt keep a page out of Google?

Not entirely. Blocking crawling does not stop Google from indexing the URL if others link to it. To keep it out, let it be crawled and use a noindex tag.

Should I block GPTBot and other AI crawlers?

It depends on your business. Blocking training crawlers does not stop assistants from citing you. Blocking those that read the web to answer (OAI-SearchBot, PerplexityBot…) removes you from those answers.

Does Google respect Crawl-delay?

No. Google ignores Crawl-delay; it adjusts crawl speed on its own based on how your server responds.

Where do I upload the file?

At the domain root: https://yoursite.com/robots.txt. Each subdomain needs its own file.

CONTACT

Let's talk

Choose how you'd prefer to connect. The first conversation is direct and commitment-free.