Free tool
robots.txt generator and tester
Create your site’s robots.txt in a minute, with a clear policy for AI crawlers, and check which URLs each crawler can fetch. We flag the mistakes that can take you out of Google.
Google’s rules apply: the most specific group for the crawler and, within it, the longest rule; on a tie Allow wins. Other crawlers may read some cases differently.
Would you rather we did it for you?
We run your site’s SEO every month: content, structured data, links and the technical side.
More free tools
AI visibility checker →
Whether ChatGPT, Perplexity, Gemini and Google can get into your site and whether your content is easy for AI to read and quote. Score from 0 to 100.
Free 10-page SEO audit →
Checks the home page and nine more: titles, descriptions, H1, canonical, hreflang, structured data, robots.txt, sitemap, broken links and speed. Score from 0 to 100.
llms.txt generator →
Create your site’s llms.txt (llmstxt.org format) from your own website: name, summary and key pages.
How to use it
- Choose the type of site, tick the areas not worth crawling and decide what to do with AI crawlers.
- Add your sitemap and, if needed, your own rules. Check the warnings: some mistakes remove pages from Google.
- Download the file and upload it to your domain root (yoursite.com/robots.txt). Then use "Test a URL" to check specific pages.
FAQ
Does robots.txt keep a page out of Google?
Not entirely. Blocking crawling does not stop Google from indexing the URL if others link to it. To keep it out, let it be crawled and use a noindex tag.
Should I block GPTBot and other AI crawlers?
It depends on your business. Blocking training crawlers does not stop assistants from citing you. Blocking those that read the web to answer (OAI-SearchBot, PerplexityBot…) removes you from those answers.
Does Google respect Crawl-delay?
No. Google ignores Crawl-delay; it adjusts crawl speed on its own based on how your server responds.
Where do I upload the file?
At the domain root: https://yoursite.com/robots.txt. Each subdomain needs its own file.