Search Results
All Results

Robots.txt Tool

Verify existing robots.txt files or generate new ones in seconds.

User-agent: *

Frequently Asked Questions

What is robots.txt?

It is a text file placed at the root of your website that instructs search engine crawlers (like Googlebot) which pages they can or cannot access.

Why do I need it?

It prevents search engines from indexing private pages (like admin panels) and helps manage server load by blocking unwanted bots.

What is User-agent: *?

The asterisk (*) is a wildcard that represents "All Robots". Rules following this line apply to every crawler unless specified otherwise.

Does Disallow hide pages from Google?

Not entirely. It stops Google from crawling the content, but the URL might still appear in search results if linked elsewhere. Use "noindex" meta tags for full removal.

What is Crawl-Delay?

It tells bots to wait a certain number of seconds between requests to avoid overloading your server. Note: Google ignores this directive.

Where should I put this file?

It must be placed in the **root directory** of your domain (e.g., `https://yoursite.com/robots.txt`). Subfolders won't work.

Is this tool free?

Yes, both the checker and generator are 100% free and unlimited.

How do I add a Sitemap?

Simply add `Sitemap: https://yoursite.com/sitemap.xml` at the bottom of your robots.txt file to help bots discover all your pages.