What is robots.txt?

robots.txt is a plain text file at the root of your website, such as https://example.com/robots.txt, that tells crawlers which URLs they may or may not crawl. It is a request that well-behaved crawlers follow, not a security feature, and blocking a URL from crawling does not reliably keep it out of Google's index.

Why it matters

The file lets you steer crawlers away from areas that waste their time, such as internal search results or admin screens, and point them to your sitemap. A single wrong line can also hide your whole site from search engines, which is why it deserves a careful look before launch.

How to check

Open https://www.example.com/robots.txt in a browser. It should load as plain text with status 200. A site with no file at all is allowed to be crawled everywhere.
Read the groups of rules. Each group starts with a User-agent line naming the crawler, followed by Disallow and Allow lines. A Sitemap line gives the address of your sitemap.

User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /admin/help/

Sitemap: https://www.example.com/sitemap.xml

Check the rules against your real addresses. Disallow: /admin/ blocks every address that begins with /admin/, and paths are case sensitive.
Make sure you do not see Disallow: / under User-agent: *, unless you truly want the whole site blocked. This line is often left over from a staging copy.
Use the robots.txt report in Google Search Console to see how Google fetched and read the file, and the URL Inspection tool to test a single page.
Keep the file small. Google reads only the first 500 KiB of a robots.txt file.

Each host and protocol has its own file, so the www and non-www versions, and a subdomain, each need their own.

Crawvix's scan checks that robots.txt is reachable and that it does not block Googlebot, Bingbot or your pages.

Common mistakes

Leaving Disallow: / in place after launch.
Using robots.txt to hide private files. The file is public and lists exactly what you wanted to hide. Protect private files with a login.
Blocking the CSS or JavaScript files that Google needs to display your pages.
Expecting a Disallow rule to remove a page that is already in search results.
Placing the file in a subfolder, where crawlers do not look for it.

Frequently asked questions

Is robots.txt required?

No. If the file is missing, crawlers assume they may visit everything. You only need one when you want to restrict areas or declare a sitemap.

Does robots.txt stop a page being indexed?

Not reliably. It stops crawling, but a blocked address can still appear in results if other sites link to it. To keep a page out of search, use a noindex instruction on a crawlable page.

Can I block only some crawlers?

Yes. Write a separate group under the User-agent name of that crawler. A crawler follows the most specific group that matches its name.

To check your own robots.txt and sitemap, Try the free website sample.