Does a Disallow hide a page?

No. robots.txt asks a polite crawler not to fetch a path. The URL can still be linked, shared, and indexed. Do not use it as a lock on private content.

What if robots.txt is missing?

A 404 means there is no file. Crawlers treat that as allow all. Add a file if you want a Sitemap line or a rule for a specific agent.

Why flag a relative Sitemap?

The robots.txt spec requires each Sitemap value to be an absolute URL. A path such as /sitemap.xml is not a valid Sitemap line.

Which mistakes matter most?

A response that is HTML instead of a robots file, User-agent * with Disallow /, and a Sitemap value that is not absolute. Those three change what crawlers are allowed to fetch or find.

robots.txt checker

Read allow and disallow rules, sitemap links, and the mistakes that make a robots.txt file lie.

In A public domain, a robots.txt URL, or a pasted robots.txt file./Out User-agent groups, sitemap links, and pass, warn, or fail rows for common mistakes.

Fetches the public URL

Input

Domain or robots.txt URL

A bare domain is fetched at /robots.txt.

Output

Rules show up here

Allow and Disallow paths, sitemap links, and the mistakes that change what crawlers fetch.

How to use it

1Enter a domain or a robots.txt URLA bare domain is fetched at /robots.txt. A URL that already ends in robots.txt is fetched as written.
2Read the groupsEach User-agent lists Allow and Disallow paths. Sitemap lines are listed separately. You can also paste a file and audit it locally.
3Fix the flagged rowsStart with fails: a file that is HTML, a User-agent * that disallows the whole site, and a Sitemap that is not an absolute URL.