Category
AI search
Files and checks that decide what an AI crawler can fetch, cite, and skip.
Tools
3
Read a public sitemap and write an llms.txt index, plus an optional llms-full.txt.
Open instrumentScore crawler access, llms.txt, structured data, and the on-page signals AI search uses.
Open instrumentRead allow and disallow rules, sitemap links, and the mistakes that make a robots.txt file lie.
Open instrument