Who this checker is for
You want to know whether search engines and AI crawlers are allowed to read your website, or you have just changed your robots.txt or sitemap and want a second pair of eyes before anything goes wrong. A single misplaced line in robots.txt can hide a whole site from search, and these files are rarely looked at once they are set up.
The checker is built for owners, office staff and freelancers who can open a text file but would rather not read the technical specifications. It explains what it finds in plain words.
Paste your files
Open your-domain/robots.txt in a new tab, copy everything and paste it below. For the sitemap, open the sitemap address and paste its XML. This page does not fetch anything from other websites; it only reads what you paste.
How to use it
Copy the full text of your robots.txt from your browser and paste it into the first box. The report shows each user-agent group, then a table for six common crawlers: whether each one is allowed or blocked at the root of the site, and which group of rules decided it. It also lists any Sitemap lines and flags lines that crawlers are likely to misread or ignore.
If you paste sitemap XML as well, the checker counts the addresses inside it, tells you whether it is a regular sitemap or a sitemap index, and flags addresses that are not complete, duplicates, mixed hosts, and files over the 50,000-address limit for a single sitemap.
Use "Load an example" to see a typical file: everything allowed for all crawlers except a cart and checkout area, with one AI training crawler blocked entirely.
Reading the crawler table
A few points help you make sense of the results.
- Search and AI are separate decisions. Some crawlers gather pages for search results, others collect material for training AI models. You can allow one kind and block another; the table labels what each crawler is used for, according to its operator.
- A crawler follows one group. A crawler that has its own named group follows only that group and ignores the * group. That is why adding a named group can accidentally allow or block more than you expected.
- Allowed at / is not the whole story. A crawler can be allowed at the root but blocked from particular folders. The table counts those extra blocked paths so you can review them.
- Blocking is not hiding. robots.txt asks well-behaved crawlers to stay away. It does not make a page private, and a blocked page can still appear in search if other sites link to it.
Limits worth knowing
- The checker reads only what you paste. It cannot see your live file, redirects, server errors or whether your sitemap address loads.
- It follows the published robots.txt rules that major search engines use, but individual crawlers may interpret edge cases differently.
- It reports what the rules say, not whether a particular crawler actually respects them.
Not sure what your rules should say?
When we build or move a website onto our managed hosting, the essential search setup is part of the package. If pages are missing from search, read what we check when a website is not showing on Google or tell us about your site.
How it works
1. Consultation
We learn your services, customers and goals, in English or Spanish.
2. We write and design
We draft the words and build the site around your business.
3. 10 revision rounds
You review and request changes until it is right.
4. Launch and care
We launch, then Website Care keeps it hosted, backed up, checked weekly and A/B tested.
Keep going
llms.txt generator · AI visibility check · Website not showing on Google · Redesign without losing traffic
