- Home
- Tools
- Website Grader
- robots.txt AI Crawler Checker
Free SEO audit · AI search ready · Search Console quick wins
robots.txt AI crawler checker: who can read your site?
Your robots.txt decides which crawlers may read your pages. Grade your page and the report lists 12 search and AI crawlers, says which are allowed or blocked and whether each is named in your file, so you can see whether you are open to the products you want.
اے آئی ایس ای او چیکر: دیکھیں کہ ChatGPT، Perplexity، Claude اور گوگل اے آئی آپ کی ویب سائٹ پڑھ سکتے ہیں یا نہیں۔
AI SEO चेकर: देखें कि ChatGPT, Perplexity, Claude और Google AI आपकी वेबसाइट पढ़ पा रहे हैं या नहीं।
- Results in seconds
- 40 checks
- AI search ready
- Search Console data stays private
- Free
Loading the Website Grader...
The grader reads only the public page you type. Your Search Console file is read in your browser and is not uploaded to FileXL.
Scores are a practical guide based on public Google guidance, not a Google score. Checks reviewed November 2026. This tool does not give legal or business advice.
How was the website grader?
One tap. That’s all it takes.
Thank you! 🎉
That means a lot to a small team. Would you share it on Trustpilot? It takes about 10 seconds and helps other people find a safe way to send files.
Thanks for being honest
What could we do better? Tap what fits, or write a line. Both are optional.
Prefer to say it publicly? Review us on Trustpilot
Thank you
We read every note and use it to make FileXL better.
Want it to count publicly too? Review us on Trustpilot
- 40scored checks
- 6areas, 100 points
- 12AI and search crawlers
- 0 bytesof Search Console data uploaded
By FileXL Team | Last reviewed | [email protected]
How do I check if GPTBot or ClaudeBot is blocked on my website?
Grade your page with the tool above. In the AI and answer engines area, a table shows 12 crawlers including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Googlebot and Bingbot, and whether your robots.txt allows each one and whether it is named explicitly.
How it works
Check your robots.txt in three steps
We read https://your-site/robots.txt and test each crawler against the rules for the page you typed.
Type your website address
Enter the address of the page you want to grade, for example example.com or a full page address. The home page is a good first choice. No sign-up is needed.
Read your score and the fix list
In about ten seconds you get a score out of 100, a grade from A to F, a score for six areas and a list called Fix these first, sorted by the points each fix wins back. Open any check to see why it matters and how to fix it, with code you can copy.
Fix, check again, then scan the site
Fix the top items, wait a few minutes and run the check again. Then press Scan more pages to look for duplicate titles, missing descriptions, thin pages and broken links across up to twelve pages.
Examples
Three common choices
Open to everything: no rules for AI bots at all, and a Sitemap line. This is the default for most sites and the simplest way to be visible in search and AI answers.
Visible in search, not used for training: add a block that disallows training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended) and leave the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) alone.
Closed to AI: disallow all of them. You will not appear in those AI products, and robots.txt is a request, not a lock, so bad bots can ignore it. Use your firewall for those.
Syntax
How the rules look
| Goal | robots.txt lines |
|---|---|
| Block GPTBot (training) | User-agent: GPTBot then Disallow: / |
| Allow ChatGPT search | User-agent: OAI-SearchBot then Allow: / |
| Block ClaudeBot (training) | User-agent: ClaudeBot then Disallow: / |
| Add your sitemap | Sitemap: https://www.example.com/sitemap.xml |
Each User-agent line starts its own group. Put the file at the root of the site and repeat it for each subdomain. Changes can take about a day to be picked up.
Limits
What this tool cannot do
Read these before you rely on a result.
- It reads one page, plus a sample of up to 12 more. It is not a full crawl of a large site, so it will not find every broken link or every duplicate page.
- It reads the HTML your server sends. Content that only appears after JavaScript runs is not seen, which is also what many crawlers see. The check Readable without JavaScript tells you when this is a problem.
- The speed section uses server response time and build habits, measured from our server. It is not a lab test of how the page feels on a phone. Where Google Lighthouse numbers are switched on, they are shown separately.
- It cannot see private pages, pages behind a login, or sites that block automated visits. Some firewalls block us; then you will see an error instead of a grade.
- The score is a guide made from rules we chose. Google does not publish a score and does not rank pages by ours. A high grade does not promise a high ranking, and a low grade does not mean you will not rank.
- It does not check backlinks, keyword rankings or competitors. Those need data from a crawler of the whole web. For rankings of your own keywords, use the Quick-Wins Finder with your Search Console data.
- The Quick-Wins numbers are estimates. They use rounded typical click rates by position and they add keywords together even when several keywords lead to one page. Use the order of the list, not the exact total.
- Rules and advice change. The checks and the sources on each page were reviewed in November 2026. Check the current Google documentation before you make big changes.
Questions
robots.txt AI Crawler Checker: frequently asked questions
Should I block GPTBot, ClaudeBot and other AI crawlers?
It is your choice, and the bots do different things. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended) collect content for model training; blocking them does not remove you from search. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) decide whether you can appear in those products; blocking them removes you. User bots (ChatGPT-User, Claude-User) fetch a page when a person asks. The grader shows each one.
Does robots.txt keep a page out of Google?
No. robots.txt controls crawling. A blocked page can still be listed if other sites link to it. To keep a page out of results, let it be crawled and use noindex, or protect it with a password.
Do I need an llms.txt file?
No. llms.txt is a community proposal for a short guide to your site for language models. It is not an official standard, and Google says AI features need nothing special. The grader reports whether you have one but gives no points, so you are not pushed to add something of unproven value.
What is AI SEO, AEO or GEO?
These names describe making your content easy for answer engines and AI search (Google AI Overviews and AI Mode, ChatGPT search, Perplexity, Claude and others) to find, read and cite. Google says there are no extra requirements for its AI features beyond normal SEO best practice, such as allowing crawling and giving a good page experience. The grader checks the parts you control: crawler access, text in the HTML, clear answer-style structure and trust signals.
How do I get cited by ChatGPT, Perplexity or Google AI?
There is no switch. What helps is the same as for search: let the right crawlers in, put the answer in plain text near the top of the page, use clear headings and short lists, show who wrote the page and when, keep facts accurate and sourced, and earn mentions on other trusted sites.
Why does it say it could not reach my site?
The most common reasons are a typing mistake in the name, a site that is down, a firewall or bot-protection service that blocks automated visits, or a server that answers very slowly. Check that the page opens in a private browser window, then allow our grader in your firewall if you use one. Our visits identify as FileXLGrader.
Do I need an XML sitemap?
It is not required, but it helps search engines find new pages and see which ones you consider important. Generate it from your CMS or SEO plugin, list it in robots.txt and submit it in Search Console. A sitemap does not guarantee that a page is indexed.
Sources
Sources and further reading
Sources marked as third-party are not official. Read the original pages for the full detail.
- OpenAI: Overview of OpenAI crawlersOpenAI documents separate robots.txt tokens: OAI-SearchBot for search, GPTBot for training, and ChatGPT-User for actions a person starts.
- Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?Anthropic documents three separate bots: ClaudeBot for training data, Claude-User for fetches a person asks for, and Claude-SearchBot for search quality.
- Google Search Central: Introduction to robots.txtOfficial explanation of robots.txt. It controls crawling, it is not a way to keep a page out of Google, and noindex is the tool for that.
- Google Search Central: AI features and your websiteOfficial page for AI Overviews and AI Mode. It says there are no extra requirements to appear and that normal SEO best practice continues to apply, including allowing crawling and a good page experience.
- sitemaps.org: Sitemap protocolThe shared XML sitemap format used by Google, Bing and others.
FileXL is not affiliated with Google, OpenAI, Anthropic, Perplexity or any other company named on this page. Product and crawler names belong to their owners and are used to describe what the tool checks.
Keep reading
Related guides
- AI SEO CheckerThe full AI readiness check.
- Technical SEO Auditrobots.txt in the wider technical audit.
- Free SEO AuditFree SEO audit.
- Website GraderWebsite Grader home.
Works well with
Other free FileXL tools
- Invoice, Quotation and Receipt MakerRun a business? Make a professional invoice or quotation with the Invoice Maker.
- Graphical Abstract MakerPublishing research? Build a journal-ready graphical abstract with the Graphical Abstract Maker.
- WebP to PNG ConverterTurn WebP images into PNG, or the other way, with the image converters.
- PDF CompressorMake your PDF downloads smaller and faster with the PDF Compressor.
About this tool
Made and checked by the FileXL Team
This tool is built and maintained by the FileXL Team, the people behind the FileXL file sharing service. We test the grader on real and broken sites, and we review the checks against Google Search Central when the guidance changes. Last reviewed November 2026.
- Questions, bugs or ideas: [email protected]
- Author: FileXL Team
Help us improve it
Something wrong, or a check missing?
This tool grows from what site owners, agencies and bloggers ask for. Tell us which check is wrong, which one is missing, or what you would like the Quick-Wins Finder to do. It takes a minute and you do not need an account. Please do not send passwords or private data.
Prefer email? Write to [email protected].
Need to send a report, a contract or large files?
FileXL sends large files free, with no sign-up. Upload your files, share the short code, and your client downloads them from the Receive page without creating an account.
Send files with FileXL I have a codeMore from FileXL
- All free tools
- AI Background Remover
- Thumbnail Maker
- CV Maker and ATS Checker
- Job Application Kit
- Invoice, Quotation and Receipt Maker
- Video to Reels and Shorts Maker
- Advanced Video to Reels Maker
- Advanced Video Editor
- Free Online Video Editor
- PDF to Word Converter
- Word to PDF Converter
- Graphical Abstract Maker
- HEIC to JPG Converter
- JFIF to JPG and PNG Converter
- WebP to PNG Converter
- PDF Compressor
- Merge PDF
- Urdu OCR
- Passport Photo Maker
- CNIC Copy Maker
- Document Scanner
- Greeting Card Maker
- Zakat Calculator
- Image Compressor
- JPG to PDF
- Bank Statement to Excel
- Bank Statement Analyzer
- Photo Resizer
- Free AEO Checker
- QR Code Generator
- Send large files free
- Receive a file with a code
- FileXL blog