1. Home
  2. Tools
  3. Website Grader
  4. robots.txt AI Crawler Checker

Free SEO audit · AI search ready · Search Console quick wins

robots.txt AI crawler checker: who can read your site?

Your robots.txt decides which crawlers may read your pages. Grade your page and the report lists 12 search and AI crawlers, says which are allowed or blocked and whether each is named in your file, so you can see whether you are open to the products you want.

اے آئی ایس ای او چیکر: دیکھیں کہ ChatGPT، Perplexity، Claude اور گوگل اے آئی آپ کی ویب سائٹ پڑھ سکتے ہیں یا نہیں۔

AI SEO चेकर: देखें कि ChatGPT, Perplexity, Claude और Google AI आपकी वेबसाइट पढ़ पा रहे हैं या नहीं।

Loading the Website Grader...

The grader reads only the public page you type. Your Search Console file is read in your browser and is not uploaded to FileXL.

Scores are a practical guide based on public Google guidance, not a Google score. Checks reviewed November 2026. This tool does not give legal or business advice.

Was this useful? Your rating helps others.
Advertisement

By | Last reviewed | [email protected]

How do I check if GPTBot or ClaudeBot is blocked on my website?

Grade your page with the tool above. In the AI and answer engines area, a table shows 12 crawlers including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Googlebot and Bingbot, and whether your robots.txt allows each one and whether it is named explicitly.

How it works

Check your robots.txt in three steps

We read https://your-site/robots.txt and test each crawler against the rules for the page you typed.

Type your website address

Enter the address of the page you want to grade, for example example.com or a full page address. The home page is a good first choice. No sign-up is needed.

Read your score and the fix list

In about ten seconds you get a score out of 100, a grade from A to F, a score for six areas and a list called Fix these first, sorted by the points each fix wins back. Open any check to see why it matters and how to fix it, with code you can copy.

Fix, check again, then scan the site

Fix the top items, wait a few minutes and run the check again. Then press Scan more pages to look for duplicate titles, missing descriptions, thin pages and broken links across up to twelve pages.

Examples

Three common choices

Open to everything: no rules for AI bots at all, and a Sitemap line. This is the default for most sites and the simplest way to be visible in search and AI answers.

Visible in search, not used for training: add a block that disallows training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended) and leave the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) alone.

Closed to AI: disallow all of them. You will not appear in those AI products, and robots.txt is a request, not a lock, so bad bots can ignore it. Use your firewall for those.

Syntax

How the rules look

robots.txt examples
Goalrobots.txt lines
Block GPTBot (training)User-agent: GPTBot then Disallow: /
Allow ChatGPT searchUser-agent: OAI-SearchBot then Allow: /
Block ClaudeBot (training)User-agent: ClaudeBot then Disallow: /
Add your sitemapSitemap: https://www.example.com/sitemap.xml

Each User-agent line starts its own group. Put the file at the root of the site and repeat it for each subdomain. Changes can take about a day to be picked up.

Limits

What this tool cannot do

Read these before you rely on a result.

  • It reads one page, plus a sample of up to 12 more. It is not a full crawl of a large site, so it will not find every broken link or every duplicate page.
  • It reads the HTML your server sends. Content that only appears after JavaScript runs is not seen, which is also what many crawlers see. The check Readable without JavaScript tells you when this is a problem.
  • The speed section uses server response time and build habits, measured from our server. It is not a lab test of how the page feels on a phone. Where Google Lighthouse numbers are switched on, they are shown separately.
  • It cannot see private pages, pages behind a login, or sites that block automated visits. Some firewalls block us; then you will see an error instead of a grade.
  • The score is a guide made from rules we chose. Google does not publish a score and does not rank pages by ours. A high grade does not promise a high ranking, and a low grade does not mean you will not rank.
  • It does not check backlinks, keyword rankings or competitors. Those need data from a crawler of the whole web. For rankings of your own keywords, use the Quick-Wins Finder with your Search Console data.
  • The Quick-Wins numbers are estimates. They use rounded typical click rates by position and they add keywords together even when several keywords lead to one page. Use the order of the list, not the exact total.
  • Rules and advice change. The checks and the sources on each page were reviewed in November 2026. Check the current Google documentation before you make big changes.

Questions

robots.txt AI Crawler Checker: frequently asked questions

Should I block GPTBot, ClaudeBot and other AI crawlers?

It is your choice, and the bots do different things. Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended) collect content for model training; blocking them does not remove you from search. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot) decide whether you can appear in those products; blocking them removes you. User bots (ChatGPT-User, Claude-User) fetch a page when a person asks. The grader shows each one.

Does robots.txt keep a page out of Google?

No. robots.txt controls crawling. A blocked page can still be listed if other sites link to it. To keep a page out of results, let it be crawled and use noindex, or protect it with a password.

Do I need an llms.txt file?

No. llms.txt is a community proposal for a short guide to your site for language models. It is not an official standard, and Google says AI features need nothing special. The grader reports whether you have one but gives no points, so you are not pushed to add something of unproven value.

What is AI SEO, AEO or GEO?

These names describe making your content easy for answer engines and AI search (Google AI Overviews and AI Mode, ChatGPT search, Perplexity, Claude and others) to find, read and cite. Google says there are no extra requirements for its AI features beyond normal SEO best practice, such as allowing crawling and giving a good page experience. The grader checks the parts you control: crawler access, text in the HTML, clear answer-style structure and trust signals.

How do I get cited by ChatGPT, Perplexity or Google AI?

There is no switch. What helps is the same as for search: let the right crawlers in, put the answer in plain text near the top of the page, use clear headings and short lists, show who wrote the page and when, keep facts accurate and sourced, and earn mentions on other trusted sites.

Why does it say it could not reach my site?

The most common reasons are a typing mistake in the name, a site that is down, a firewall or bot-protection service that blocks automated visits, or a server that answers very slowly. Check that the page opens in a private browser window, then allow our grader in your firewall if you use one. Our visits identify as FileXLGrader.

Do I need an XML sitemap?

It is not required, but it helps search engines find new pages and see which ones you consider important. Generate it from your CMS or SEO plugin, list it in robots.txt and submit it in Search Console. A sitemap does not guarantee that a page is indexed.

Sources

Sources and further reading

Sources marked as third-party are not official. Read the original pages for the full detail.

FileXL is not affiliated with Google, OpenAI, Anthropic, Perplexity or any other company named on this page. Product and crawler names belong to their owners and are used to describe what the tool checks.

Keep reading

Related guides

Works well with

Other free FileXL tools

About this tool

Made and checked by the FileXL Team

This tool is built and maintained by the , the people behind the FileXL file sharing service. We test the grader on real and broken sites, and we review the checks against Google Search Central when the guidance changes. Last reviewed November 2026.

Help us improve it

Something wrong, or a check missing?

This tool grows from what site owners, agencies and bloggers ask for. Tell us which check is wrong, which one is missing, or what you would like the Quick-Wins Finder to do. It takes a minute and you do not need an account. Please do not send passwords or private data.

Prefer email? Write to [email protected].

Need to send a report, a contract or large files?

FileXL sends large files free, with no sign-up. Upload your files, share the short code, and your client downloads them from the Receive page without creating an account.

Send files with FileXL I have a code