How do I check if my robots.txt is valid?

Paste your robots.txt content into the validator above. It parses every User-agent block, flags syntax problems (unknown directives, rules before any User-agent, empty values), and gives a verdict of Valid, Valid with warnings, or Invalid — plus a per-bot table showing which of 25 crawlers are blocked.

How can I test my robots.txt before publishing?

Copy the draft file content into the "Robots.txt content" field and run the tool. Review the warnings (e.g. a site-wide Disallow: / under "*" that would block indexing), then publish the corrected file at yoursite.com/robots.txt and re-check it live in Google Search Console's robots.txt tester.

How do I block AI crawlers like GPTBot in robots.txt?

Add a dedicated block for each AI crawler: "User-agent: GPTBot" followed by "Disallow: /" (same for ClaudeBot, PerplexityBot, CCBot, Google-Extended). Paste the file into this validator to confirm each bot shows as Blocked — note this is a request, not enforcement: compliant bots obey it, scrapers may not.

Why is my site not indexed — could robots.txt be the cause?

A "Disallow: /" under "User-agent: *" blocks every crawler from your whole site and is the most common self-inflicted deindexing. Run your robots.txt through the validator — it explicitly warns about this case. If the file is clean, check noindex tags and Search Console coverage next.

What does "Disallow:" with nothing after it mean?

An empty Disallow means "disallow nothing" — i.e. allow everything for that group. It is valid per the spec, but this parser does not track it in the rule breakdown, so the validator warns you about it to avoid confusion.

Does robots.txt support Crawl-delay?

Crawl-delay is not part of the official standard; Google ignores it, while Bing and some others honor it. This validator flags Crawl-delay as "not interpreted" because the engine only checks User-agent, Allow, Disallow, and Sitemap — set crawl rate in Search Console or Bing Webmaster Tools instead of relying on the directive alone.

Robots.txt Validator

What is Robots.txt validator?

Robots.txt Validator — Validate your robots.txt free: syntax checks, per-user-agent rule breakdown, plus allow/disallow verdicts for 25 crawlers including AI bots.

This tool validates robots.txt files with the open-source robotstxt-ai parser (sharozdawa/robotstxt-ai, MIT), vendored unchanged into this tool. The engine parses each User-agent block into Allow/Disallow rules and evaluates 25 tracked crawlers — Googlebot, Bingbot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot, Google-Extended, and more — reporting which are blocked by a "Disallow: /" match. A line-level audit adds honest warnings: unrecognized directives (only User-agent, Allow, Disallow, and Sitemap are interpreted), rules placed before any User-agent block, empty Disallow lines, duplicate groups, a site-wide "Disallow: /" under "*", and a missing Sitemap line. Limitations: the engine does not model longest-match precedence or Allow-over-Disallow tie-breaking; verdicts mean "matches Disallow: /", not a full crawl simulation. This is a syntax and intent check only — it never fetches your live file, so confirm the file is actually published at yoursite.com/robots.txt and served as text/plain.

Why use this robots.txt validator?

🎯

Built for robots.txt validator

Validate your robots.txt free: syntax checks, per-user-agent rule breakdown, plus allow/disallow verdicts for 25 crawlers including AI bots.

📋

Transparent & honest

Only User-agent, Allow, Disallow, and Sitemap directives are interpreted — anything else (Crawl-delay, Host, Noindex) is flagged, not evaluated.

🔒

Private by design

Everything runs in your browser. Your inputs never leave your device.

📱

Works everywhere

Fully responsive — perfect on desktop, tablet, and mobile. No app needed.

🔄

Regularly updated

Regularly maintained and reviewed to ensure accuracy.

💡

Free forever

No accounts, no limits, no watermarks. Use Robots.txt Validator as much as you need.

How to use Robots.txt Validator

1Copy your full robots.txt file (from yoursite.com/robots.txt or your CMS) into the "Robots.txt content" box.
2Run the tool to validate the robots.txt syntax: every line is checked for unrecognized directives, misplaced rules, and empty values.
3Read the validation verdict: Valid, Valid with warnings, or Invalid — and work through the Warnings list top to bottom.
4Scan the per-user-agent rule breakdown to confirm each crawler group (including the "*" wildcard) allows and disallows exactly what you intend.
5Check the 25-bot crawler verdicts table — it flags search engines and AI crawlers like GPTBot, ClaudeBot, and PerplexityBot that are blocked, so you can unblock the ones you want.

What can you do with Robots.txt Validator?

Robots.txt checker — Validate your robots.
Robots.txt tester — Validate your robots.
Check robots.txt syntax — Validate your robots.
Robots.txt ai crawlers — Validate your robots.
Validate robots.txt file — Validate your robots.

Good to know

Only User-agent, Allow, Disallow, and Sitemap directives are interpreted — anything else (Crawl-delay, Host, Noindex) is flagged, not evaluated.
Bot verdicts cover the engine's 25 tracked bots; all other crawlers follow the "*" group shown at the top of the breakdown.
Verdicts mean "matches Disallow: /" — longest-match precedence and Allow-over-Disallow tie-breaking are not modeled.
This is a syntax and intent check on pasted content; it does not fetch your live robots.txt or verify it is served correctly.

How it works

This tool validates robots.txt files with the open-source robotstxt-ai parser (sharozdawa/robotstxt-ai, MIT), vendored unchanged into this tool. The engine parses each User-agent block into Allow/Disallow rules and evaluates 25 tracked crawlers — Googlebot, Bingbot, GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot, Google-Extended, and more — reporting which are blocked by a "Disallow: /" match. A line-level audit adds honest warnings: unrecognized directives (only User-agent, Allow, Disallow, and Sitemap are interpreted), rules placed before any User-agent block, empty Disallow lines, duplicate groups, a site-wide "Disallow: /" under "*", and a missing Sitemap line. Limitations: the engine does not model longest-match precedence or Allow-over-Disallow tie-breaking; verdicts mean "matches Disallow: /", not a full crawl simulation. This is a syntax and intent check only — it never fetches your live file, so confirm the file is actually published at yoursite.com/robots.txt and served as text/plain.

Examples

Standard WordPress robots.txt

Verdict: Valid — the wildcard group blocks wp-admin but allows admin-ajax.php, and the sitemap is declared.

Blocking AI training crawlers

Verdict: Valid with warnings (no Sitemap line) — GPTBot and ClaudeBot show as Blocked in the crawler verdicts, everything else Allowed.

Accidentally blocking the whole site

Verdict: Valid with warnings — the tool flags that Disallow: / under "*" blocks ALL crawlers from the entire site, which deindexes you if intentional-by-mistake.

People also search for these — this tool handles them all:

robots.txt validatorrobots.txt checkerrobots.txt testercheck robots.txt syntaxrobots.txt ai crawlersvalidate robots.txt file