Skip to content

Robots.txt Tester & Generator

Check whether Googlebot, Bingbot or an AI crawler may fetch a URL, see the exact rule that decides it, and fix the file before you upload it. Or build a new robots.txt from scratch.

Why test robots.txt URL by URL

One wrong line in robots.txt can hide a whole section of a site from search, and nothing on the page itself warns you. Google removed its robots.txt Tester from Search Console in December 2023. The robots.txt report that replaced it shows whether Google could fetch and parse the file, but it does not tell you whether /products?sort=price is blocked. This tool answers that question, for any crawler, before a change goes live.

How crawlers decide whether a URL is allowed

Google, Bing and most other major crawlers follow the rules standardised in RFC 9309. In order:

  1. Pick the group. A crawler uses the group whose User-agent line names it most specifically, and ignores every other group. Googlebot-Image uses a Googlebot-Image group if there is one, otherwise Googlebot, otherwise *. Groups naming the same crawler are merged.
  2. Find every rule that matches the URL path, including the query string. Rules match from the start of the path.
  3. The longest matching rule wins. If an Allow and a Disallow rule are the same length, the Allow wins.
  4. No matching rule means allowed. So does an empty Disallow:.

The group rule catches people out most often. In the file below, Googlebot may crawl /cart/, because it reads only its own group and never sees the rules under *:

User-agent: *
Disallow: /cart/
Disallow: /checkout/

User-agent: Googlebot
Disallow: /internal-search/

If a crawler has its own group, repeat in it every rule you want that crawler to follow.

Wildcards and pattern matching

Two special characters are supported: * matches any sequence of characters, and $ marks the end of the URL. Paths are case-sensitive.

RuleMatchesDoes not match
/fish/fish, /fish.html, /fishing//Fish, /catfish
/fish//fish/, /fish/salmon/fish, /fish.html
/*.pdf$/files/guide.pdf/guide.pdf?v=2, /pdf-tools/
/*?Any URL with a query string/shop/shoes
/*?sort=/shop?sort=price/shop?page=2&sort=price
/*sort=/shop?sort=price, /shop?page=2&sort=price/shop?order=price

The last two rows are a common mistake: /*?sort= only matches when sort is the first parameter. To block a parameter in any position, leave out the ?.

How Google handles the robots.txt HTTP status

The fetch result above shows the status code your robots.txt returns, because it changes how Google treats the whole site:

StatusWhat Google does
2xxReads the file and follows its rules. Only the first 500 KiB is processed.
3xxFollows at least five redirect hops. If it still has no file, it treats robots.txt as missing.
4xx (except 429)Treats the site as having no robots.txt: everything may be crawled.
5xx or 429Treats the whole site as disallowed and stops crawling. After 30 days of errors it uses the last cached copy, or crawls without restrictions if it has none.

A robots.txt that returns a server error is far worse than one that returns 404. If your CDN or firewall blocks bots, check that it does not block robots.txt itself.

What robots.txt can't do

  • Remove pages from search. Blocked URLs can still be indexed from links. Use noindex on a crawlable page instead, and check it with the Page Indexability Checker.
  • Protect private content. The file is public, and anyone can read the paths you list. Use authentication for anything sensitive.
  • Hide redirects. If you block old URLs that now redirect, Google cannot see the redirect and may keep the old URLs indexed. Test the redirects with the Redirect Checker and leave them crawlable.
  • Control crawl rate in Google. Googlebot ignores Crawl-delay. Bing and Yandex honour it.

Blocking AI crawlers

AI companies publish separate user-agent tokens, so you can opt out of model training without leaving search. The generator includes the common ones:

TokenUsed forAffects search rankings?
GPTBotOpenAI model trainingNo
OAI-SearchBotResults in ChatGPT searchRemoves you from ChatGPT search answers
ClaudeBotAnthropic model trainingNo
Google-ExtendedGemini training and groundingNo; Google Search and AI Overviews use Googlebot
Applebot-ExtendedApple AI trainingNo; Applebot still crawls for Siri and Spotlight
PerplexityBotPerplexity search indexRemoves you from Perplexity answers
CCBotCommon Crawl, used by many AI datasetsNo

Honouring robots.txt is voluntary. The major companies above document that they follow it; scrapers that ignore it have to be blocked at the server or CDN.

Frequently asked questions

How do I test my robots.txt file?

Enter your domain and fetch the live file, or paste the contents. Then list the URLs you care about and pick a crawler. The tester shows whether each URL is allowed or blocked and which rule, on which line, decided it. You can edit the file in place and the results update immediately, so you can try a fix before uploading it.

Does Disallow in robots.txt stop a page from being indexed?

No. Disallow stops crawling, not indexing. Google can still index a blocked URL if other pages link to it, and show it without a description. To keep a page out of search results, allow crawling and add a noindex robots meta tag or X-Robots-Tag header. If the page is blocked, Google never sees the noindex.

Where does robots.txt have to be?

At the root of the host, named robots.txt in lowercase: https://example.com/robots.txt. A file in a subfolder is ignored. Each subdomain and each protocol needs its own file, so blog.example.com and shop.example.com are not covered by the file on example.com.

Is robots.txt case-sensitive?

The paths are. Disallow: /Private/ does not block /private/. Directive names such as Disallow and user-agent names such as Googlebot are not case-sensitive.

What happened to Google's robots.txt Tester?

Google removed the robots.txt Tester from Search Console in December 2023 and replaced it with the robots.txt report, which shows which robots.txt files Google found, when it last fetched them and any parse errors. The report does not test individual URLs against your rules, which is what this tool does.

Should I block AI crawlers in robots.txt?

It depends on whether you want your content used for AI training or shown in AI answers. Blocking GPTBot, ClaudeBot or Google-Extended does not affect your rankings in Google or Bing search. Blocking OAI-SearchBot or PerplexityBot can remove you from those products' search answers. Compliance is voluntary, so robots.txt is a request, not an access control.

Related Tools

  • Meta Tag Checker — Check a page's title, meta description, robots directives, canonical, viewport and hreflang tags, with a Google result preview and a fix for each issue.
  • Open Graph Checker — Preview how a link appears on Facebook, LinkedIn and X, check Open Graph and Twitter Card tags, and test the og:image size, format and reachability.
  • HTTP Header Checker — View raw HTTP response headers for every redirect hop and check security headers, caching, compression, HTTP version, CDN and X-Robots-Tag.
  • llms.txt Generator & Validator — Create an llms.
  • SEO Title Checker — Check a title tag for length, keyword position and truncation, preview it in desktop and mobile results, and get a 0–100 headline score.
  • Schema Markup Generator — Generate valid JSON-LD structured data for 31 schema types — Article, Product, FAQ, Local Business, Organization, Review and more.