Is your robots.txt blocking ChatGPT, Claude or Perplexity?
This free robots.txt checker for AI crawlers shows whether ChatGPT, Claude, Perplexity, Google, Bing and 17 other crawlers may fetch a page on your site. Paste your file, pick a path, and it splits the bots into search, user-fetch and training types, so you can keep training bots out without disappearing from AI answers.
Part 1
Check your file
Paste your robots.txt, test a path and read the result.
1. Which AI crawlers can reach this path?
Paste the full text of your robots.txt below. To get it, open yourdomain.com/robots.txt in a browser and copy everything.
Check a live site without copying and pasting
- Drag Send robots.txt to checker to your bookmarks bar, or and paste it as the address of a new bookmark.
- Open the website you want to check.
- Click the bookmark. This page opens with that site's robots.txt filled in.
Directional Your browser carries the text between the two tabs and no server is involved. A few sites block bookmarks with a security policy, and it works on desktop browsers only
2. How do you read the result?
Each row says whether that bot may crawl the path you entered, and which single rule decided it.
| Allowed | A matching Allow rule won, or no rule blocks the path |
|---|---|
| Blocked | A Disallow rule is the longest match for this bot and path |
| No rules for this bot | Your file has no group for it and no User-agent: * group, so it is allowed |
| From User-agent: * | The bot has no group of its own and inherited the wildcard rules |
The tool also flags file problems it can see in the text: a size over 500 KiB, rules before any User-agent line, unrecognized or misspelled directives, paths without a leading slash, a missing Sitemap line and Crawl-delay.
Confirmed Matching follows RFC 9309 and Google's documented behavior: the longest matching rule wins, and the least restrictive rule wins a tie
3. What does robots.txt control for AI?
It tells crawlers which URLs they may fetch. It does not hide pages, remove them from an index, or force any bot to obey.
Well-behaved crawlers read the file at the root of each host, such as example.com/robots.txt, before they fetch anything. A subdomain needs its own file. Because compliance is voluntary, treat robots.txt as a polite request and use passwords or server rules for anything private.
Confirmed Standard behavior under RFC 9309; compliance by individual AI bots is Directional and set out in the table below
Part 2
Know the bots
Which crawlers exist, what each one does and which assistant it powers.
4. Why are the bots split into three types?
Because blocking a training bot and blocking a search bot have different results: the first limits model training, the second can remove you from AI answers.
Search bots
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot
Build the index an engine uses to find and link your pages.
If blocked: your pages may not appear in that engine's answers. OpenAI says opted-out sites are not shown in ChatGPT search answers.
User-fetch bots
ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot
Open one page when a person asks an assistant about it.
If blocked: the assistant may fail to read your page on request. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
Training bots
GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot, cohere-ai
Collect content that may be used to train or ground models.
If blocked: your content is kept out of future training, as far as the vendor honors it. Answer visibility is not directly affected.
A fourth group, shown as "Other" in the tool, holds special-purpose bots: GoogleOther (generic Google product teams), Google-CloudVertexBot (crawls that site owners request for Vertex AI agents) and OAI-AdsBot (checks landing pages submitted as ChatGPT ads).
Directional Types follow each vendor's published bot documentation, which changes often. Check the vendor page before relying on a block
5. Which bot powers which AI assistant?
Each assistant uses a different set of bots, so the robots.txt line that matters depends on which assistant you want to appear in.
| Assistant or surface | Search bot | User-fetch bot | Training control |
|---|---|---|---|
| ChatGPT | OAI-SearchBot | ChatGPT-User | GPTBot |
| Claude | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | Perplexity says PerplexityBot is not used for foundation models |
| Google Search, AI Overviews and AI Mode | Googlebot | Not applicable | Not controlled by Google-Extended |
| Gemini | Googlebot | Not applicable | Google-Extended token |
| Microsoft Copilot and Bing | Bingbot | Not applicable | Not separately named |
| Apple search and Apple Intelligence | Applebot | Not applicable | Applebot-Extended token |
| DuckDuckGo DuckAssist | Not applicable | DuckAssistBot | Not applicable |
| Meta AI | Not applicable | Not applicable | Meta-ExternalAgent |
Directional OpenAI, Anthropic, Perplexity and Google rows follow their own documentation; the Microsoft, Apple, DuckDuckGo and Meta rows are compiled from public descriptions and may lag
6. Which bot does what, and who runs it?
The checker tests these 22 crawler names, listed with their operator, type and what each is used for.
| Bot | Operator | Type | Used for | Follows robots.txt? |
|---|---|---|---|---|
OAI-SearchBot | OpenAI | Search | Finding pages to show in ChatGPT search | Yes, per OpenAI |
Claude-SearchBot | Anthropic | Search | Improving the quality of Claude search results | Yes, per Anthropic |
PerplexityBot | Perplexity | Search | Surfacing and linking sites in Perplexity results | Yes, per Perplexity |
Googlebot | Search | Google Search, including its AI features | Yes | |
Bingbot | Microsoft | Search | Bing Search | Yes |
Applebot | Apple | Search | Apple search features such as Siri and Spotlight | Yes |
ChatGPT-User | OpenAI | User fetch | Opening a page when a person asks ChatGPT, or uses a GPT Action | OpenAI says rules may not apply |
Claude-User | Anthropic | User fetch | Opening a page when a question to Claude needs it | Yes, per Anthropic |
Perplexity-User | Perplexity | User fetch | Opening a page when a person asks Perplexity | Perplexity says it generally ignores robots.txt |
DuckAssistBot | DuckDuckGo | User fetch | Fetching pages to answer DuckAssist questions | Yes, per DuckDuckGo |
GPTBot | OpenAI | Training | Collecting content that may train OpenAI models | Yes, per OpenAI |
ClaudeBot | Anthropic | Training | Collecting public content that may train Claude | Yes, per Anthropic |
Google-Extended | Training (token) | A switch for using crawled content in Gemini training and in grounding for Gemini Apps and Vertex AI; no separate crawler | Token; does not affect Search | |
Applebot-Extended | Apple | Training (token) | A switch for using Applebot content in Apple's models | Token, not a crawler |
CCBot | Common Crawl | Training | Building the open dataset many models learn from | Yes |
Meta-ExternalAgent | Meta | Training | Collecting content for Meta's AI models | Yes, per Meta |
Bytespider | ByteDance | Training | Collecting content for ByteDance's AI products | Reported as inconsistent |
Amazonbot | Amazon | Training | Crawling for Amazon services and AI | Yes, per Amazon |
cohere-ai | Cohere | Training | Collecting content for Cohere models | Not confirmed |
GoogleOther | Other | Generic crawling by Google product teams; no effect on Search | Yes | |
Google-CloudVertexBot | Other | Crawls that site owners request to build Vertex AI agents | Yes | |
OAI-AdsBot | OpenAI | Other | Checking landing pages submitted as ChatGPT ads | Not stated |
Directional OpenAI, Anthropic, Perplexity and Google entries were checked on 14 October 2026 against their own documentation; the other operators' entries come from public descriptions and third-party reports, so verify before you ship a rule
Part 3
Understand the rules
How a crawler picks a rule, the limits of the file and what happens when it breaks.
7. How does a crawler pick which rule applies?
It finds the group that names it, collects that group's rules, and obeys the longest rule that matches the URL path.
- Find the group named for this botFor example
User-agent: GPTBot - No such group? Use
User-agent: *A named group replaces the wildcard group entirely - Keep only rules that match the path
*matches any text,$marks the end - Longest matching rule winsOn a tie, the less restrictive rule (Allow) wins
- No rule matches? The path is allowedSilence means yes
Confirmed RFC 9309 and Google's documentation describe this order; the order of groups in the file does not matter
8. What does a worked example look like?
With the file below, GPTBot is blocked from the whole site, but every other bot can still crawl everything except the admin area.
User-agent: *
Disallow: /wp-admin/
User-agent: GPTBot
Disallow: /| Bot | Path | Result |
|---|---|---|
| GPTBot | /blog/ | Blocked by Disallow: / |
| OAI-SearchBot | /blog/ | Allowed: no rule matches |
| OAI-SearchBot | /wp-admin/ | Blocked by the wildcard group |
Confirmed Results shown are what the checker above returns for this file
9. What technical rules and limits apply to the file itself?
The file must be UTF-8 text at the root of each host, and Google reads only the first 500 KiB of it.
| Topic | Rule |
|---|---|
| Location | Valid only for the protocol, host and port it is served from, so blog.example.com needs its own file |
| Format | UTF-8 plain text; invalid lines and a byte order mark are ignored |
| Size | Google ignores content after 500 KiB |
| Case | Directive names and User-agent values are not case-sensitive; paths are, so /Fish and /fish differ |
| Wildcards | * matches any characters and $ marks the end of the URL |
| Caching | Google generally caches the file for up to 24 hours; OpenAI says ChatGPT search can take about 24 hours to reflect a change |
| Redirects | Google follows up to five hops, then treats the file as missing |
| Crawl-delay | Google ignores it; Anthropic says its crawlers support it |
| Emerging controls | Proposals such as Cloudflare's Content-Signal directive and the IETF AI preferences work are not yet something to rely on |
Confirmed Limits are from Google's robots.txt documentation, RFC 9309 and the vendors' own pages; the emerging controls row is Directional
10. What happens if robots.txt is missing or returns an error?
A missing file means no restrictions, but a server error can make crawlers pause, so a broken robots.txt is worse than none.
| Response | What Google does |
|---|---|
| 200 with the file | Reads and applies the rules |
| 404 or another 4xx (not 429) | Treats it as no robots.txt, so nothing is restricted. Do not use 401 or 403 to limit crawling |
| 429, 5xx or a network error | Pauses crawling for about 12 hours and retries, then uses the last good copy for up to 30 days |
| Redirect | Follows up to five hops, then treats it as a 404 |
Confirmed Behavior documented by Google; other crawlers may handle errors differently
Part 4
Choose and write your setup
Pick a goal, copy a template and avoid the common mistakes.
11. Which setup fits your goal?
Most sites that want AI visibility should allow search bots and decide separately about training bots; blocking everything is rarely the goal.
| Bot type | Open to all | Block training only | Block all AI bots |
|---|---|---|---|
| Search bots | Allowed | Allowed | Blocked |
| User-fetch bots | Allowed | Allowed | Blocked |
| Training bots | Allowed | Blocked | Blocked |
| Likely effect | Maximum reach, content may train models | Stays in AI answers, limits training | Likely absent from AI answers |
Directional Effects depend on each vendor honoring the file; blocking is a request, not a lock
12. When should you run this check?
Run it after any launch, migration or security change, and whenever AI assistants stop mentioning your brand.
| Situation | What to look for |
|---|---|
| Your brand has vanished from AI answers | Any search bot blocked, in its own group or through a User-agent: * block |
| A site just launched or migrated | A leftover Disallow: / from staging that blocks every bot |
| You publish original research or paid content | Training bots blocked while search bots stay allowed |
| An online store | Cart, checkout and filter URLs blocked without blocking product pages |
| A SaaS or documentation site | Docs paths open to user-fetch bots so assistants can read them on request |
| A news or media site | A deliberate policy on training bots, written down and tested |
| An agency auditing a client | Whether the file matches the client's stated AI policy, and has a Sitemap line |
| A security or legal review | Whether robots.txt is being used as the only control, when it should not be |
13. What can you copy as a starting point?
Start from the template that matches your goal, then paste it into the checker above before you publish it.
Stay in AI answers, block training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlKeep every bot out of one section
User-agent: *
Disallow: /members/
Disallow: /cart/
Sitemap: https://example.com/sitemap.xmlAllow only one folder to AI training bots
User-agent: GPTBot
User-agent: ClaudeBot
Allow: /open-research/
Disallow: /Block all AI bots, search included (rarely wise)
User-agent: OAI-SearchBot
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
Disallow: /Replace example.com with your domain. If a bot also has its own group, repeat these rules inside that group, because a named group does not inherit the wildcard rules. The last template does not block Googlebot, so you stay in Google Search.
Directional Templates are starting points, not advice for your site; whether to block training is a business decision
14. What mistakes does this check catch most often?
The costliest mistakes are a leftover Disallow: /, a rule meant for one bot that hits all, and treating robots.txt as a way to remove pages.
| Mistake | What happens | Fix |
|---|---|---|
Disallow: / left from staging | Every bot without its own group is blocked from the whole site | Remove it, then re-test the home page path |
| CMS setting such as "discourage search engines" left on | The CMS can generate a site-wide block without anyone editing the file | Turn the setting off and re-check the live file |
| Expecting Google-Extended to hide you from Google Search | Nothing changes in Search; it only controls Gemini use | Use Googlebot rules or noindex for Search |
| Using Disallow to remove a page from Google | The URL can still be indexed without its content | Allow crawling and use a noindex tag |
| Adding a bot group and forgetting the wildcard rules | That bot ignores your * rules entirely | Repeat the rules in its group |
| Blocking CSS or JavaScript folders | Crawlers may not render the page the way people see it | Allow the files a page needs to render |
| Wrong letter case in a path | The rule does not match, because paths are case-sensitive | Match the exact case of the URL |
| Relying on Crawl-delay for Google | Google ignores it | Control crawl rate in other ways |
| File on the wrong host or path | It is ignored; it must be at the root of each host | Serve it at /robots.txt on every subdomain |
| Misspelled directive such as "Dissallow" | The rule is silently ignored | Fix the spelling; the checker flags unrecognized lines |
| No Sitemap line | Crawlers must discover pages by links alone | Add Sitemap: with the full URL |
| Server error (5xx) or timeout on robots.txt | Google pauses crawling and falls back to its last good copy | Make sure the file returns a normal 200 response |
| 401 or 403 on robots.txt, used to "block" crawlers | Google treats a 4xx as no robots.txt, so nothing is restricted | Use rules in the file, or a firewall rule, not an error code |
Confirmed Behaviors above are documented in RFC 9309 and Google's robots.txt guidance; error handling varies by crawler
Part 5
Fix access problems
Why a bot can still fail when robots.txt says yes, and how to check each layer.
15. Why can a bot still be blocked when robots.txt allows it?
Because robots.txt is only the first of four layers: your firewall, CDN, page tags and page content can each stop a crawler on their own.
1. robots.txt
Permission
States which URLs a bot may request.
Test with: the checker above.
2. Server, CDN, firewall
Access
Can answer a bot with 403, 429, a challenge page or a timeout, whatever robots.txt says.
Test with: the curl commands below and your server logs.
3. Page directives
Use
noindex, nosnippet and similar tags limit whether the page can be shown or quoted.
Test with: view-source and response headers.
4. Content
Extractability
Content that needs a login or JavaScript, or that is hard to quote, is reachable but not usable.
Test with: the page with JavaScript off.
A quick live check you can run yourself
curl -I -A "GPTBot" https://example.com/
curl -I -A "OAI-SearchBot" https://example.com/
curl -I -A "ClaudeBot" https://example.com/
curl -I https://example.com/A 200 for the normal request but a 403, 429 or 503 for a bot name means something on the server reacts to the name. A firewall that verifies bots by IP range may treat your test differently from the real bot, so a pass here is not proof. Compare with your logs.
Where the vendors publish their IP ranges
- OpenAI: GPTBot, OAI-SearchBot and ChatGPT-User
- Perplexity: PerplexityBot and Perplexity-User
- Anthropic: its crawler list; it advises against blocking by IP because that can stop the crawler from reading robots.txt
A message you can send your host or CDN support
Please check whether any firewall, CDN or bot-protection rule blocks or
challenges these crawlers, by user agent or IP range: OAI-SearchBot,
Claude-SearchBot, PerplexityBot. They are search crawlers we want to
allow. Please also confirm that /robots.txt returns a normal 200 response
to them.Directional Some CDNs and security plugins block AI bots by default or by a single setting; check yours rather than assume
16. Can a bot reach your page and still not be allowed to quote it?
Yes. Page-level directives such as noindex and nosnippet can keep a crawlable page out of results or limit how much text may be shown.
| noindex | Keeps the page out of the index; the page must stay crawlable for the tag to be seen |
|---|---|
| nosnippet or max-snippet | Limits the text a search engine may show, including in its AI features |
| data-nosnippet | Excludes one section of a page from snippets |
| X-Robots-Tag header | Sets the same directives in the HTTP response, useful for files such as PDFs |
Directional Google documents these controls for Search and says its AI features follow the same eligibility; other engines document their own
17. How do you edit robots.txt on WordPress, Shopify or Wix?
Most hosted platforms generate the file for you and give you a setting or template to change it, so you rarely upload a file by hand.
| Platform | Where the file comes from | Common trap |
|---|---|---|
| WordPress | A virtual file made by WordPress unless a real robots.txt sits in the site root; SEO plugins add an editor | The "discourage search engines" setting, or a security plugin that blocks bots separately |
| Shopify | Generated by Shopify; changed through the robots.txt.liquid theme template | You cannot upload a file, and a broken template edit can drop rules |
| Wix | Generated by Wix; an editor sits in its SEO tools | Settings differ by plan and may change |
| Custom or static site | A real file in the web root | The file is not deployed from staging to production |
Directional Platform menus and plan limits change; check your platform's current help pages before editing
Part 6
Troubleshoot and maintain
A diagnosis order, a live example and how often to re-check.
18. What order should you check things when AI never mentions you?
Check access first and content last: confirm the file, then the server, then the page tags, then whether the content is worth quoting.
- Test your key paths aboveHome page and one article, search bots first
- Open the live file yourselfRight host, status 200, not a redirect to a login page
- Run the curl checksLook for 403, 429 or challenge pages
- Check CDN, firewall and pluginsAI-bot toggles and rate limits
- Check noindex and nosnippetPage tags and response headers
- Read your server logsDid the bots actually request the page?
- Still not mentioned? It is not an access problemLook at entity clarity and answer-first content
Directional A working order for diagnosis, not a guarantee: allowing a bot never guarantees a citation
19. How is this site's own robots.txt set?
It lets every AI crawler in this tool read the public pages, training bots included, because the goal of this site is to be found and cited.
User-agent: *
Allow: /
Disallow: /wp-admin/
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
Sitemap: https://asfandjunejo.com/sitemap.xmlThat is an excerpt of the live file. A site selling original data or paid content would choose differently. What happened in search and AI answers after changes like this is logged in the experiment log.
Confirmed Excerpt of this site's current robots.txt; the choice suits this site only
20. How often should you re-check robots.txt?
Check it after every launch, migration, CDN or plugin change, and review the bot list at least every quarter.
- After a redesign, migration or move to a new host or CDN
- After installing or updating a security, SEO or caching plugin
- When a vendor announces a new crawler or renames one
- Each quarter, as a routine, with a note of what you chose and why
- Whenever AI assistants stop mentioning the brand
A bot name in a log can be faked, so confirm important ones against the IP ranges the vendor publishes.
Directional Cadence is a practical default, not a rule from any vendor
Part 7
What do popular sites actually do?
An original snapshot of how 103 well-known websites treat AI crawlers in robots.txt.
21. How many popular sites block AI crawlers?
In this sample, 32% block GPTBot and 44% block Common Crawl, but only 17% block OpenAI's search bot, so most sites that block training still stay open to AI search.
| Crawler | Operator and use | Sites blocking it | By a rule naming it |
|---|---|---|---|
GPTBot | OpenAI training | 33 of 103 (32%) | 28 |
ClaudeBot | Anthropic training | 43 of 103 (42%) | 37 |
CCBot | Common Crawl training | 45 of 103 (44%) | 37 |
Google-Extended | Google Gemini control token | 30 of 103 (29%) | 25 |
Applebot-Extended | Apple training control token | 41 of 103 (40%) | 33 |
Meta-ExternalAgent | Meta training | 38 of 103 (37%) | 31 |
Bytespider | ByteDance training | 46 of 103 (45%) | 37 |
PerplexityBot | Perplexity search | 31 of 103 (30%) | 25 |
Claude-SearchBot | Anthropic search | 29 of 103 (28%) | 22 |
ChatGPT-User | OpenAI user fetch | 22 of 103 (21%) | 17 |
OAI-SearchBot | OpenAI search | 18 of 103 (17%) | 13 |
Googlebot | Google Search | 1 of 103 (1%) | 0 |
Bingbot | Bing Search | 1 of 103 (1%) | 0 |
Confirmed Observed on 2026-10-14: each site's robots.txt was fetched once and tested against the path / using the same matching rules as the checker above
22. What stands out in the data?
Blocking is common for training bots, uneven for search bots, and almost never applied to Google or Bing.
- 47 of 103 sites (46%) name none of six major AI bots in their robots.txt
- 16 sites block GPTBot but allow OAI-SearchBot, the "block training, keep search" setup
- 34 sites (33%) block at least one of the three AI search bots, and 15 block all three
- 5 sites block every bot that has no rule of its own, which shuts out AI crawlers by default
- Only 1 site blocks Googlebot or Bingbot
- 19 sites (18%) list no sitemap in robots.txt
Directional Interpretation of one snapshot; policies change often
23. How was this measured, and what are the limits?
115 well-known sites were picked by hand across news, technology, shopping, reference and travel; 103 returned a usable robots.txt.
- This is not a random sample, so it does not show what "the web" does
- Each file was fetched once on 2026-10-14 by an unnamed research bot; some sites serve different rules to different visitors
- 12 sites could not be read and are left out
- It tests robots.txt only, not firewalls or whether a bot obeys
Confirmed Method and limits as stated; re-run the checker on any file to reproduce a row
24. What did each site do?
Every site in the sample is listed below with its result for seven crawlers.
Show all 103 sites
| Site | GPTBot | ClaudeBot | CCBot | Google-Extended | PerplexityBot | OAI-SearchBot | Googlebot |
|---|---|---|---|---|---|---|---|
| ahrefs.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| airbnb.com | Blocked | Blocked | Allowed | Allowed | Allowed | Allowed | Allowed |
| alibaba.com | Blocked | Allowed | Allowed | Blocked | Allowed | Allowed | Allowed |
| aliexpress.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| allrecipes.com | Allowed | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed |
| amazon.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| anthropic.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| apnews.com | Blocked | Blocked | Blocked | Allowed | Blocked | Allowed | Allowed |
| apple.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| atlassian.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| axios.com | Allowed | Allowed | Blocked | Allowed | Allowed | Allowed | Allowed |
| backlinko.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| bankrate.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| bbc.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| bbcgoodfood.com | Blocked | Allowed | Blocked | Allowed | Allowed | Allowed | Allowed |
| bestbuy.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| bloomberg.com | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed |
| booking.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| britannica.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| businessinsider.com | Allowed | Blocked | Blocked | Allowed | Allowed | Allowed | Allowed |
| canva.com | Blocked | Blocked | Blocked | Allowed | Allowed | Allowed | Allowed |
| cloudflare.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| cnet.com | Blocked | Blocked | Blocked | Allowed | Blocked | Blocked | Allowed |
| cnn.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| coursera.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| dailymail.co.uk | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| dropbox.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| ebay.com | Blocked | Blocked | Blocked | Allowed | Blocked | Allowed | Allowed |
| elpais.com | Allowed | Blocked | Blocked | Allowed | Blocked | Allowed | Allowed |
| etsy.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| expedia.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| facebook.com | Allowed | Allowed | Blocked | Allowed | Allowed | Blocked | Allowed |
| figma.com | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed | Allowed |
| foxnews.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| github.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| glassdoor.com | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed | Allowed |
| google.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| harvard.edu | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| healthline.com | Blocked | Blocked | Blocked | Allowed | Allowed | Allowed | Allowed |
| hubspot.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| huffpost.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| ikea.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| imdb.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| indeed.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| investopedia.com | Allowed | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed |
| lemonde.fr | Allowed | Blocked | Blocked | Blocked | Allowed | Allowed | Allowed |
| linkedin.com | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed |
| mailchimp.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| medium.com | Blocked | Blocked | Allowed | Allowed | Allowed | Allowed | Allowed |
| microsoft.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| mit.edu | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| moz.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| nbcnews.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| nerdwallet.com | Allowed | Allowed | Allowed | Blocked | Allowed | Allowed | Allowed |
| netflix.com | Allowed | Allowed | Blocked | Allowed | Allowed | Allowed | Allowed |
| nike.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| notion.so | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| nytimes.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| openai.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| perplexity.ai | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| pinterest.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| quora.com | Blocked | Blocked | Allowed | Allowed | Blocked | Blocked | Allowed |
| reddit.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked |
| reuters.com | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed |
| salesforce.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| searchenginejournal.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| searchengineland.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| semrush.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| shopify.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| slack.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| spiegel.de | Blocked | Blocked | Blocked | Allowed | Allowed | Blocked | Allowed |
| spotify.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| squarespace.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| stanford.edu | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| stripe.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| target.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| techcrunch.com | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed | Allowed |
| telegraph.co.uk | Blocked | Blocked | Blocked | Allowed | Blocked | Blocked | Allowed |
| theatlantic.com | Allowed | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed |
| theguardian.com | Allowed | Blocked | Blocked | Allowed | Blocked | Allowed | Allowed |
| theverge.com | Allowed | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed |
| tomsguide.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| tripadvisor.com | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed | Allowed |
| udemy.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| usatoday.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| vox.com | Allowed | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed |
| walmart.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| washingtonpost.com | Allowed | Blocked | Blocked | Allowed | Blocked | Allowed | Allowed |
| wayfair.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| webmd.com | Blocked | Blocked | Blocked | Allowed | Allowed | Allowed | Allowed |
| who.int | Allowed | Allowed | Blocked | Allowed | Allowed | Allowed | Allowed |
| wikipedia.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| wired.com | Allowed | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed |
| wix.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| wordpress.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| wordpress.org | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| wsj.com | Allowed | Blocked | Blocked | Blocked | Blocked | Allowed | Allowed |
| x.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| yelp.com | Blocked | Blocked | Blocked | Blocked | Blocked | Blocked | Allowed |
| yoast.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| youtube.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| zillow.com | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
| zoom.us | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed | Allowed |
Confirmed Results as fetched on 2026-10-14; a site that changed its file since is not reflected
Part 8
Questions answered
Short answers to the questions that come up most.
25. What do people ask about AI crawlers and robots.txt?
The ten questions below cover blocking, Google-Extended, compliance, citations and privacy. Tap a question to open the answer.
Does blocking GPTBot remove you from ChatGPT?
No. GPTBot is OpenAI's training crawler; ChatGPT search uses OAI-SearchBot, so blocking GPTBot alone does not remove you from search answers.
Confirmed OpenAI's crawler documentation says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, and that disallowing GPTBot signals content should not be used for training
Does Google-Extended affect Google Search or AI Overviews?
No. Google-Extended is a control token for Gemini training and grounding, and Google says it does not affect inclusion in Google Search.
AI Overviews and AI Mode draw on Google Search, which is crawled by Googlebot. To limit how your text is used there, use page-level controls such as nosnippet, not Google-Extended.
Confirmed Google-Extended behavior is from Google's crawler documentation; the AI Overviews point is Directional and based on Google's published guidance
Should you block AI crawlers at all?
Block training bots if you object to model training; keep search bots open if you want AI assistants to find and cite your pages.
Directional A business choice, not a fixed rule; the trade-off is control of your content against visibility in AI answers
Do AI bots actually obey robots.txt?
Most major vendors say their search and training crawlers do, but user-fetch bots often do not, and some smaller crawlers are reported to ignore it.
OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores them, because a person asked for the page. If you need to stop a bot, use a server or firewall rule as well.
Directional Vendor statements are documented; independent verification of compliance is limited
Does blocking a bot remove content it already collected?
No. A block asks a bot not to fetch your pages from now on; it does not delete anything already collected.
Confirmed Robots.txt governs future fetching only; takedown of existing data is a separate request to each vendor
What is the difference between robots.txt and llms.txt?
Robots.txt is an established standard that sets what crawlers may fetch; llms.txt is a proposed convention for pointing language models at key pages, and Google does not use it.
Neither one guarantees a citation. Fix access with robots.txt first, because a file that guides a model cannot help if the model's crawler is blocked.
Directional Based on Google's guidance on generative search; individual AI vendors may document their own use of llms.txt
Does allowing a bot get you cited?
No. Allowing a crawler only removes one barrier; being cited depends on whether your content answers the question clearly and is trusted.
Directional Citation behavior differs by engine and changes over time
Can you slow AI crawlers with Crawl-delay?
Sometimes. Anthropic says its crawlers support Crawl-delay, but Google ignores it, so do not rely on it for every bot.
Directional Support is set by each operator; check the vendor's page
How many websites block GPTBot?
In a snapshot of 103 well-known sites on 2026-10-14, 32% blocked GPTBot, and 17% blocked OpenAI's search bot.
Confirmed Hand-picked sample, not a random one; see the data in Part 7
Do big publishers block AI search bots as well as training bots?
Some do: 34 of 103 sites blocked at least one AI search bot, and 15 blocked all three, but most blocking targets training bots.
Confirmed Same sample and date as above
How many times can you use the free tools?
You get 3 free runs across all the free tools. After that, joining the Search Signals newsletter with a business email unlocks 10 more.
Loading an example, copying and downloading do not use a run. The count is kept in your browser, so clearing site data resets it.
Directional A fair-use limit that may change; the count is a convenience, not a security control
Why does the unlock need a business email?
To keep the newsletter list made of real business contacts: personal addresses such as Gmail or Yahoo, and temporary addresses, are not accepted.
If your work address is rejected by mistake, email asfand@asfandjunejo.com and the tools can be unlocked by hand.
Directional The check uses lists of free and disposable providers plus a mail-server lookup, so it is strong but not perfect
Is the file you paste sent anywhere?
No. The check runs in your browser, and the tool does not send your text to a server. The only thing a tool can send is the business email you choose to give when you unlock more runs.
The page itself is loaded and measured like any other page on this site. You can confirm the check sends nothing by watching the network panel in your browser while you run it.
Directional Stated by the builder; verifiable in your own browser
Why does the tool not fetch my robots.txt for me?
Browsers stop one site reading another site's files, so a client-side tool cannot fetch them. Use the bookmark above to send a live site's robots.txt to the checker, or paste the text.
The bookmark runs on the site you are viewing and passes the text to this page through your browser, so no server is involved.
Confirmed This is the browser's same-origin policy; fetching for you would need a server, which this tool deliberately avoids
Part 9
Limits and sources
What this tool cannot do and where the facts come from.
26. What can this checker not tell you?
It reads only the robots.txt text it is given, so it cannot see server rules, firewalls or CDN settings that block bots without robots.txt.
| Reads | User-agent groups, Allow and Disallow rules, wildcards and the end anchor, from pasted text or the bookmark |
|---|---|
| Cannot see | Firewall or CDN bot blocking, login walls, meta robots tags |
| Cannot prove | That a bot actually obeys the file |
| Does not give | A score or grade; the rules either match or they do not |
Directional Prototype tool; compare with your server logs before changing live rules
27. Where do the facts on this page come from?
From each operator's own documentation where one exists, checked on 14 October 2026.
Blocked a search bot by mistake, or not sure why AI ignores you? See why AI assistants ignore a brand → Or see the AI visibility service → Back to all free tools →