AI Search Visibility Audit: A Technical Eligibility Checklist for Google, ChatGPT, Claude & Perplexity

What is an AI search technical eligibility audit?

An AI search technical eligibility audit is a structured investigation into whether a website meets the observable access, delivery, indexing, and participation criteria for specific AI search platforms. It identifies confirmed technical barriers, separates optional optimization opportunities from actual requirements, and records the evidence necessary to justify remediation.

Traditional technical SEO provides much of the foundation for this work. Search engines must discover and retrieve URLs, interpret their contents, evaluate indexing signals, and select appropriate documents for search results. When important information cannot be obtained or processed, downstream search visibility can be affected.

AI-assisted search introduces additional considerations, but it does not eliminate these fundamentals. Google’s (opens in a new tab)official guidance for AI features (opens in a new tab) continues to link eligibility for AI Overviews and AI Mode to the existing Google Search index and standard technical requirements. Other providers publish their own crawler access recommendations, making it necessary to evaluate the platform being investigated rather than assuming that Google’s requirements apply everywhere.

Technical eligibility and AI visibility are different measurements

A technically eligible website has satisfied the applicable conditions for consideration within a particular system. Whether that website is subsequently retrieved, selected as supporting evidence, cited, or visited by a user remains a separate question.

Consider a publicly accessible article that returns an HTTP 200 response, permits Googlebot, appears in Google’s index, and is eligible to display a search snippet. Those findings provide meaningful evidence about its eligibility for Google Search and its associated generative AI experiences, subject to the relevant Search Console participation setting. They do not prove that the article will appear in an AI Overview for a particular search.

The distinction becomes even more important when evaluating platforms that do not provide comparable indexation or citation reports. An analyst may establish that an AI search crawler is permitted by robots.txt and has successfully accessed a page, yet still lack sufficient information to establish whether that page has been selected for a user-facing answer.

A responsible audit therefore produces conclusions at the level supported by the available evidence.

The purpose of the audit is not to convert these different measurements into one artificial indicator of success. It is to determine which conditions are known, which remain uncertain, and which require action.

What should be audited?

The audit should concentrate on URLs that materially represent a website’s content and commercial purpose. A homepage alone is not a sufficient sample because different templates, subdirectories, subdomains, and application components may have different technical behavior.

A representative assessment could include the homepage, important service pages, product pages, informational articles, research resources, JavaScript-dependent pages, and known redirect destinations. Where a website contains several significantly different templates, each should be tested independently.

The number of audited URLs should be determined by the site’s architecture and the consistency of observed behavior rather than by a fixed industry-wide requirement. Testing a small number of URLs can reveal a shared configuration problem, but it cannot establish that every page on a large website behaves identically.

For every selected URL, the audit should record the precise location, intended platform, test date, relevant technical observations, and supporting evidence. An unavailable observation should remain unverified rather than being converted into a passing result.

How does AI search eligibility differ across Google, ChatGPT, Claude and Perplexity?

AI search eligibility is platform-specific because Google, OpenAI, Anthropic, and Perplexity use different crawling systems, retrieval mechanisms, and publisher controls. A website must therefore be evaluated against the documented behavior of each relevant platform. No single robots.txt directive, indexing report, structured data format, or technical readiness score establishes universal AI search eligibility.

The differences are easiest to understand by examining how the major platforms describe their own infrastructure.

Google AI Overviews and AI Mode

Google’s AI Overviews and AI Mode are built upon the broader Google Search ecosystem. According to Google’s (opens in a new tab)technical requirements (opens in a new tab), Googlebot must not be blocked, the page must be functional and return an appropriate success response, and the page must contain indexable content.

For a page to qualify as a supporting link in AI Overviews or AI Mode, Google further requires that it be indexed and eligible to appear in Search with a snippet.

In addition to those ordinary requirements, Google’s current guidance identifies a separate site-level participation condition: the website must be included in Search generative AI features through the relevant Search Console control.

This distinction matters because the Search Console setting governs participation in specific generative AI features without replacing Google’s ordinary crawl, indexation, or content-quality systems.

Google also distinguishes the Google-Extended control from its general search-crawling infrastructure. Google-Extended concerns particular AI uses and training permissions. Its behavior should not be confused with the Search Console generative AI participation control or with Googlebot’s ability to access a page.

The resulting audit must investigate three separate conditions: ordinary Google Search eligibility, snippet availability, and the effective generative AI participation setting.

ChatGPT Search and OpenAI crawlers

OpenAI publishes separate crawler identities for search discovery, model-development crawling, and certain user-triggered requests.

Its (opens in a new tab)official crawler documentation (opens in a new tab) identifies OAI-SearchBot as the crawler used to surface websites in ChatGPT’s search features. OpenAI recommends allowing this crawler in robots.txt and permitting requests from its published IP address ranges.

The same documentation describes GPTBot as a crawler whose collected material may be used for developing AI foundation models. A website owner can express different preferences for search crawling and model-training crawling rather than treating the two functions as inseparable.

OpenAI also documents ChatGPT-User, which may be used when a person initiates an action that requires retrieving a website. Because these requests are user-triggered, their relationship to robots.txt differs from that of automated search crawling.

A technical audit should therefore evaluate OpenAI’s search crawler separately from its training crawler and user-triggered fetcher. Blocking GPTBot is not equivalent to blocking OAI-SearchBot, and permitting ChatGPT-User does not establish that OAI-SearchBot has successfully crawled the site.

Claude and Anthropic crawlers

Anthropic documents three crawler categories with distinct purposes.

ClaudeBot is associated with the collection of web content that may contribute to model training. Claude-SearchBot is used to support search-related discovery and result quality. Claude-User is associated with website access initiated through users’ interactions with Claude.

In its (opens in a new tab)crawler documentation (opens in a new tab), Anthropic explains that restricting Claude-SearchBot may reduce search-related visibility, while restricting Claude-User can affect user-directed retrieval.

These distinctions mean that a website may intentionally permit search-oriented access while restricting training-related crawling. Such a configuration should be documented as a policy decision rather than automatically classified as a technical failure.

The audit must also distinguish between a documented crawler preference and an observed access event. Anthropic’s published definitions describe the roles of the crawlers, but they do not establish that a particular website has actually been fetched or incorporated into a user-facing answer.

Perplexity search and user-directed retrieval

Perplexity similarly distinguishes between its automated search crawler and user-initiated fetching infrastructure.

According to (opens in a new tab)Perplexity’s official crawler documentation (opens in a new tab), PerplexityBot supports the discovery and surfacing of websites in search results. Perplexity-User supports requests initiated by users and is not used as an ordinary automated search crawler.

Perplexity states that user-triggered retrieval generally ignores robots.txt rules. Therefore, a robots.txt assessment alone should not be used to make a universal claim about how Perplexity will respond to a particular user-directed fetch request.

The documentation also provides official IP address lists and recommendations for Web Application Firewall configuration. These are relevant because allowing a crawler in robots.txt does not guarantee that security infrastructure will allow the request to reach the application.

Cross-platform crawler reference

Each provider separates a search crawler, a model-training crawler and a user-triggered fetcher, so treat them as three different policies.

The crawler names must be interpreted in accordance with their providers’ published policies, which may change over time.

This matrix does not establish the complete retrieval architecture of each provider. AI systems can use multiple sources, infrastructure layers, and retrieval methods, some of which are not comprehensively described in public documentation.

For that reason, the audit should make platform-specific conclusions only where the underlying access requirement or observed behavior can be verified.

What are the five stages of AI search visibility?

A practical AI search audit should separate discovery, access, eligibility, content interpretation, and observed visibility. These stages provide an analytical structure for identifying where a website may encounter technical barriers. They should not be interpreted as an exact representation of the proprietary internal pipeline used by every AI search provider.

The framework begins with whether a platform can find the URL, proceeds to its ability to retrieve and process the content, and ends by examining whether any user-facing visibility has actually been observed.

Stage 1: Discovery

Discovery concerns whether a platform has an identifiable route to the URL.

For Google, relevant evidence can include internal links, XML sitemap references, known URL discovery records, and Search Console information. Other providers may rely on their own crawling systems or additional search infrastructure, but their complete discovery processes are not always publicly observable.

The presence of an XML sitemap supports URL discovery, but it does not establish successful crawling, indexing, or AI citation eligibility.

A page isolated from a website’s normal navigation may warrant further investigation, particularly if no reliable discovery pathway is available. However, the absence of a link in one crawl does not prove that every search platform is unaware of the URL.

Stage 2: Access

Access concerns whether an intended crawler or fetcher can obtain the resource.

This depends on more than robots.txt. DNS resolution, TLS negotiation, server availability, CDN configuration, firewall rules, request rate limits, redirects, and access challenges can all affect whether the content is delivered.

An audit should distinguish between a crawler being permitted to request a URL and the server successfully returning the intended content.

A technically public page can still become inaccessible to a particular crawler because an infrastructure rule classifies its traffic as suspicious.

Stage 3: Eligibility

Eligibility concerns whether the available content meets the platform’s documented participation requirements.

For Google AI Overviews and AI Mode, the evidence includes Google Search indexation, snippet eligibility, and the effective Search generative AI inclusion setting.

For other platforms, publicly documented crawler access preferences can be assessed, but it may not be possible to establish an equivalent complete eligibility verdict. The absence of an official platform-specific indexation report should be recorded as a measurement limitation.

A successful HTTP response cannot replace an eligibility assessment.

Stage 4: Content interpretation

Once the content is accessible, the audit examines whether its substantive information can be extracted and understood by the tested systems.

Important questions include whether the initial HTML contains useful text, whether JavaScript is required to reveal critical information, whether canonical signals conflict, and whether the page identifies its central entities consistently.

A page that shows only a loading interface before JavaScript execution may create more retrieval uncertainty than a page that delivers its principal information in the initial HTML.

However, this should be framed as a technical robustness consideration rather than a claim that all AI systems require server-side rendering.

Stage 5: Observed visibility

The final stage examines what can actually be measured.

Evidence may include Google generative AI impressions, Bing AI citation reporting, recorded AI answer citations, verified crawler activity, or visits attributable to identifiable AI referrals.

These observations are not interchangeable. A server log entry records a request, a citation records a source reference, and an analytics event records website behavior.

The audit should preserve those differences rather than combining them into a single undifferentiated visibility score.

Business outcomes such as qualified leads, transactions, and revenue should be analyzed once visibility has been established and sufficient evidence is available. They form a separate commercial measurement layer, not a technical eligibility condition.

How do you audit robots.txt and AI crawler permissions?

A robots.txt audit determines which automated crawlers are instructed to access or avoid particular URL paths. For AI search, the audit must evaluate the actual rules applying to each relevant crawler, distinguish search access from training controls, and confirm that the policy is consistent with the website owner’s intentions. A robots.txt permission result alone cannot prove successful retrieval.

The robots.txt file is normally located at the root of a website’s host and protocol.

For example:

https://example.com/robots.txt

Its rules communicate access preferences to compliant automated crawlers. They do not provide authentication or prevent unauthorized clients from requesting publicly available content.

Google’s (opens in a new tab)robots.txt documentation (opens in a new tab) describes crawler groups, Allow and Disallow directives, path matching, and the interpretation of conflicting rules.

Step 1: Retrieve the actual robots.txt response

Request the file directly, save the headers and body, and read the response before you interpret any rule.

A basic request can be made with:

curl -sS -D robots-headers.txt \
  -o robots-body.txt \
  https://example.com/robots.txt

The request saves the response headers and body separately, allowing the auditor to inspect both the returned policy and the server’s delivery behavior.

The response should be examined before interpreting any rules. A successful request that returns HTML instead of a robots.txt policy, redirects to an unexpected location, or returns an infrastructure-generated error may require further investigation.

robots.txt responses are interpreted according to crawler-specific rules, particularly when the file is missing or returns a server error. A missing file should not automatically receive the same interpretation as a deliberate Disallow: / directive.

Because the policy applies to a specific host and protocol, each host and its relevant subdomains should be checked individually. A rule observed for www.example.com should not be assumed to govern every other hostname used by the organization.

Step 2: Identify the intended crawler policies

The audit should begin with an explicit access-policy decision.

A website owner may want Googlebot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot to access its public informational pages while excluding certain training crawlers.

That is a legitimate policy objective and should not be treated as an error simply because different crawlers receive different instructions.

An illustrative policy might be:

User-agent: Googlebot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /

This example demonstrates crawler-specific preferences. It is not a complete recommended robots.txt configuration and should not be copied into production without evaluating the site’s existing rules, private paths, application behavior, and business policies.

Step 3: Evaluate effective rules, not individual lines

Judge the effective rule for each crawler and URL, not single lines or whether a crawler name appears in the file.

A robots.txt file can contain multiple crawler groups, specific paths, wildcards, and rules that appear contradictory when read in isolation.

For Google, the most specific applicable path rule generally determines whether the requested URL is allowed or disallowed, with additional documented handling for conflicts.

Other crawler providers may implement robots.txt interpretation differently. The audit should use the relevant platform documentation or a verified policy evaluator when making a platform-specific determination.

For example, a file containing a broad Disallow: / rule does not necessarily establish that every named crawler is blocked if more specific crawler groups and applicable rules are present.

Likewise, an Allow: / directive should not be interpreted without considering which crawler group it belongs to and whether more specific path rules apply.

The correct question is not whether a crawler’s name appears in the file, but whether the complete policy permits that crawler to request the selected URL.

Step 4: Distinguish search crawling from training restrictions

One of the most common analytical errors is treating every AI-related crawler as a search-discovery crawler.

OpenAI explicitly allows website owners to express independent preferences for OAI-SearchBot and GPTBot. Anthropic also distinguishes search-related crawling from its training-related crawler.

An intentional training exclusion should therefore be recorded separately from a search access failure.

The audit should not recommend removing a training-related restriction without understanding the publisher’s actual data-use policy.

Step 5: Record the result and its limitations

For every crawler and URL, record the host, response, crawler token, URL, matching rule, interpretation, source and test date.

For each evaluated crawler and URL, the audit should retain:

A complete robots.txt audit establishes the stated access policy. It does not establish that the crawler has made a request or that network security systems will permit one.

Those questions require independent delivery and log-level evidence.

How do you detect CDN, WAF and HTTP access failures?

A website may permit an AI crawler in robots.txt while blocking that crawler at the network, CDN, Web Application Firewall, or application layer. Effective access verification therefore requires inspecting HTTP responses, redirect destinations, security events, and authentic crawler requests. A successful request from a normal browser or command-line client does not establish that every provider’s crawler receives the same response.

Modern websites frequently rely on CDNs, automated bot management, rate limiting, and application security controls. These systems are valuable for protecting infrastructure, but incorrectly configured rules can also interfere with legitimate crawling.

The resulting problem may remain invisible during a routine website audit because a human visitor receives the expected page while a crawler receives a different response.

Test the complete response, not only the status code

A basic HTTP inspection should record the final URL, response headers, status code, and returned body.

An example command is:

curl -sS -L \
  -D response-headers.txt \
  -o response-body.html \
  -w 'Status: %{http_code}\nFinal URL: %{url_effective}\n' \
  https://example.com/page/

The command follows redirects and saves the response body for subsequent inspection.

A separate request without automatic redirect following may be necessary to examine each redirect hop individually.

A HEAD request can provide useful header information, but its behavior is not necessarily identical to a normal GET request. A technical audit should not rely exclusively on curl -I when the objective is to establish what page content is actually delivered.

Google’s (opens in a new tab)HTTP status code documentation (opens in a new tab) explains how different response classes affect Google’s crawling and processing.

Interpret HTTP responses correctly

Read each status code against the URL's intended state: the same code can be correct for one URL and a failure for another.

The status should always be interpreted in relation to the intended URL.

A 404 response is appropriate for a genuinely removed document. It becomes a technical failure when an important live page is expected to be publicly available at that location.

Similarly, a temporary 503 during maintenance does not have the same operational significance as persistent server failures affecting legitimate crawler requests.

Detect security challenges disguised as successful responses

A CDN or security service may deliver a challenge page rather than the intended content. In some configurations, a request can return a successful HTTP status while the body contains only a challenge message, JavaScript verification, or an application-generated error.

The auditor should therefore inspect the returned document for substantive content on each page.

A valid HTML response should not be marked as successfully accessible merely because it contains a title element and a generic page shell.

For important URLs, the response should contain the expected page identity and meaningful content, or a documented mechanism by which the intended crawler can obtain that content.

Verify crawler identity before trusting logs

A request presenting itself as Googlebot, OAI-SearchBot, or another known crawler is not automatically authentic.

Unrelated clients can copy user-agent strings.

Google documents a verification procedure that uses reverse DNS followed by forward DNS confirmation, or by comparison against its published crawler IP ranges.

Reference: (opens in a new tab)Google, Verify Requests from Google Crawlers and Fetchers (opens in a new tab).

OpenAI and Perplexity also publish crawler IP addresses to support verification.

For example:

• OpenAI search crawler IP ranges (opens in a new tab)
• Perplexity crawler documentation and IP ranges (opens in a new tab)

An audit should use current provider-published information rather than a permanent, manually maintained assumption about which addresses belong to a crawler.

Investigate WAF behavior without weakening security

When authentic crawler requests are blocked, the remediation should be narrowly scoped.

A security rule may be based on the request’s IP address, user-agent characteristics, geographic origin, bot classification, request frequency, or combinations of these factors.

The correct remediation depends on which rule is responsible.

For example, an unnecessary challenge may be removed for verified search-crawler IP ranges while maintaining protections against unrelated automated traffic.

It would be inappropriate to disable an entire WAF solely to address a single crawler access problem.

Perplexity’s official documentation specifically recommends combining published IP verification with user-agent matching when configuring relevant WAF rules.

Preserve evidence before changing infrastructure

A meaningful delivery audit should retain the request timestamp, original URL, final URL, response status, selected response headers, relevant body characteristics, and available CDN or firewall events.

Where authentic crawler logs are available, record the verified identity, request path, response status, and action taken by the security infrastructure.

If no authentic request has been observed, the result should not be presented as proof that the crawler has been blocked or allowed.

The proper finding may simply be that the public endpoint responds successfully under the tested conditions, while the authenticity of the crawler delivery remains unverified.

How do you check Google AI Overviews and AI Mode eligibility?

A website’s eligibility for Google’s generative AI Search features depends on ordinary Google Search requirements, snippet eligibility, and the effective Search generative AI participation setting. These conditions must be checked independently because successful crawling does not guarantee indexing, indexing does not automatically establish snippet availability, and participation settings can exclude a website from specified generative AI experiences.

This is one of the most important technical distinctions in the entire audit.

Requirement 1: The page must be accessible to Googlebot

Google’s minimum Search technical requirements identify Googlebot accessibility as a foundational condition.

If a robots.txt rule blocks an important URL, Google may be unable to crawl its content. However, Google notes that a blocked URL may still sometimes appear in Search results without its content being crawled.

Consequently, the audit should not equate the presence of a URL in search results with successful access to the underlying document.

Googlebot accessibility should be established through the actual robots.txt policy and appropriate Google inspection or verified evidence from a crawler.

Requirement 2: The page must be indexed

Google’s AI feature guidance states that a page must be indexed and eligible to appear with a snippet to qualify as a supporting link in AI Overviews or AI Mode.

The Google Search Console URL Inspection tool provides direct evidence about Google’s indexed version of a URL.

For an authorized property, inspect the exact URL and review its index status, last crawl information, crawl eligibility, page fetch result, and Google-selected canonical where available.

A live URL test serves a different purpose. It evaluates the current page under the conditions tested by Google’s inspection infrastructure, but it cannot guarantee future indexing.

The distinction between indexed data and a live test is documented in Google’s (opens in a new tab)URL Inspection guidance (opens in a new tab).

A live test reporting that a page is available to Google should not be presented as confirmation that the page has already entered the Google index.

Requirement 3: The page must be eligible for a search snippet

A page needs no directive that blocks indexing or snippets: noindex, nosnippet and max-snippet:0 all affect eligibility.

Google supports several directives that affect how a page’s content can appear in Search.

The noindex directive prevents a page from being indexed when Google can access and process the directive.

The nosnippet directive prevents a text snippet from being displayed and also restricts content use in Google’s AI Overviews and AI Mode.

The max-snippet directive limits the amount of text available for snippets and can restrict the content used directly by those AI features.

A value of max-snippet:0 has the same practical snippet-blocking effect as nosnippet.

The data-nosnippet attribute can restrict particular portions of visible HTML content rather than applying a blanket restriction to the entire page.

These controls should be interpreted according to Google’s (opens in a new tab)robots meta tag and X-Robots-Tag specification (opens in a new tab).

The audit should record whether the page contains such directives and whether they are consistent with the publisher’s desired participation policy.

Requirement 4: The website must be included in Search generative AI features

Google’s Search Console generative AI control represents an additional site-level participation condition that older technical SEO checklists may not include.

Google documents the control under:

Search Console → Settings → Search generative AI

The setting governs participation in applicable generative AI features such as AI Overviews, AI Mode, and supported generative AI features in Discover.

The available options include inclusion, exclusion, and, where applicable, inheritance of a parent property’s setting.

An audit should inspect the effective control value rather than assuming the default applies to every property.

For example, a URL-prefix property may inherit a parent property’s exclusion preference. In that case, inspecting only the child property’s visible configuration without considering inheritance can lead to an incorrect conclusion.

The control should not be confused with Google’s ordinary indexing directives or Google-Extended training-related controls.

Google also states that the generative AI control is not a general ranking or inclusion signal for unrelated parts of Search.

Reference: (opens in a new tab)Google Search Console, Search Generative AI Control (opens in a new tab).

The complete Google eligibility assessment

Treat Google eligibility as six separate tests; passing all of them is a stronger position, not a guarantee of selection.

Passing these tests establishes a stronger technical eligibility position for Google, but it does not guarantee that a particular page will be retrieved or selected as a supporting source.

The audit should not attribute a lack of AI impressions to a technical failure when all applicable eligibility conditions appear to be satisfied.

At that point, the investigation may need to examine content relevance, source selection, competition, search demand, or measurement limitations.

How do you audit canonical URLs, indexing directives and redirects?

Canonicalization, indexing directives, and redirects influence how search engines interpret URL identity and content availability. A technical AI search audit should examine whether the intended content is delivered consistently at the intended location, whether important pages are unintentionally excluded from indexing, and whether canonical signals conflict with the preferred URL structure.

These checks are especially relevant to websites with multiple URL variants, parameter-driven pages, subdomains, content migrations, or JavaScript-based routing.

Canonical URLs are preferences, not absolute commands

A canonical URL identifies a preferred version of duplicate or substantially similar content.

Website owners can communicate that preference through rel="canonical" annotations, redirects, and other supported mechanisms.

Google treats canonical declarations as signals rather than commands. It may select a different canonical URL when its systems determine that another version is more appropriate.

The audit should therefore distinguish between the publisher’s declared canonical and Google’s observed canonical selection.

Reference: (opens in a new tab)Google, Canonical URL Documentation (opens in a new tab).

A typical declaration may appear as:

<link rel="canonical" href="https://example.com/guides/ai-search-audit/">

The presence of this element does not, by itself, prove that Google has chosen the declared URL.

The correct investigation should compare the requested URL, the redirected destination, the declared canonical, and the Google-selected canonical, where available.

Look for conflicting canonical signals

Canonical conflicts commonly arise when a website sends contradictory instructions through different technical layers.

For example, a page may redirect to a newer URL while retaining a canonical declaration pointing back to the old address. Another page may declare a canonical target that is blocked from crawling or excluded from indexing.

A content management system may also generate more than one canonical element, or JavaScript may modify the canonical element after the initial HTML response.

Each condition requires investigation because the relevant signals no longer describe a consistent preferred document.

The audit should not automatically classify the absence of a self-referencing canonical as a failure. Google can determine canonical URLs without an explicit declaration, and the importance of that declaration depends on the site’s actual duplication and URL structure.

Identify accidental noindex directives

A noindex directive can sit in the HTML or in an HTTP header, so inspect both locations.

A noindex directive can be applied through an HTML robots meta element:

<meta name="robots" content="noindex">

It can also be delivered through an HTTP response header:

X-Robots-Tag: noindex

Both locations should be inspected because an auditor examining only the HTML document can miss restrictions imposed through server configuration, reverse proxies, or other infrastructure components.

A useful distinction is that Google must generally be able to access the page to discover and honor a noindex directive.

Blocking a URL through robots.txt while also expecting Google to process a noindex directive on that same inaccessible page creates a technical contradiction.

Google explains this interaction in its (opens in a new tab)noindex documentation (opens in a new tab).

Evaluate the complete redirect destination

A redirect should be assessed as a relationship between the original URL and the destination, not merely as a successful status code.

The audit should identify whether the redirect is permanent or temporary, where it leads, whether the target returns the intended content, and whether the destination is itself eligible for indexing.

Redirect loops, unnecessary chains, broken destinations, and unexpected moves between hosts can create accessibility problems.

A URL that redirects successfully to an unrelated page may technically resolve without delivering the information that the original URL was expected to contain.

Similarly, a redirect to a generic homepage should not automatically be classified as a successful migration of an informational article.

The assessment must consider the intended relationship between the content.

Record URL-level consistency

Record each URL's request, redirect chain, declared and selected canonical, robots signals and finding side by side.

The purpose of the audit is not to force every URL to be independently indexed.

Duplicate and alternate URLs may legitimately consolidate into a preferred document. The important question is whether the content intended for search visibility remains appropriately represented and accessible.

Can AI search systems read your content? Testing raw HTML and JavaScript rendering

Content accessibility depends on what the relevant client can receive and process, not merely on how a page appears in a browser. A reliable technical audit should compare the initial HTML response with the rendered page, verify that essential information is available via the intended access method, and identify content that depends on JavaScript, authentication, or interactive actions.

This is particularly important for modern websites built with client-side rendering frameworks, JavaScript-heavy themes, API-driven content, and interactive components.

The difference between source HTML and rendered content

A server-rendered page may return its principal heading, explanatory paragraphs, links, and supporting information in the initial HTML response.

A client-rendered application may initially return a relatively empty document containing scripts and a container into which content is inserted after JavaScript executes.

Both approaches can produce a visually similar experience for a human visitor. However, they expose different initial information to clients that do not execute the application’s scripts.

Google documents its own rendering process in its (opens in a new tab)JavaScript SEO guidance (opens in a new tab). Googlebot can render JavaScript using its web rendering infrastructure, though rendering occurs as part of a broader processing pipeline rather than being guaranteed immediately upon fetching the initial document.

Google also recommends server-side or pre-rendering approaches as useful options because not all automated clients can execute JavaScript.

That observation supports a technical robustness recommendation: important content should be available as reliably as possible without depending on unnecessary interactive steps.

It does not establish that all AI search engines use the same rendering process or that JavaScript content is inherently ineligible for AI search.

Step 1: Examine the initial HTML response

Save the raw HTML the server returns and check whether the page's main content is already in it.

Retrieve the page using a normal HTTP request:

curl -sS https://example.com/page/ -o page-source.html

Inspect the saved file for the principal page heading, substantive content, meaningful links, metadata, and any relevant indexing directives.

The presence of content in raw HTML is useful evidence because it demonstrates that the initial response contains information independent of subsequent browser-side rendering.

However, raw HTML should not be treated as a complete view of what Google or another capable browser-based system will ultimately process.

Step 2: Compare the browser-rendered page

Open the page in a modern browser and inspect the resulting document structure after its scripts have executed.

Compare the rendered content with the initial response.

The objective is to determine which parts of the page exist immediately and which require additional processing.

Important differences might include:

• A central article body inserted only after a JavaScript request.
• Product descriptions loaded after the initial page response.
• Navigation links generated only after an interaction.
• Important content hidden behind authentication.
• Canonical or robots directives changed by JavaScript.
• Content missing because an API request fails.
• Structured information present in the initial HTML but absent from the rendered page.

These observations should be documented individually rather than reduced to an unqualified statement that the page is or is not JavaScript-friendly.

Step 3: Examine rendering through Google’s inspection tools

For Google Search, use the URL Inspection tool to review the page’s indexed information and, where supported, perform a live test.

A successful live test can provide access to rendered HTML, loaded resources, JavaScript errors, and a rendered screenshot.

These records help identify whether Google’s testing infrastructure can access information that appears only after rendering.

A live inspection cannot guarantee that the page will be indexed or selected as a result. It also does not establish rendering behavior for OpenAI, Anthropic, or Perplexity.

Each test is evidence about the system that performed it.

Step 4: Check mobile content equivalence

Google uses mobile-first indexing, making the content available through the mobile version particularly important.

Its (opens in a new tab)mobile-first indexing guidance (opens in a new tab) recommends that important content remain equivalent between desktop and mobile versions.

A website that removes substantial explanatory content or important internal links from its mobile presentation may result in differences in what Google can process.

Responsive design does not necessarily imply a problem, and content placed inside accessible tabs or accordions is not automatically excluded.

The audit should examine whether substantive information remains available in the mobile document and rendered output.

Step 5: Identify rendering-related risks without inventing crawler behavior

A technical report may discover that a page relies heavily on JavaScript to expose its primary content. That is an observation.

The assertion that a specific AI provider cannot retrieve that content requires additional evidence about the provider’s retrieval implementation or actual observed behavior.

Where such evidence is unavailable, the finding should be classified as a content-delivery risk or an inconclusive platform-specific condition rather than a confirmed exclusion from AI search.

A careful report might state:

Interim conclusion The page’s principal content is absent from the initial HTML and becomes available only after client-side rendering. Google’s live inspection can be used to evaluate its rendering behavior. Equivalent processing by other AI search crawlers has not been independently established.

That conclusion is more defensible than declaring that JavaScript-dependent pages cannot appear in AI-generated answers.

Step 6: Document content availability by extraction layer

Record, for each content component, whether it appears in the raw HTML, the rendered page, Google's inspection and other crawlers.

This table is a recording template. Its unverified values must be replaced with actual test evidence before being presented as the result of a website audit.

Why content delivery matters beyond rendering

A webpage may render successfully yet present its information in a structure that is difficult to interpret accurately.

For example, a service page may contain accurate business facts but fail to associate them with the responsible organization clearly. A technical article may contain valid information but bury its main answer inside unrelated navigation, promotional elements, and poorly labeled interactive components.

These issues do not establish a platform eligibility failure. They raise a different question concerning the clarity, accessibility, and consistency of the information being supplied.

The next stage of the audit should therefore shift from the ability to retrieve individual pages to the systems that help ensure important content remains discoverable and correctly understood: internal links, XML sitemaps, URL relationships, structured data, and entity consistency.

Those elements should be evaluated for their documented technical functions rather than presented as hidden or guaranteed AI ranking signals.

How do internal links, XML sitemaps and URL architecture affect AI search discovery?

Internal links, XML sitemaps, and coherent URL structures help search engines discover important content and understand relationships between pages. For Google, crawlable links and sitemaps are established discovery mechanisms. Their effectiveness, however, does not guarantee equivalent behavior across all AI search providers, nor does successful discovery guarantee indexing, retrieval, citation, or inclusion in an AI-generated response.

A technical eligibility audit should investigate how important URLs become discoverable before interpreting their absence from downstream reporting. This is particularly relevant for newly published websites, large content libraries, JavaScript-driven interfaces, and websites whose navigation does not reflect their actual content architecture.

Audit the internal-link discovery pathways

Google’s (opens in a new tab)link best practices (opens in a new tab) recommend using standard HTML anchor elements with valid href attributes. Such links allow Google to discover destinations and use surrounding context and anchor text to understand their relevance.

Consider a website with an extensive research section accessible only via an interactive JavaScript search interface. Human visitors may find individual reports by entering keywords, but an automated crawler may not receive a conventional set of discoverable links to those reports.

The presence of a working search interface does not necessarily establish a reliable crawl path.

A straightforward crawlable internal link resembles:

<a href="/research/ai-crawler-access/">
  AI crawler access research
</a>

By contrast, a JavaScript event attached to an element without a crawlable destination introduces additional uncertainty:

<span onclick="openResearchReport()">
  AI crawler access research
</span>

The second example may function correctly for a browser user, but Google does not recommend relying on this kind of structure for URL discovery.

An audit should examine both the source HTML and the rendered document, particularly when navigation elements are dynamically generated.

Identify orphaned or weakly connected pages

An orphaned page is generally understood to be a URL that has no discoverable internal links from pages included in the crawl.

The finding should be interpreted carefully. A crawler may fail to discover a link due to sampling limitations, blocked resources, incomplete JavaScript execution, or a crawl configuration. Therefore, the absence of an internal link in one crawl should initially be classified as an observation requiring verification.

For important pages, evaluate whether there is a reasonable discovery pathway from the homepage, a relevant category page, an editorial hub, or another established part of the website.

Also examine whether internal links reflect meaningful relationships. A technical guide about AI crawler access may appropriately link to research on robots.txt behavior, indexing documentation, and related diagnostic resources. Inserting unrelated links solely to increase internal link counts is unlikely to improve the reader’s understanding.

Google does not prescribe a universal number of internal links that every page must contain. The appropriate structure depends on the site’s size, hierarchy, content relationships, and navigational requirements.

Verify XML sitemap quality

An XML sitemap provides search engines with information about URLs that a publisher considers relevant for discovery.

Google’s (opens in a new tab)sitemap documentation (opens in a new tab) explains that sitemaps can be especially useful for large websites, newly established sites, and sites with specialized media.

A sitemap does not guarantee that its URLs will be crawled or indexed.

The audit should verify whether the sitemap is reachable, contains valid XML, uses correct URL references, and accurately represents the website’s intended indexable content.

A simplified sitemap entry may look like:

<url>
  <loc>https://example.com/research/ai-search-eligibility/</loc>
  <lastmod>2026-10-20</lastmod>
</url>

The lastmod value should reflect a genuine, significant update to the page it represents. It should not be changed arbitrarily to create the appearance of freshness.

Google’s (opens in a new tab)sitemap best practices (opens in a new tab) also recommend using fully qualified URLs and maintaining consistency between submitted URLs and canonical preferences.

Look for structural inconsistencies

A website may have a technically valid sitemap while presenting contradictory signals elsewhere.

For example, a sitemap might list a URL that redirects to a different destination, include a noindex directive, or specify a different canonical URL.

These conditions do not always indicate an error. Some may be intentional during migrations or content consolidation. However, they should be examined because the sitemap and the page are communicating different preferences.

The same principle applies to inconsistent trailing slashes, protocol changes, subdomain variants, and URLs whose visible content is accessible via multiple paths.

The objective is to determine whether important content is consistently represented, not to force every accessible URL into the index.

Discovery audit evidence

Combine link, sitemap and canonical evidence to find pages with a candidate discovery problem.

The result should identify pages requiring discovery improvements without claiming that every AI provider uses the same sitemap or link-discovery process.

Structured data, entity consistency and llms.txt: what is actually required?

Structured data and clear entity descriptions can help search systems interpret website content, but they are not universal requirements for AI citations. Google’s 2026 guidance explicitly states that no special schema markup or llms.txt file is required for its generative AI Search features. A technical audit should therefore validate existing markup and factual consistency while distinguishing supported SEO practices from speculative AI-specific optimization claims.

This distinction matters because AI visibility recommendations frequently combine established technical standards with emerging conventions that have not been adopted as mandatory requirements by major search platforms.

A responsible technical audit must separate those categories before assigning severity.

Structured data is useful, but not an AI citation guarantee

Structured data provides machine-readable information describing entities, attributes, and relationships represented on a webpage.

For example, an article may use Article or BlogPosting markup to describe its headline, author, publication date, and publisher. An organization may use relevant organizational markup to identify its official name, website, and other supported properties.

Google recommends supported structured data for eligible Search features and provides detailed requirements for specific rich-result types.

However, Google’s (opens in a new tab)AI optimization guidance (opens in a new tab) clarifies that structured data is not required to participate in generative AI Search and that there is no special AI-specific schema format that publishers must implement.

The practical audit question is therefore whether existing structured data accurately represents the visible content and satisfies any applicable feature-specific requirements.

It is not whether the website has implemented a fictional universal AI schema.

Validate structured data against visible content

Structured data must match the visible page, even when its syntax is valid.

Consider an article whose structured data identifies an author, publication date, and headline that differ from the information displayed to readers.

Even if the markup is syntactically valid, the inconsistency should be investigated because the page communicates conflicting information about the same resource.

A simplified JSON-LD example illustrates the intended relationship:

{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "AI Search Visibility Audit",
  "author": {
    "@type": "Person",
    "name": "Example Author"
  },
  "datePublished": "2026-10-20",
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://example.com/ai-search-audit/"
  }
}

This example is illustrative and intentionally omits other properties that may be appropriate for a production implementation.

The audit should check whether the markup accurately reflects the represented article, whether the referenced entity URLs are appropriate, and whether the supported Google rich-result types meet Google’s additional requirements.

Google’s (opens in a new tab)structured data guidelines (opens in a new tab) state that structured data should describe the content of the page on which it appears and meet relevant content policies.

A successful structured data test does not guarantee the appearance of rich results, let alone inclusion in an AI-generated answer.

Audit entity identity and factual consistency

Entity consistency extends beyond structured data.

A website may identify the same organization through its homepage, service pages, author biographies, research articles, and external professional profiles.

If these sources contain contradictory names, outdated descriptions, unsupported credentials, or conflicting organizational details, an auditor should determine which statements are authoritative.

The objective is to reduce ambiguity and factual inconsistency for readers and systems processing the information.

Relevant checks include whether an organization’s public identity is described consistently, whether authorship is attributable, whether significant factual claims have inspectable evidence to support them, and whether pages clearly identify the products, services, organizations, or concepts they discuss.

These are information-quality and semantic clarity checks. They should not automatically be classified as technical eligibility blockers.

There is no public Google requirement establishing that a website must achieve a particular proprietary entity score before its pages can appear in AI Overviews.

What is llms.txt?

llms.txt is a proposed convention for presenting website information and documentation links in a format intended to be convenient for language-model-oriented tools.

The proposal is associated with the publicly available specification at (opens in a new tab)llmstxt.org (opens in a new tab).

Its existence, however, should not be confused with universal adoption by AI search providers.

Google has clarified its position on these files. Its (opens in a new tab)official AI optimization guidance (opens in a new tab) states that Google Search does not use llms.txt as a special eligibility or ranking mechanism and that maintaining such a file neither improves nor harms Google Search visibility.

This means that a website without an llms.txt file should not fail a Google AI Overview eligibility audit.

An organization may still maintain such a file for another service that explicitly consumes it or for its own documentation workflows. The relevant question is whether the intended consumer actually uses the format.

How to audit an existing llms.txt file

When a website already provides llms.txt, the auditor can record its status and inspect its contents.

A basic request might be:

curl -sS -L \
  -D llms-headers.txt \
  -o llms-body.txt \
  https://example.com/llms.txt

The resulting response should be evaluated for its actual content rather than simply for a successful HTTP status code.

A route that redirects to the homepage may return HTTP 200 after redirects without providing an llms.txt resource. A response containing HTML instead of the expected text should be classified according to the audit’s defined content-type and resource-validation rules.

Where the file exists, inspect whether it contains functioning links, accurately describes the website, and avoids exposing private or unpublished resources.

The finding should be recorded as an optional resource assessment unless the organization has identified a specific technical dependency on the file.

Required versus optional measures

Only four checks are mandatory for Google AI eligibility; every other item is a supporting practice or an optional assessment.

The outcome should be a precise distinction between documented participation requirements, established supporting practices, and optional experimentation.

How do you verify authentic AI crawler activity in server and CDN logs?

Server and CDN logs can provide direct evidence that a particular client requested a website resource, but a user-agent string alone does not establish the client’s identity. A valid crawler-access investigation should examine request metadata, verify the client against provider-published mechanisms where available, and distinguish successful content delivery from requests that were rejected or challenged.

This is an important transition from static website configuration to observed infrastructure behavior.

A robots.txt file may indicate that a crawler is permitted. A command-line test may show that the page is publicly accessible. Neither proves that a genuine search crawler has requested the page and received the intended content.

Identify the available log sources

Depending on the hosting environment, request evidence may be distributed across several infrastructure layers.

Origin web server logs can record requests that reach the application server. CDN logs can record requests processed at the edge. WAF logs may identify security rules that challenged or blocked clients before they reached the origin.

This distinction matters because a blocked crawler may never appear in the origin logs.

An organization should therefore identify which infrastructure layers are responsible for receiving requests before interpreting the absence of evidence.

Where access is available, the audit should record the request timestamp, host, URL path, HTTP method, response status, client IP address, user-agent string, and relevant security action.

Sensitive values should be protected in accordance with the organization’s logging and retention policies.

Why user-agent matching is insufficient

Clients supply HTTP user-agent strings. They can be copied, modified, or falsified.

A request claiming to be Googlebot is not necessarily from Google.

Google documents two methods for verifying requests from its crawlers and fetchers: using appropriate reverse and forward DNS verification or matching client IP addresses against the relevant officially published crawler ranges.

Reference: (opens in a new tab)Google, Verify Requests from Google Crawlers and Fetchers (opens in a new tab).

The precise verification method should follow the provider being investigated.

For OpenAI, published crawler IP ranges are available through its (opens in a new tab)crawler documentation (opens in a new tab).

For Perplexity, official IP sources are documented for both PerplexityBot and Perplexity-User.

Anthropic’s crawler documentation should be consulted for its current identification and access guidance rather than assuming that another provider’s verification process applies unchanged.

A safe Googlebot verification procedure

Verify Googlebot with a reverse DNS lookup followed by a forward lookup that returns the same IP address.

For a Google crawler request, obtain the client IP address from the relevant log entry.

Perform a reverse DNS lookup using an appropriate DNS tool:

host CRAWLER_IP_ADDRESS

Inspect the returned hostname to determine whether it matches Google’s documented crawler hostname patterns.

Then resolve that hostname forward:

host VERIFIED_HOSTNAME

The forward lookup should resolve back to the original client IP address.

The complete result must match Google’s current verification guidance. Merely finding the word “google” somewhere in a hostname is not sufficient.

For larger log datasets, an automated verification process can use Google’s official crawler IP range files instead of performing individual DNS lookups.

Distinguish request outcomes

Classify each log result separately: a URL with no observed request is not a blocked URL.

Consider the following example record structure:

This is a blank audit template, not a representation of observed crawler traffic.

The verification record should distinguish:

• An authentic request that successfully received the intended page.
• An authentic request that was redirected.
• An authentic request rejected by access controls.
• A request that claims a known crawler identity but fails verification.
• A URL for which no qualifying request was observed.

The final category must not automatically be interpreted as a crawler block.

The crawler may simply not have requested that URL during the period covered by the available logs.

Calculate observed crawler delivery rates

When sufficient verified requests are available, an audit can summarize their outcomes.

A basic metric is:

Observed success rate = verified successful requests ÷ verified requests evaluated × 100

The definition of success must be declared.

For example, a response that returns HTTP 200 but serves a security challenge should not be automatically classified as successful content delivery.

Similarly, a 301 redirect may be the correct behavior for a migrated URL but may not satisfy a test that expects the final article body at the original address.

The analysis should include the reporting window, sample size, provider, URL population, response classification rules, and known missing logs.

An aggregate success rate calculated from a small or unrepresentative sample should not be generalized to the entire website.

How do you measure AI search impressions, citations, mentions and referral traffic?

AI search visibility should be measured using the reporting available on each platform, with impressions, citations, mentions, crawler visits, and referral traffic treated as separate metrics. No single measure establishes the complete user journey from discovery to AI source selection and business conversion, and the available reporting differs across Google, Microsoft, OpenAI, Anthropic, and Perplexity.

This distinction is particularly important in 2026 because first-party AI visibility reporting has become more detailed while remaining uneven across providers.

Google Search Console generative AI reporting

Google’s (opens in a new tab)Generative AI Performance report (opens in a new tab) provides information on impressions for supported generative AI features in Google Search.

According to Google’s documentation, the report includes AI Overviews and AI Mode, with separate reporting available for supported generative AI experiences in Discover.

An impression is recorded when a link to the website is shown to a user in a covered feature under Google’s reporting rules.

The report supports analysis by dimensions such as pages, countries, devices, and dates.

However, its available dimensions and data limitations must be respected. The report does not provide a complete query-level reconstruction of every AI-generated answer that included the website.

Google also notes that the report may not appear when a website has insufficient qualifying impressions.

Consequently, the absence of a report should not automatically be classified as a technical eligibility failure.

The auditor should check the effective Search generative AI inclusion setting, examine the site’s eligibility evidence, and distinguish unavailable reporting from confirmed exclusion.

Bing Webmaster Tools AI Performance

Microsoft introduced the (opens in a new tab)AI Performance report in Bing Webmaster Tools (opens in a new tab) in February 2026.

The reporting covers citation activity across supported Microsoft Copilot, Bing, and partner AI experiences.

Its original metrics include total citations, cited-page activity, grounding-query information, and trends over time.

Microsoft subsequently expanded its (opens in a new tab)AI visibility insights (opens in a new tab) with additional thematic and comparative capabilities.

These reports provide useful evidence about observed references to publisher content, but the meaning of the measurements must remain clear.

A citation count is not a conventional ranking position. A citation-share measure is not equivalent to organic market share or commercial conversion share. Sampled grounding queries should not be presented as a complete list of user searches.

The audit should use the metric definitions published for the exact report being analyzed.

AI answer citations and brand mentions

For platforms that do not expose comparable first-party publisher dashboards, controlled answer observation may provide an additional, narrower form of evidence.

A citation occurs when an AI experience displays a source reference under the observed conditions.

A brand mention occurs when the brand is named in the answer, regardless of whether the answer links to the brand’s website.

These events should not be conflated.

A response may mention an organization while citing an independent publisher. Another response may cite an article without naming the organization that published it.

An audit should separately record the cited URL, cited domain, brand mention, platform, prompt, date, and any available contextual information.

Design a reproducible prompt observation

A controlled visibility test should use a documented set of questions related to the website’s subject matter.

For example, a business offering technical signage services might test questions about permitting requirements, installation planning, and accessibility standards. The selected questions should represent real informational or commercial needs rather than being constructed solely to force the company’s name into the answer.

The test record should contain:

Repeated observations may be useful because AI responses can vary across queries, sessions, models, locations, and dates.

However, a set of manually selected prompts should not be represented as statistically representative of all users unless its sampling design supports that claim.

AI referral traffic is a separate measurement

Website analytics may identify visits referred by recognizable AI platforms or search experiences.

Those sessions can provide evidence of visits associated with identifiable referrers, but the measurement is subject to browser behavior, referral handling, attribution rules, and limitations in analytics collection.

Some visits may arrive without useful referrer information. Others may be grouped into channels that do not clearly identify the AI experience involved.

The absence of identified AI referral traffic does not prove that a brand was never mentioned or cited in an AI response.

Similarly, a referral session does not independently establish which specific answer or citation led to the visit.

Use a measurement boundary matrix

Keep availability, exposure and commercial outcome as separate levels of evidence; no single metric covers all three.

The purpose of the measurement framework is to preserve the difference between evidence of availability, evidence of exposure, and evidence of commercial outcomes.

How do you classify and prioritize technical AI search failures?

Technical AI search findings should be prioritized according to documented platform requirements, confirmed evidence, affected content, business importance, and remediation risk. A confirmed eligibility blocker deserves different treatment from an optional improvement whose influence on citation selection has not been demonstrated. A single numerical AI-readiness score can obscure this distinction and should not replace a severity-based decision process.

The first and most important step is determining whether the reported condition is genuinely a failure.

A site that intentionally excludes a training crawler has not necessarily made a technical mistake. A missing llms.txt file does not violate Google’s AI Search eligibility requirements. An important indexed page returning an unexpected noindex directive presents a substantially different concern.

These findings should not be combined into the same failure category.

Establish a severity model

Rate every finding P0, P1, P2, P3 or Informational against the platform and the owner's intent.

Severity must be assessed relative to the platform and business intent.

For example, an exclusion from Google’s generative AI features may constitute a critical issue for an organization actively seeking visibility in those features. The same setting may be correct for a publisher that has deliberately opted out.

The auditor should preserve that distinction rather than treating every exclusion as an error.

Add evidence confidence

Severity describes the potential or confirmed importance of a finding. Evidence confidence describes how reliably the finding has been established.

The two dimensions should remain independent.

A suspected critical problem based on one inconclusive test should not be treated as a confirmed critical failure.

Conversely, a fully verified low-severity issue should not receive unnecessary attention simply because the evidence is strong.

A practical evidence classification can use:

Verified: A reliable test or authoritative property record directly supports the conclusion.

Strongly supported: Multiple consistent observations support the finding, although some limitations remain.

Inconclusive: The available observations do not distinguish the suspected condition from plausible alternatives.

Not tested: No relevant verification has been conducted.

Contradicted: Reliable evidence conflicts with the proposed finding.

These labels should appear in the audit record alongside severity and remediation recommendations.

Prioritize remediation using expected consequences

After confirmed blockers have been identified, the remaining issues should be evaluated according to affected URLs, business importance, implementation effort, risk, and the ability to verify the outcome.

For example, resolving an authenticated crawler access failure affecting an entire product category is generally a more direct eligibility intervention than adding optional descriptive markup to an already accessible article.

However, the article should not claim that the first change will necessarily generate more citations. It may remove an access barrier without changing the platform’s source-selection behavior.

A defensible remediation recommendation should explain what technical condition will change and what test will establish that the change worked.

Separate remediation from outcome measurement

A successful remediation can have two forms of validation.

The first is technical verification: the original condition has changed as intended.

The second is visibility observation: a relevant downstream outcome has appeared or changed after the correction.

These should not be treated as equivalent.

A WAF correction can be successfully verified when genuine authorized crawler requests are no longer blocked. That does not guarantee that the page will begin receiving AI citations.

The outcome assessment may require a longer observation window and consideration of other factors that are changing.

What does a reproducible AI search eligibility investigation look like?

A credible AI search audit example should demonstrate how a suspected technical issue moves from an initial observation to a verified diagnosis and, where appropriate, a documented remediation. The evidence must identify the affected URL, platform, applicable requirement, exact test conditions, and result. An illustrative scenario should never be presented as a completed audit of a real website.

The following example demonstrates the required analytical structure. All findings and responses are hypothetical, included solely to show how an investigation should be documented.

Illustrative Every finding in this example is hypothetical; it shows the documentation structure and is not a completed audit.

Example scenario: A page appears accessible, but an AI search crawler encounters a challenge

The page looks accessible, yet an AI search crawler is challenged; the steps below show how to prove that and what the proof does not cover.

Assume that an informational website has published an in-depth technical guide at:

https://example.com/guides/ai-search-readiness/

The page is publicly accessible in a standard browser and has a valid canonical URL.

The website owner also reports that the page is not appearing in their selected AI search observations.

That absence is an initial concern, not evidence of a crawler failure.

Investigation step 1: Examine published crawler policy

The auditor first retrieves the website’s robots.txt file and evaluates the relevant crawler’s effective policy.

For this illustrative scenario, assume the rules permit OAI-SearchBot to access the guide.

The appropriate interim conclusion would be:

Interim conclusion The website’s declared robots.txt policy permits the selected search crawler to access this URL.

It would be premature to conclude that the crawler has successfully received the page.

Investigation step 2: Check ordinary HTTP delivery

A normal unauthenticated GET request is issued from the auditor’s testing environment.

Assume the server returns HTTP 200 with the expected article content.

This establishes successful delivery under the tested client and network conditions.

It does not establish that the actual provider crawler receives the same response.

Investigation step 3: Examine verified crawler activity

Next, the auditor reviews the relevant CDN and WAF logs over an explicitly recorded period.

For this example, assume a request claiming that the OAI-SearchBot identity has been found and that the source IP address is confirmed against the currently published crawler ranges.

Assume further that the security infrastructure records a managed challenge for that request.

The observation now supports a narrower, stronger conclusion:

Interim conclusion The security infrastructure challenged an authenticated crawler request for the selected URL during the observed period.

The original absence of citations remains a separate observation. It cannot be attributed conclusively to the challenge without additional evidence.

Investigation step 4: Identify the responsible security rule

The auditor examines the matched security rule, its intended purpose, and the conditions under which the request was challenged.

Assume the challenge originated from an overly broad automated bot-management rule.

The proposed remediation would be a narrowly scoped adjustment permitting verified search-crawler requests while maintaining protections for unrelated traffic.

The responsible infrastructure owner should approve the change.

Disabling the entire WAF would introduce unnecessary risk and would not be justified by this finding.

Investigation step 5: Verify the approved correction

After the infrastructure change, the auditor should inspect subsequent authentic crawler requests or conduct an approved controlled test supported by the provider’s verification mechanisms.

The remediation should be marked technically resolved only when the original challenge condition is shown to have changed.

If no qualifying request occurs during the available observation window, the appropriate status remains pending verification.

Investigation step 6: Evaluate downstream visibility separately

A later AI citation observation may provide evidence that the page has appeared in the selected AI search experience.

However, a subsequent citation does not automatically establish that the WAF correction caused the result.

Search relevance, the topic being queried, platform retrieval behavior, source selection, and other factors may have changed.

The audit should report technical verification and downstream visibility as separate outcomes.

Illustrative evidence record

The record separates what is illustrative from what is still unverified.

This example shows why an evidence-led audit can establish a technical barrier without claiming to explain every aspect of AI search visibility.

A production case study must replace the illustrative observations with actual, safely recorded evidence.

What is the complete AI search technical eligibility checklist?

A complete technical eligibility checklist should verify platform-specific requirements, content delivery, indexing, discoverability, and observed retrieval conditions, while maintaining an explicit status for each test. The checklist should identify what was examined, why it matters, which platform it applies to, and what evidence is available. Optional practices should never be scored as mandatory eligibility requirements.

The following inventory consolidates the core checks addressed throughout this guide.

A. Discovery and crawler access

Five checks cover whether the crawler policy is readable, applied correctly and verified at the network layer.

B. HTTP delivery and content processing

Five checks cover whether the intended page and its content are actually delivered and readable.

C. Indexing and platform eligibility

Six checks cover Google indexing, directives, canonicals and the site-level generative AI setting, plus platform-specific access.

D. Discovery architecture and interpretation

Five checks cover how pages are found and how clearly their content and entities are described.

E. Visibility and evidence

Five checks cover observed crawler activity, AI exposure, citations and the business metrics kept separate from them.

Each control should use one of five operational states:

Pass, Fail, Inconclusive, Not Tested, or Not Applicable.

Where a control fails, the finding should include a specific explanation and supporting evidence.

Where a result cannot be established, the auditor should retain the inconclusive or not-tested status rather than infer a failure.

Example audit register

Every test starts as Not Tested with pending evidence until a real observation replaces it.

The complete audit should also contain the test date, implementation details, reference documentation, proposed remediation, approval status, retest date, and final verified outcome.

What constitutes an acceptable audit conclusion?

An acceptable conclusion should state the investigated URL or property, applicable platform, documented requirement, observation, evidence source, degree of certainty, and next justified action.

For example, if Google’s URL Inspection tool reports that a noindex directive excludes a target page, and the publisher intends that page to be indexed, the auditor can identify a specific indexing barrier.

The conclusion should not be expanded into the unsupported claim that removing noindex will guarantee AI citations.

The removal would address the documented exclusion condition. Any subsequent indexing or AI visibility outcome must be observed independently.

Frequently asked questions about AI search technical eligibility

What is an AI search visibility audit?

An AI search visibility audit evaluates a website’s readiness and observable performance across relevant AI search experiences. A technical eligibility audit specifically examines discoverability, crawler access, delivery, indexing, preview controls, and platform-specific participation conditions. Actual citations, mentions, and traffic are measured separately because technical eligibility does not guarantee the selection of AI sources.

How do I check whether my website is eligible for Google AI Overviews?

Verify that Googlebot can access the page, that the intended content is available, that Google has indexed the page, and that the page is eligible for a Search snippet. Also inspect the effective Search generative AI inclusion control in Search Console. Meeting these conditions establishes eligibility, not a guarantee of inclusion in AI Overviews or AI Mode.

Does allowing Googlebot make my website accessible to ChatGPT?

No. Googlebot and OpenAI’s OAI-SearchBot operate under different providers and access policies. A successful Google crawl or indexation test does not establish that OpenAI can retrieve the same URL. The audit should independently evaluate OpenAI’s published crawler rules and infrastructure requirements.

Does blocking GPTBot prevent ChatGPT Search citations?

Blocking GPTBot is not equivalent to blocking OAI-SearchBot. OpenAI documents them as separate crawlers with different functions. Search-oriented access should be evaluated through OAI-SearchBot and any relevant user-triggered retrieval behavior. Citation selection remains a separate question.

Is llms.txt required for AI search visibility?

No universal requirement has been established. Google explicitly states that llms.txt is not required for its Search or generative AI features and does not affect Google Search rankings or visibility. The file may be required by other systems that specifically support it, but its absence should not be classified as a Google AI eligibility failure.

Does valid schema markup increase the likelihood of AI citations?

Structured data can help search systems interpret supported information and may enable specific Search features. However, Google does not require special structured data for generative AI Search, and valid markup does not guarantee AI citations. The audit should assess factual consistency and feature-specific validity rather than invent an AI schema score.

How can I tell whether AI crawlers have visited my website?

Inspect relevant web-server, CDN, and WAF logs, then verify crawler identities using the provider’s documented methods. User-agent strings alone are insufficient because they can be spoofed. The absence of qualifying requests in a limited log period does not establish that a provider has never accessed the site.

Why is my website indexed in Google but absent from AI-generated answers?

Google indexation is a prerequisite for eligibility as an AI Overview or AI Mode supporting link, but it does not guarantee source selection. The audit should verify snippet eligibility and the generative AI participation control before considering other explanations such as query relevance, source selection, competition, or insufficient observation data.

Can I measure AI search visibility in Google Search Console?

Yes. As of 2026, Google’s dedicated Generative AI Performance report provides impressions for supported AI Overviews and AI Mode experiences, along with available breakdowns by page, date, country, and device. Report availability and data completeness remain subject to Google’s documented conditions and limitations.

Can I measure AI citations through Bing Webmaster Tools?

Bing Webmaster Tools provides AI Performance reporting for citations across supported Microsoft and partner AI experiences. Citation counts and related metrics describe observed references under Microsoft’s reporting definitions. They should not be interpreted as traditional rankings, complete market share, or direct evidence of conversions.

How often should a technical AI search audit be repeated?

There is no universal interval suitable for every website. Reassessment is justified after significant website migrations, infrastructure changes, crawler-policy updates, indexing incidents, platform documentation changes, or observed access failures. Higher-risk websites may require more frequent checks, but the schedule should reflect operational needs and available verification capacity.

Final analysis: technical eligibility is measurable, but AI citation selection is not guaranteed

AI search visibility should not be reduced to a single checklist score or a collection of unconventional optimization techniques. It begins with established technical conditions that can be investigated directly, including whether intended crawlers can access a website, whether its resources are delivered successfully, whether important pages are eligible for indexing, and whether relevant participation controls permit inclusion.

These conditions do not operate identically across Google, ChatGPT, Claude, Perplexity, and Bing. Google’s generative AI features depend on its Search infrastructure and documented eligibility requirements, while other providers publish distinct crawler identities, access recommendations, and retrieval behaviors. A website may satisfy one platform’s requirements while remaining inaccessible or unverified in another.

The most useful technical audit therefore preserves platform boundaries and evidence quality. A successful HTTP request is reported as an access observation. An indexed page is reported as an indexing observation. An AI impression or citation is reported according to the platform’s actual measurement definition. None of these states should be substituted for another.

This approach also changes how remediation decisions are made. A confirmed crawler block or indexing restriction can be investigated and corrected through an accountable technical process. An optional file or unsupported optimization theory should not be given equivalent priority merely because it is associated with AI search.

A technically eligible page may still be absent from an AI-generated answer because the system has not selected it as useful supporting evidence for that particular request. In such cases, the next investigation may involve topical relevance, factual quality, source credibility, competing evidence, or the limitations of available reporting rather than another change to the website’s robots.txt file.

For website owners and SEO professionals, the practical objective is to remove demonstrated barriers, preserve trustworthy evidence, and measure downstream visibility without inventing certainty.

Eligibility should be verified. Visibility should be observed. Causality should be established only when the evidence permits it.

That distinction is the foundation of a reliable AI search technical audit.

Sources and method

The platform requirements and technical methods in this guide rest on the primary documentation below; the worked example and templates are illustrations, not completed audits.

The following documentation provides the primary basis for the platform requirements and technical methods described throughout this guide.

1. Google Search Central: AI Features and Your Website (opens in a new tab). Indexing, snippet eligibility, and foundational Search requirements.
2. Google: Optimizing for Generative AI Features (opens in a new tab). Technical guidance and clarification of unsupported AI optimization claims.
3. Google Search Console: Search Generative AI Control (opens in a new tab). Site participation settings and inheritance.
4. Google Search Console: Generative AI Performance Report (opens in a new tab). AI impressions and available reporting dimensions.
5. Google: Link Best Practices (opens in a new tab). Crawlable internal links and discovery.
6. Google: Sitemaps Overview (opens in a new tab). Sitemap functions and limitations.
7. Google: Structured Data Guidelines (opens in a new tab). Feature-specific validity and content consistency.
8. Google: Verify Crawler Requests (opens in a new tab). Genuine crawler identification.
9. OpenAI: Web Crawler Documentation (opens in a new tab). Search, training, and user-triggered crawlers.
10. Anthropic: Claude Crawler Documentation (opens in a new tab). Anthropic crawler distinctions and controls.
11. Perplexity: Crawler Documentation (opens in a new tab). Search crawling, user fetching, IP addresses, and WAF recommendations.
12. Microsoft: AI Performance in Bing Webmaster Tools (opens in a new tab). AI citation reporting and metric definitions.
13. Microsoft: Expanded AI Visibility Insights (opens in a new tab)—thematic and comparative AI visibility measurements.

Method This article distinguishes between official platform requirements, technical recommendations, and illustrative diagnostic procedures. No sample crawler observations, visibility results, or remediation outcomes have been presented as completed tests on a production website.