Vague complaints about AI-search visibility can hide two very different problems: a page may be blocked, unreachable, or ineligible to be shown, or it may be reachable and eligible but not worth citing. The two need different diagnostics, and mixing them produces advice nobody can act on. This page is about drawing the line before you write any test.

Evidence note — sources

Claims below are attributed to Google Search Central and OpenAI developer documentation, linked inline. Where this page reasons beyond those documents, it says so. Nothing here is a diagnosis of a particular site, and this publication publishes no verification results or certification.

Start with access, because it has the cleanest answers

Access questions are usually binary and checkable. Google states that to be eligible as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, meeting the Search technical requirements — and that there are no additional technical requirements for these features (Google, AI features). Google adds that meeting every requirement does not guarantee crawling, indexing, or serving.

That single paragraph reorders most investigations. If the page is not indexed, or not snippet-eligible, content quality is not the first question. It is a second question you may not have reached.

robots.txt is narrower than people assume

Google describes robots.txt as telling crawlers which URLs they can access, used mainly to avoid overloading a site with requests, and states plainly that it is not a mechanism for keeping a page out of Google (Google, robots.txt introduction). To keep a page out of results, Google points to noindex or password protection.

The same document notes that a page disallowed in robots.txt can still be indexed if other sites link to it, with the URL and possibly other public information such as anchor text still appearing in search. It also warns that rules cannot enforce crawler behaviour and that different crawlers may read syntax differently.

Inference — not a documented claim

Reasoning from those two points: a site owner who disallowed a path to “remove” it from AI answers may have achieved neither removal from index nor removal from snippets, while losing the crawl that would let the content be evaluated at all. That is an access change with an ambiguous content effect, which is exactly the situation this guide exists to untangle. It is a plausible reading of the documentation, not a statement Google makes.

AI-search crawler controls are per-bot and independent

OpenAI documents OAI-SearchBot and GPTBot robots.txt tags so webmasters can manage how their sites work with AI, and states each setting is independent of the others (OpenAI, crawlers overview). OAI-SearchBot is for search; sites opted out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. GPTBot governs crawling that may be used in training.

OpenAI also notes roughly 24 hours for systems to adjust after a robots.txt update, and that ChatGPT-User handles user-initiated actions rather than automatic crawling, so robots.txt rules may not apply to it. For Search opt-outs, OpenAI directs site owners to OAI-SearchBot.

Practical consequence

A change intended to affect one surface may have altered another. Record each bot's directive separately, with the date, before you interpret any change in what appears.

Now the content question, asked in Google's own terms

Once access is settled, Google frames quality around who created content, how it was created (including AI or automation disclosures where reasonably expected), and why — with the “why” ideally being to help people (Google, helpful content). Google says its systems are designed to prioritise helpful, reliable information rather than content made to manipulate rankings, and that trust is the most important aspect of E-E-A-T, while stating E-E-A-T is not itself a specific ranking factor.

That document also states that fabricating creator profiles — AI-generated headshots, made-up names, false credentials — is deception and a signal of a low-quality page. This publication's refusal to present a fabricated consultant biography is a direct application of that guidance, not a marketing position.

Marginal checklist: sort the claim before testing

  • Is the specific URL indexed, or is the claim based on a search you ran yourself?
  • Is the page eligible to show a snippet, or has something limited that?
  • Which bots are addressed in robots.txt, for which paths, since when?
  • Could a CDN or hosting layer block the crawler independently of robots.txt? Google lists this as a best practice to check (AI features).
  • If access is clear, what is the page's stated purpose, and does the visible text carry the answer?
  • Is the claimed problem actually about a different query, page, or audience than the one named?

If more than two of these cannot be answered from existing evidence, the honest next step is not a content rewrite. It is a small access check first.

Move from sorting to testing

Sorting a claim into access or content is preparation, not proof. The next step is a test small enough that a result changes one belief. Continue with Write a small test with an interpretable result, or start earlier with Frame an AI-search diagnostic. To record known pages, the claimed issue, existing evidence and the comparison, use the private worksheet at /diagnostic-brief/ — it is a planning tool, not a monitoring dashboard, and this site does not require Search Console access.

When the sorting is done, decide what the person acting next actually needs, covered in Make an advisory handoff useful.