Independent diagnostic notebook · AI search
Find out what is actually blocking your pages from AI search answers
Before you rewrite anything, separate the problems: a page that cannot be crawled, a page that cannot be indexed, a page that is indexed but thin, and a page that is fine but simply not chosen. Each one has a different fix and a different owner. This notebook helps you name the problem with evidence, run one small test that can actually be read, and hand the finding to whoever acts next.
Open the private diagnostic worksheet Or start with how to frame the question
01 · Orientation
Start from the claim, not the tactic
An AI-search conversation can begin with a remedy — add schema, add an llms.txt file, add a special crawler file. Google's own documentation on AI features in Search says the opposite: the same search fundamentals apply to AI Overviews and AI Mode, and there are no additional requirements or special optimizations to appear in them. Google also states you do not need to create new machine-readable files, AI text files, or special schema.org structured data for those features. What matters is that a page is indexed and eligible to be shown in Search with a snippet, meeting the ordinary technical requirements.
That is a narrow, testable claim. So the first job is to write down what is being claimed about your site, by whom, and on what evidence. A vague worry (“we are not in AI answers”) is not a diagnostic. A stated claim (“our comparison pages do not appear in AI Overviews for product queries”) can be checked, and can fail for reasons you can list.
Three questions to settle before any test
- Which pages? Name exact URLs. “The whole site” is not a sample you can inspect.
- For whom? Name the audience and the kind of query they would type. A page cannot be judged helpful without a person it helps.
- Compared to what? Which pages, competitors or queries are you comparing against, and why is that comparison fair?
These are the first fields of the diagnostic worksheet, which stays private to you. The fuller reasoning about what a diagnostic can and cannot establish is in Frame an AI-search diagnostic.
02 · The main fork
Separate access problems from content problems
A page can be missing from an answer because a crawler was never allowed in, or because the page is in and simply did not earn the citation. These are different failures and they are fixed by different people. Mixing them produces the familiar loop of rewriting copy while a robots rule quietly blocks the crawl.
Access, in the order it fails
- Crawl. A
robots.txtfile tells crawlers which URLs they can access. Google describes it mainly as a way to avoid overloading a site with requests, and states plainly that it is not a mechanism for keeping a web page out of Google. - Index. Disallowing a URL in robots.txt does not remove it from results. If other sites link to it, the URL — and potentially other publicly available information such as anchor text — can still appear. Blocking the crawl is not the same as blocking the listing.
- Display limits. Directives such as
nosnippet,data-nosnippet,max-snippetandnoindexcan limit what is shown from a page in Search. If a snippet cannot be shown, a supporting link in an AI feature is not the next thing to fix. - Third-party crawlers. OpenAI documents separate robots.txt tags — OAI-SearchBot and GPTBot — so site owners can manage how their content works with AI, and states each setting is independent of the others. Sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links; OpenAI says allowing about 24 hours after a robots.txt change for its systems to adjust.
- User-initiated fetches. ChatGPT-User is documented for user-initiated actions rather than automatic crawling, and robots.txt rules may not apply to it. OpenAI directs site owners to OAI-SearchBot in robots.txt for managing search opt-outs and automatic crawl.
Two cautions belong here. Other crawlers may interpret robots.txt differently or ignore it, so a rule is not a guarantee of behavior anywhere except where the operator says so. And rules only take effect if they are visible to the crawler and the page is recrawled — an edit is not an observation.
Content, once access is clear
If the page is crawlable, indexable and snippet-eligible, the question becomes whether it deserves the citation. Google describes its automated ranking systems as designed to prioritize helpful, reliable information created to benefit people rather than content made to manipulate rankings, and calls trust the most important aspect of E-E-A-T — while stating that E-E-A-T is not itself a specific ranking factor. Google recommends asking who created the content, how it was created — including disclosure of AI or automation where reasonably expected — and why it was created, with the “why” ideally being to help people.
One documented warning is directly relevant to consultants. Google states that fabricating creator profiles — AI-generated headshots, made-up names, false credentials — is a form of deception and a signal of a low-quality page. This notebook therefore carries no invented practitioner biography, and neither should the pages you are diagnosing.
The full separation procedure, including what to record at each step, is in Separate access problems from content problems.
03 · Method
Write a small test with an interpretable result
An uninterpretable test is worse than none, because it produces confidence without information. The fix is to decide, in advance, what each possible result would mean. Google notes that AI Overviews and AI Mode may use a query fan-out technique — issuing multiple related searches — and that the two features may use different models and techniques, so their responses and links vary. Any test that assumes a single fixed pipeline will misread noise as signal.
- State one change. A test with three simultaneous changes cannot tell you which one did anything.
- State the observable. Which feature, which query, which page, and what appearance or absence counts. Record answers as observed, not as remembered.
- State what would falsify it. If the page stays absent after the change and the access checks pass, that is a content or relevance result, not an access result.
- Keep the window honest. Where a fix depends on recrawling, note the time of the change. For robots.txt changes, OpenAI describes roughly 24 hours before its systems adjust; Google states indexing and serving are not guaranteed even when requirements are met. Absence shortly after a change proves little.
- Separate measurement from this site's own practice. Google documents that pages appearing in AI features are counted in overall search traffic and reported in the Performance report under the “Web” search type. That is a description of Google's reporting; this notebook does not require or use Search Console for you.
Worked sequencing and the failure modes of over-broad tests are set out in Write a small test with an interpretable result.
04 · Transfer
Make an advisory handoff useful
This is an investigation and advice practice, not a delivery team. The output that matters is a finding someone else can act on without you in the room: what was checked, what was observed, what remains unknown, and what the next owner should test. A recommendation without the observation behind it is a guess wearing a recommendation's clothes.
What the handoff should contain
- The named pages and the audience they serve.
- The original claimed issue, quoted as claimed.
- The evidence actually gathered, with dates and the feature observed.
- The access findings: crawl, index, display limits, third-party rules.
- The content judgment, marked as a judgment.
- One next test, with its falsifying condition.
What it should not contain
- An invented expert, biography or certification.
- Client names, results or service history that cannot be sourced.
- “We optimized for AI” with no described change or observation.
- A promise of indexing or inclusion. Google states neither is guaranteed.
The handoff fields sit at the end of the private diagnostic worksheet, alongside the page list, audience, claimed issue, existing evidence, comparison and next test. Guidance on writing the transfer itself is in Make an advisory handoff useful.
Every factual statement in this notebook is drawn from primary documentation — Google Search Central on AI features, robots.txt and helpful content, and OpenAI's crawler documentation. Limits and qualifications are stated where the sources state them. This site reports no verification results, holds no certification, and presents no fabricated expert identity.