How to measure AI search visibility, and tell real change from noise

Most AI visibility numbers you'll see in a deck are one answer, screenshotted. Someone asked ChatGPT a question, the brand came up (or didn't), and that became the headline. It's the equivalent of judging your SEO by one search on one afternoon from one laptop, except worse, because AI answers vary far more than search results do.

Measuring this honestly is not hard, but it has four parts: the right questions, enough answers, the right counts, and a way to tell a change from chance.

Why one answer isn't a measurement

Assistants write a new answer every time. Repeat one buying question to ChatGPT a few times and the lists won't match: the same two leaders, then a rotating cast below them. Gemini and Perplexity behave the same way, and Google's AI Overviews change with the searches they fan out to. Google says its AI features may run "query fan-out", several related searches behind one question, and which searches run, and which pages come back, moves from day to day.

So a single answer tells you whether you can be named, not how often you are. The number that matters is a proportion: across many answers to one question, how many name you. That rate is what moves when your work moves the market, and it's the only thing worth reporting upward.

Step 1: choose the questions

Measure the questions your buyers actually put to assistants, in their words, not your keywords. A useful set per product line is 20 to 30 questions, split by intent:

• Buying questions: "which tool should a small team use for X", "cheaper alternatives to Y". These decide revenue.
• Preparing questions: "how do I do X", "what should I look for in Y".
• Fact questions: "what is X", "does Y integrate with Z".

Keats calls these question tiers, and weights buying questions most. Choosing the questions to track covers where to find them and which phrasings to throw out.

Step 2: sample each question

Put every question to every assistant you care about several times in a week. One answer per question per week gives you a coin flip; five or more starts to give you a rate. More questions with fewer samples each is usually better than a few questions sampled heavily, because a whole set averages out the quirks of any one phrasing.

Keep everything else constant: the same wording, no account history steering the answer, the same country. Changing the wording mid-stream resets the baseline.

Step 3: count three things

For each question, and then for each tier, record:

Read the same three for your two or three closest rivals on the same answers. "We're named in 40% of buying answers" means little alone. "We're named in 40%, the leader in 70%, and the gap closed by 8 points this month" is a plan.

If you want one headline number, weight the tiers and combine them, which is what an AI visibility score does. Keep the three counts visible under it; a score that rises because fact questions rose while buying questions fell is not good news.

Step 4: put a noise band on every change

Because the rate comes from a sample, it wobbles even when nothing changes. A noise band is the range a number would move by chance, given how many answers sit behind it. Forty answers give a wide band; a few hundred give a narrow one.

The rule is simple: a week-on-week change inside the band is not a change. Don't celebrate it, don't panic about it, and don't let anyone build a story on it. Read the trend over several weeks, and act on moves that clear the band.

In Keats it looks like this, on Prepco, a fictional brand from the demo: each week's number with its change and its band.

Expand

Step 5: separate your work from the market

A real rise still isn't proof that your work caused it. Assistants update their models, rivals publish, a roundup adds you. The way to tell is control questions: tracked questions your change didn't aim at. If your targets rose and the controls stayed flat, the work is the likely cause. The method, with its day-21 and day-35 reads, is in proving AEO results.

What to report to your boss

One sentence, three numbers, and the work:

1. The week in a sentence. "We're named in 46% of buying answers on ChatGPT and Gemini, up 9 points in a month, outside the noise band; the leader is at 61%."
2. Named, named first and cited, each with its change and band.
3. What shipped and its verdicts: the pages you changed, the questions they targeted, and whether those moved more than the controls.

Leave out single answers, unless one shows a wrong fact that needs fixing. And when the numbers are flat, say so: flat inside the band is a real result. Agencies can use the same structure for clients; AEO for agencies adds the monthly cadence.

Where traffic fits

Visibility and traffic are different measures. Many people read the answer and never click, so a rise in mentions can come with no rise in visits. Track AI referral traffic in GA4 as well, but don't use it as the visibility number: it counts the clicks, not the answers.