Why an AI Visibility Score Needs a Confidence Interval

An AI visibility score looks precise. A dashboard says you are named in 12% of answers, and the number sits there in large type. It is much less certain than it looks.

A visibility score is a count of the answers that name your brand divided by the number of answers you checked. If you checked a few dozen answers, the true rate could sit anywhere in a wide range, so the honest way to report it is the score together with a range. In our own run, 15 mentions in 213 answers is 7.0%, and the 95% range is 4.3% to 11.3%.

This page is for marketing leads and founders who track how AI assistants describe their company, and for anyone buying that tracking from a vendor. It shows the method, a worked example with our real counts, and five rules for reporting. It does not tell you what your own score will be. Our methodology page describes how we run the panel.

Why One Number Misleads

Most AI visibility tools report a single percentage: the share of tracked answers in which your brand is named. That is a sample, not a census. Ask the same question again and the engine may answer differently. Check 20 questions instead of 200 and the figure moves a lot, even if nothing about your brand has changed.

The variation in AI answers is not small. A study by SparkToro and Gumshoe, reported by Search Engine Journal on 30 January 2026, ran 2,961 prompts across ChatGPT, Claude and Google's AI features. It found that repeated runs of the same prompt almost never produced the same brand list. ChatGPT and Google returned the same list less than 1% of the time. The authors also note that the volunteers used their normal tool settings and that the report is not peer-reviewed, so treat it as a strong signal and not a precise rate. The same coverage reports that how often a brand appears was steadier than where it appeared in a list.

Google describes a related source of variation. Its Search Central documentation on AI features says AI Overviews and AI Mode may run several related searches across subtopics while building a response. A different set of searches can bring in a different set of supporting pages. So the answer to one prompt on one day is a single draw from a spread of possible answers.

That changes the useful question. It is not "what is our score". It is "what range of true scores is consistent with what we saw".

The Method We Use: The Wilson Score Interval

For k mentions in n answers, the Wilson score interval gives a 95% range for the true mention rate. Edwin Wilson published it in 1927 in the Journal of the American Statistical Association. In 2001, Brown, Cai and DasGupta compared several methods for a binomial proportion in Statistical Science and recommended the Wilson interval or the Jeffreys prior interval for small samples. It behaves well when the sample is small and when the rate is near 0% or 100%, which is exactly where the simple plus-or-minus formula fails. That simple formula can return a lower limit below zero.

The US National Institute of Standards and Technology makes the same point in its engineering statistics handbook. Its page on a proportion presents the Wilson method, notes that its lower limit cannot be negative, and says the normal-approximation limits may not be accurate enough for some applications.

You do not need to calculate it by hand. The Python library statsmodels offers it as proportion_confint with the method set to wilson, and most spreadsheet or statistics packages have an equivalent. The point is to report it, not to admire the formula.

A Worked Example With Our Real Counts

On 18 October 2026 we asked 50 buyer-style prompts on five assistants: ChatGPT, Claude, Gemini, Perplexity and Google AI Overviews, each with web search. That is 250 requests, and 248 returned an answer. We looked only at generic prompts, the ones that do not name us, such as "which consultancy should a UK B2B company hire for go-to-market work". We asked each prompt once per surface.

Read it like this. Google AI Overviews named us in 5 of 41 answers and Gemini in 1 of 43. It is tempting to say AI Overviews is five times better for us. The ranges say otherwise: 5.3% to 25.5% overlaps 0.4% to 12.1%. The data is consistent with a real difference, and it is equally consistent with none. We treat it as a lead to test in later runs, not a finding.

A zero behaves the same way. Our comparison-style prompts that do not name us returned us in 0 of 39 answers. That does not mean never. With 39 answers, the true rate could still be as high as 9.0%.

Two of the 50 requests to Google AI Overviews failed with a data-provider error, so that surface has 48 answers in total, and 41 of those came from the generic prompts in this table. Failures are counted and reported, not quietly dropped.

How Sample Size Changes the Range

Here is the same observed rate, 20%, with different numbers of answers behind it.

Quadrupling the sample roughly halves the width. A dashboard that tracks 10 prompts on one engine can show 20% and be consistent with anything from 6% to 51%. If a vendor gives you a single score and does not tell you how many answers sit behind it, you cannot judge how much to trust it.

This also shows why more engines do not automatically help. Five assistants with 43 answers each give you 213 answers in total, but each surface on its own is still a small sample. The overall range is tighter, the per-surface ranges are not.

Five Rules We Apply

1. Report the count and the range with every rate. For example: "7 of 101 (6.9%, 95% range 3.4% to 13.6%)". The count lets a reader check your sums.
2. Split branded from unbranded prompts. A prompt that names you will mention you, and mixing it in inflates the score. In the same run the five brand prompts returned us in 24 of 25 answers (96.0%, range 80.5% to 99.3%). Counting them with everything else would lift the headline to 19.8% (49 of 248). We do not quote that as visibility.
3. Count failures. Answers that errored are removed from the denominator and reported separately. Here that was 2 of 250 requests.
4. Do not compare weeks after the prompt set changes. A new prompt set is a new baseline, not a decline or a gain.
5. Call a change real only when the ranges stop overlapping across repeated runs. Even then, say observed or associated, not caused.

Reading a Change From One Week to the Next

Suppose your tool shows 8% last week and 11% this week. With a few dozen answers behind each figure, the two ranges overlap heavily and the movement is consistent with no change. That is not a reason to ignore the tool. It is a reason to wait for more runs before telling a board that visibility is rising.

There are three honest ways to move from a lead to a supported finding. Run the same locked prompts more often, so the count behind each figure grows. Add prompts that mean the same thing in different words, so one phrasing does not drive the result. Or look for the same direction of change on more than one surface. If all three agree over several weeks, you can say the data is consistent with a real change. You still cannot say what caused it, because AI engines change their models, their sources and their ranking logic without notice.

This is why we run our own prompt set weekly and compare each run with a fixed baseline, using the ranges described above. Our first baseline and the questions it answers are in How to Run an AI Visibility Check on Your Company, and the way buyers meet a company in AI assistants is covered in AI Visibility for B2B Brands. For how a share of voice is defined in Google's AI Overviews specifically, see our share-of-voice guide.

What to Ask a Vendor That Reports a Score

If you are paying for AI visibility tracking, five questions make the number honest. How many prompts and how many answers sit behind each figure? Are branded prompts separated from unbranded ones? Is there a range, and how is it calculated? How are failed answers handled? What happens to the history when the prompt set changes? A vendor that can answer all five is measuring. One that cannot is decorating. Our buyer's guide to choosing a GEO agency has the wider list of questions to ask.

What This Does Not Cover

• This is one brand, one pass per prompt and surface (each prompt asked once), five assistants and one day.
• It does not say how often engines change their answers over time. Repeated runs are what would show that.
• It does not measure whether being named leads to enquiries. A mention is not a sale.
• The ranges describe sampling noise. They do not cover bias in which prompts we chose, which is a separate risk. The prompt set and its rules are on the methodology page linked above, and any error we find is logged on the errata page.

What to Do Next

Run the free AI Visibility Check for a first reading, then repeat it a week later with the same questions. Treat the first number as a starting point. If you would like a verified, repeated measurement with ranges and a named person signing the result, that is what our AI Visibility service provides.

Spotted something out of date or wrong on this page? Email om@thrivefinity.uk and we respond within 5 working days.

Pranav Unni

Founder · ThriveFinity Connect on LinkedIn →

Pranav Unni is the founder and lead verifier of ThriveFinity. He reads and signs every paid deliverable personally, and writes about go-to-market decisions for established B2B companies.