How to Benchmark AI Research Tools on Your Own Brief

General-purpose AI tools answer a founder's brief fluently. Fluent is not the same as checked. Here is how to test an output claim by claim, and what a single test can and cannot show.

Pranav Unni, Founder and Lead Verifier of ThriveFinity Pranav UnniFounder and lead verifier
Published
9 minRead time

You asked an AI tool whether your idea will work. It answered in confident, well-organised paragraphs. Now you have to decide how much of it to trust.

Most founders cannot tell from the text alone. Fluent writing looks the same whether the facts underneath are right or wrong. This page sets out how we test an AI research output claim by claim, what published studies say about error rates, and what one test can and cannot tell you.

It is for founders and commercial leads who use ChatGPT, Claude, Gemini or Perplexity for market research and want a repeatable way to check the output. It is a method, not a league table. We publish no scores for named products, and the section on what we withdrew explains why.

Why Most AI Benchmarks Mislead a Founder

Public leaderboards test models on exam-style questions with known answers. A founder asks something else: is this market worth entering, and what could stop me? The answer depends on the right category, the right region, current regulation and live competitors. None of those has a fixed answer key.

There is a second problem, and it sits inside the scoring. OpenAI's research on hallucination, published on 5 September 2025, argues that "standard training and evaluation procedures reward guessing over acknowledging uncertainty". The same piece notes that accuracy-only scoreboards dominate leaderboards and model cards, which pushes developers to build models that guess rather than hold back.

For a founder, that means a confident answer is not evidence of a correct one. A tool that never says "I could not find a source" will often look better on a leaderboard than one that does. The check has to happen on your brief, with sources you can open.

What Published Studies Show About Error Rates

Two large studies give a sense of scale. Both tested news questions, not startup research, so read them as a warning about the general behaviour of these tools, not as a measure of any one product on your brief.

In October 2025, the European Broadcasting Union and the BBC published a study run with 22 public service media organisations in 18 countries and 14 languages. It assessed more than 3,000 answers from four AI assistants. It found that 45% of answers had at least one significant issue, 31% showed serious sourcing problems and 20% contained major accuracy issues.

Share of AI assistant answers with a problem, news questions
Share of AI assistant answers with a problem, news questions
At least one significant issue45%
Serious sourcing problems31%
Major accuracy issues20%

Source: EBU and BBC, 4 AI assistants, 3,000+ answers, 18 countries, 2025. News questions, not startup research. Published 22 October 2025. Chart by ThriveFinity.

Earlier that year, the Tow Center for Digital Journalism at Columbia tested eight AI search tools on 1,600 queries, asking each to identify the source of an excerpt from a news article. Collectively, the tools gave incorrect answers to more than 60% of queries. More than half of the responses from two of the tools cited fabricated or broken links. That study was published in March 2025, so the tools have changed since. The pattern it describes, a confident answer with a source that does not check out, is the one a benchmark should look for.

Tools sold for high-stakes work are not exempt. In May 2024, Stanford researchers tested AI research tools built for lawyers, whose vendors had described them as avoiding hallucinations. The Stanford HAI write-up reports incorrect answers on more than 17% of queries for each of the three tools tested. That is the oldest study here, so treat the exact rate as dated. The lesson holds: a vendor's claim about accuracy is not a test of accuracy.

The cost of an unchecked answer is real for founders. CB Insights found poor product-market fit named in 43% of 431 venture-backed shutdowns since 2023. Harvard Business School's Tom Eisenmann argues in Harvard Business Review that many founders skip researching customer needs before testing products. An AI answer that sounds like research can make that skip easier.

None of these studies tells you how a tool will do on your brief. They show why you need to check.

The Test Protocol

The protocol has seven steps. It takes a few hours for one brief if you are careful, and most of that time is spent on step four.

  1. Fix one brief. Write the brief once: product, buyer, price, market and region. Do not tune it per tool.
  2. Submit it verbatim to each tool with the same settings, and record the date and model version.
  3. Extract every checkable claim. Figures, named competitors, rules, dates and quoted sources. Number them.
  4. Verify each claim against a primary source you can open. Regulator pages, filings, official statistics, vendor price pages. A tool citing a source does not count as verification until you have opened it.
  5. Grade each claim on the evidence scale below, and count the claims that trace to nothing.
  6. List the risks a careful adviser would raise. Mark which tools raised them without being asked.
  7. Repeat on more briefs. A result from one brief is an anecdote.

Keep the raw outputs. If you later publish or act on the result, you will want to show which claim failed, on which date, and against which source. That record is also what lets a reader challenge your result, which is the point.

Six Dimensions Worth Measuring

A useful benchmark scores more than one thing, because tools fail in different ways.

  • Claim accuracy. The share of checkable claims that a primary source supports.
  • Traceable citations. The share of cited facts that lead to a real, dated, accessible source.
  • Source quality. Primary and named sources, against aggregators and unattributed figures.
  • Regulatory completeness. Whether the rules that apply to your sector and market are found and described correctly.
  • Adversarial flags. Weak or false assumptions that the tool raises without being asked.
  • Decision impact. Whether a careful reader would change the decision after reading the output.

Report each dimension separately. A single blended score hides the failure that matters most, which is usually a missed rule or an invented figure. A tool that is right about 18 of 20 facts and wrong about the licensing rule that blocks your launch has failed the test that counts.

One Evidence Scale, A to E

We grade claims on the same scale used in our methodology and in every paid deliverable. The scale describes the evidence behind a claim, not the confidence of the tool that stated it.

  • A · Corroborated. Multiple high-authority, recent, independent sources fully corroborate the claim.
  • B · Authoritative. One or two authoritative, recent sources; no significant contradictions.
  • C · Partial. Credible but older than 12 months, or partial coverage; minor contradictions.
  • D · Weak. Weak sources, significant gaps, or major contradictions present.
  • E · Speculation. Speculation only: no retrieved evidence behind the claim.

Apply the scale to the output of any tool, including ours. Count grade D and E claims. A report where most headline numbers sit at D or E is a draft to research from, not a basis for a decision.

The Category Problem: Is the Benchmark Yours?

The same trap catches the numbers inside a report. A category benchmark is only useful if its population matches yours. A churn rate for consumer apps says little about a B2B service with annual contracts. A failure rate for venture-backed companies says little about a bootstrapped one.

When a tool quotes a benchmark, check four things: who was measured, in which country, over which dates, and how the figure was defined. If the tool does not say, the figure is a grade D claim until you find out. We cover this in The Category Problem: Why Most Benchmarks Are Wrong, and the checks for a market-size or demand figure are in the market validation guide.

What We Withdrew, and Why

In July 2026 we ran one brief through several AI tools and published scores. In October 2026 we withdrew those scores. One brief in one sector cannot support a ranking of named products, and comparative claims need a sample we can defend. We would rather publish a method you can run yourself than a number you cannot reproduce.

If we publish comparative results again, they will state the sample size, the dates, the model versions and the full set of briefs. If we ever correct one, it will appear in our errata history.

Limits of This Method

  • Model output changes between runs and versions. Record both, and expect a second run to differ.
  • Checking claims is slow work. It does not replace a signed review of your decision.
  • Sector and region change the result. Test on briefs from your own market.
  • This method grades evidence. It does not predict whether your business will succeed.
  • The studies quoted above tested news and legal questions. Rates on startup research may be higher or lower.

Where ThriveFinity Fits

If you want a verifier to apply these checks to your offer, that is what the offer validation in a Decision Brief does. The free Offer Signal is automated and unsigned: a directional read with your biggest risk named. The paid Decision Brief is signed by a named person. For how the evidence behind our own verdicts is distributed, see Verdict Distribution.

Spotted something out of date or wrong on this page? Email om@thrivefinity.uk and we respond within 5 working days. Corrections are logged on the errata page.

❓ Common Questions

How do I benchmark an AI research tool?
Give every tool the same brief, word for word. Then check each factual claim in the output against a primary source you can open. Grade each claim on one evidence scale, count what could not be traced, and repeat on more than one brief.
What should an AI research benchmark measure?
Six things: claim accuracy, citations that trace to a real source, source quality, regulatory completeness, whether weak assumptions are flagged without being asked, and whether the output would change a decision. Report each separately.
Why do AI tools state wrong facts so fluently?
OpenAI's September 2025 research argues that standard training and evaluation procedures reward guessing over acknowledging uncertainty. A tool that always answers scores better on an accuracy-only leaderboard than one that says it does not know.
How many briefs do I need before the result means anything?
One brief is an anecdote. Use at least five briefs from different sectors and regions, run each tool on all of them, and report the spread, not a single score.
Does ThriveFinity publish scores comparing AI models?
No. We published a single-brief comparison in July 2026 and withdrew it in October 2026, because one brief in one sector is too small a sample to support a ranking. This page publishes the method only.
Can I run this benchmark on my own idea?
Yes. The free Offer Signal reads your offer and names your biggest risk, so you can compare it with what a general-purpose AI tool told you. It is automated and unsigned, so treat it as a directional read.

Sources

  1. OpenAI. Why language models hallucinate. 5 September 2025.
  2. European Broadcasting Union and BBC. AI's systemic distortion of news is consistent across languages and territories. 22 October 2025.
  3. Columbia Journalism Review, Tow Center for Digital Journalism. We compared eight AI search engines. They're all bad at citing news.. 6 March 2025. Fetched 18 October 2026.
  4. Stanford HAI. AI on trial: legal models hallucinate in 1 out of 6 (or more) benchmarking queries. 23 May 2024, updated 30 May 2024. Fetched 18 October 2026.
  5. CB Insights Research. The top reasons startups fail. 5 March 2026.
  6. Tom Eisenmann. Why Start-ups Fail. Harvard Business Review, May to June 2021.
  7. ThriveFinity. Methodology and evidence scale. Errata.
Pranav Unni

Pranav Unni

Founder · ThriveFinity Connect on LinkedIn →

Pranav Unni is the founder and lead verifier of ThriveFinity. He reads and signs every paid deliverable personally, and writes about go-to-market decisions for established B2B companies.

Share this article

Free · under an hour

Test Your Own Brief

The free Offer Signal gives a directional read on your offer with your biggest risk named. Compare it with what your AI tool said.