Google AI Overviews reach 90% accuracy, but still generate millions of errors: Analysis

Google’s AI Overviews answered an ordinary factual benchmark accurately 91% of the time in February, up from 85% in October, in response to a New York Instances evaluation with AI startup Oumi.

Nonetheless, Google handles more than 5 trillion searches per year, so which means tens of hundreds of thousands of solutions each hour could also be incorrect.

Why we care. We’ve watched Google shift from linking to sources to summarizing them for greater than two years. This report suggests AI Overviews are enhancing, however nonetheless combine appropriate solutions, weak sourcing, and clear errors in methods that may mislead searchers and reshape which publishers get visibility and clicks.

The main points. Oumi examined 4,326 Google searches utilizing SimpleQA, a broadly used benchmark for measuring factual accuracy in AI methods, the Instances reported. It discovered AI Overviews had been correct 85% of the time with Gemini 2 and 91% after an improve to Gemini 3.

• The larger downside could also be sourcing. Oumi discovered that greater than half of the proper February responses had been “ungrounded,” that means the linked sources didn’t totally assist the reply.
• That makes verification tougher. The reply could also be proper, however the cited pages might not clearly present why.

What modified. Accuracy improved between October and February, however grounding worsened. In October, 37% of appropriate solutions had been ungrounded; in February, that rose to 56%.

Examples. The Instances highlighted a number of misses:

• For a question about when Bob Marley’s house turned a museum, Google answered 1987; the proper 12 months was 1986, in response to the Instances, and the cited sources didn’t assist the declare or conflicted.
• For a question about Yo-Yo Ma and the Classical Music Corridor of Fame, Google linked to the group’s web site however nonetheless stated there was no document of his induction.
• In one other case, Google gave the proper age at Dick Drago’s dying however misstated his date of dying.

Google’s response: Google disputed the Instances evaluation, saying the examine used a flawed benchmark and didn’t mirror what folks truly search. Google spokesperson Ned Adriance instructed the Instances the examine had “severe holes.”

• Google additionally stated AI Overviews use search rating and security methods to scale back spam and has lengthy warned that AI responses can include errors.

The report. How Accurate Are Google’s A.I. Overviews? (subscription required)