A curious question from my kids sent Gemini into a hallucination.
Google's AI Overview can be 100% wrong, even when SERP is right.
Google’s AI Overview can be 100% wrong, even when SERP is right.
It could be a trap when you least expect it.
Today, my kids asked me, “What is Buah Long Long?” I quickly Googled it to show them images. The initial search was good: both the AI Overview and SERP correctly identified “Buah Long Long” as Ambarella Fruit.
However, when I tapped on “Benefits of Buah Long Long,” the AI Overview hallucinated, confidently stating that Buah Long Long is also known as longan. This led to a cascade of completely wrong information.
Ironically, the SERP results for “Benefits of Buah Long Long” were accurate. AI Overview is supposed to be grounded in Google Search, so why was it so far off?
This isn’t just about fruit; it’s a critical reminder about the reliability of information, especially from AI summaries.
I tested the same prompt, “Benefits of Buah Long Long,” on Perplexity, and it had no issues. This incident, combined with my past experiences, indicates that Gemini still exhibits the worst hallucinations among frontier LLMs.
My takeaway: Always, always, always double-check facts presented by AI Overview.
What are the worst hallucinations you’ve encountered recently?
Technical observation:
It hallucinates when I least expect it. But here’s an interesting observation: Buah Long Long and Longan both contain the word “Long”. Could this be related to tokenization?
#gemini #googleai #ai #google #artificialintelligence #chatgpt #perplexity #openai #genai #llm #hallucinations
Tap to expand
Enjoyed this? Subscribe for more.
Practical insights on AI, growth, and independent learning. No spam.
More in AI Agents
When AI Hallucination Becomes A Security Feature.
Two months ago, something unexpected happened with our AI Lead Response agent.
Recently, I heard an interesting view from Jeremy Tan during a panel discussion.
It sounded absurd at first, but on second thought, I think it might actually happen.
How we ranked #1 for "Xiaohongshu Marketing" in 90 days and generated $300K+ revenue since then.
Last year, while visiting China, Terrence Ngu Teck Kong found a new business opportunity for our marketing agency, Xiaohongshu Marketing.
No, Karpathy Didn’t Say Vibe Coding Doesn’t Work
This starkly contrasts with my own experience.
Business+AI Lab with AWS: Event Recap
Held at AWS Singapore, this hands-on GenAI Lab gave participants the rare opportunity to build and deploy their own enterprise-grade chatbot, in under 3 hour...
Could we have been wrong about zero click impressions?
For a while now, we’ve been blaming AI Overviews for the rise in zero click searches. But maybe that’s only half the story.
When AI Hallucination Becomes A Security Feature.
Two months ago, something unexpected happened with our AI Lead Response agent.
No, Karpathy Didn’t Say Vibe Coding Doesn’t Work
This starkly contrasts with my own experience.
Could we have been wrong about zero click impressions?
For a while now, we’ve been blaming AI Overviews for the rise in zero click searches. But maybe that’s only half the story.
Recently, I heard an interesting view from Jeremy Tan during a panel discussion.
It sounded absurd at first, but on second thought, I think it might actually happen.
How we ranked #1 for "Xiaohongshu Marketing" in 90 days and generated $300K+ revenue since then.
Last year, while visiting China, Terrence Ngu Teck Kong found a new business opportunity for our marketing agency, Xiaohongshu Marketing.
Business+AI Lab with AWS: Event Recap
Held at AWS Singapore, this hands-on GenAI Lab gave participants the rare opportunity to build and deploy their own enterprise-grade chatbot, in under 3 hour...