Generative AI Search

Generative AI transforms internal search from a list of links into a synthesized answer, drawn from and citing multiple sources at once. It supports genuinely conversational, multi-turn search, where follow-up questions build on what came before, and it can layer personalized recommendations directly into a generated response rather than presenting them as a separate feature.

This power comes with real, current limitations that any deployment needs to account for. Hallucination is the risk of the system confidently stating something plausible-sounding but incorrect or entirely unsupported by its sources. Source obscurity is the difficulty of knowing exactly which source backs which specific claim inside a synthesized answer. Bias reinforcement means whatever biases exist in the underlying training data can surface in the generated output. And there's a broader reliability trade-off between how fast and fluent a generated answer feels and how confident anyone can be that it's actually accurate.

A generative search answer reads with the same confident, fluent voice whether it's exactly right or completely wrong, which is precisely what makes its failures so much harder to notice than a broken link or an empty results page ever was.

Exercise

The scenario: A medical information website has implemented generative AI search that synthesizes responses to health queries from multiple sources on the site, presenting a single generated answer with citations. Three real interactions with the system are documented: it confidently answered a question about drug interactions with a plausible-sounding but factually unsupported response; it failed to indicate that a particular topic fell outside its reliably sourced knowledge, presenting a thin answer with the same confident tone as everything else; and its synthesized response on a treatment topic reflected a bias present in its training data, presenting one treatment approach as more established than the evidence actually supports.

Which limitation does the confident-but-unsupported drug interaction response represent?

Hallucination
Source obscurity
Bias reinforcement

Of the three risks illustrated in this scenario, hallucination, failing to signal low confidence, and bias reinforcement, which is most consequential specifically in a medical information context, and why? Then propose two specific design interventions that would reduce the most serious risks without abandoning the generative AI implementation entirely.

Feedback

The drug interaction response is a hallucination. A confident, plausible-sounding answer that isn't actually supported by the underlying sources is exactly what hallucination describes, the system generated something that reads correctly without the substance to back it up.

In a medical information context specifically, hallucination is the most consequential of the three risks illustrated here. Confidently stated but incorrect drug interaction information isn't just an inconvenience: it's the kind of error that could directly and immediately affect someone's health decisions, in a domain where the cost of being confidently wrong is dramatically higher than it would be on, say, a recipe site or a general news search. The failure to signal low confidence on a thin-knowledge topic is a closely related and serious risk, since it compounds the hallucination problem by giving users no cue to be more skeptical. Bias reinforcement is genuinely serious too, but its harm tends to unfold more gradually, through skewed understanding over time, rather than through a single acute, confidently wrong answer someone might act on immediately.

Two reasonable design interventions: first, make the system explicitly signal its confidence level and flag when a topic falls outside strong source support, rather than presenting every answer in the same uniformly confident tone, particularly critical for anything touching drug interactions or dosing. Second, make source attribution far more granular and visible, so each specific claim in a synthesized answer links clearly back to the exact source supporting it, directly addressing source obscurity and giving users, and clinicians reviewing the system, a way to verify individual claims rather than trusting the answer as an undifferentiated whole. Neither intervention requires abandoning generative search; both make its real limitations visible instead of hidden behind a uniformly confident voice.