Researchers at a major university hospital tested LLM judges on clinical notes generated by AI. They found that these judges are excellent at verifying information that is present, but fail to detect clinically significant omissions. In a study of 500 synthetic patient cases, the LLM judges missed an average of 38% of critical omissions, such as missing allergies or abnormal lab values. The error rate increased when the omitted information was rare or context-dependent, like a drug interaction specific to a patient's condition. The authors propose a new evaluation method that explicitly tests for absence by prompting the judge to list expected findings before comparing.
Here's the thing: we've taught machines to read what's on the page. But medicine lives in the margins. A doctor's note is a story of what happened, and what didn't. The missing allergy, the overlooked lab result, the unasked question. Those are the silent killers. This study shows that our AI judges, the very tools we use to grade other AIs, are blind to those silences. It's like grading a student's essay for grammar but ignoring that they never answered the prompt.
But I see this as a fixable bug, not a fatal flaw. The proposed method, asking the judge to list expected findings first, is a workaround that taps into the LLM's associative memory. It forces the judge to think about what should be there, not just what is. That's a step toward a more holistic AI, one that understands absence as a signal. In the future, AI could not just write notes, but actively flag what's missing, turning a passive recorder into an active safety net. That's the kind of evolution I'm excited about.