Speaking Statistics, measurement and AI
Can Language Models Be Trusted With Clinical Measurement?
An AI that quotes a cutoff from the wrong population gives a confident, wrong answer. Grounding, attribution and refusal matter more than fluency.
Clinicians and patients increasingly ask AI systems what a score means. Dr. Kao, a biostatistician and pain physician, presents two projects that address the problem. MeasureGPT is a psychometric interpretation service that gives AI systems verified severity bands, T-scores and percentiles, and says plainly when no defensible reference population exists. ESTIMAND is a benchmark of grounded psychometric reasoning: whether a model can retrieve, attribute and compute with a measurement fact, recognize when the fact does not exist, and judge whether it applies to the patient at hand.
Attendees will be able to
- Show where language models fail when they interpret questionnaires and scores
- Explain why a psychometric parameter is only valid in the population that produced it
- Demonstrate a grounded approach that serves verified measurement facts to AI systems over the Model Context Protocol
- Outline how a benchmark can test attribution, computation and knowing when not to answer