Speaking Statistics, measurement and AI

Can Language Models Be Trusted With Clinical Measurement?

An AI that quotes a cutoff from the wrong population gives a confident, wrong answer. Grounding, attribution and refusal matter more than fluency.

Clinicians and patients increasingly ask AI systems what a score means. Dr. Kao, a biostatistician and pain physician, presents two projects that address the problem. MeasureGPT is a psychometric interpretation service that gives AI systems verified severity bands, T-scores and percentiles, and says plainly when no defensible reference population exists. ESTIMAND is a benchmark of grounded psychometric reasoning: whether a model can retrieve, attribute and compute with a measurement fact, recognize when the fact does not exist, and judge whether it applies to the patient at hand.

Attendees will be able to

  1. Show where language models fail when they interpret questionnaires and scores
  2. Explain why a psychometric parameter is only valid in the population that produced it
  3. Demonstrate a grounded approach that serves verified measurement facts to AI systems over the Model Context Protocol
  4. Outline how a benchmark can test attribution, computation and knowing when not to answer

Invite me to give this talkAll talks