Last week, I successfully defended my PhD dissertation. This episode is a short overview of my research on using LLMs like ChatGPT to assess threats of violence.
My Dissertation: LLMs Versus Human Experts: Mixed Methods Analysis Measuring Variance in School Shooting Threat Assessments
Abstract: This study measures the unwanted variability in expert judgments by testing six frontier large language models (LLMs) on fictitious but realistic school shooting scenarios derived from 1,000 real threats made in the United States (note: there are approximately 100,000 threats made nationwide to schools each year).
Using prior decision science research, this dissertation measures the accuracy and consistency of assessments by LLMs compared to a prior study of 245 human law enforcement officers who rated the severity of the same six threat scenarios. Quantitative results demonstrate that the LLMs produced severity ratings within one point of the mean of ratings by human experts (ΔM ≤ 1) with no statistically significant differences (p > .05), supporting the primary hypothesis that LLMs can approximate the accuracy of human threat assessments. Aggregate LLM scores displayed lower variance than human ratings, showing both a wisdom-of-crowds effect and reduced judgment noise. Qualitative thematic analysis of narrative explanations revealed that LLMs consistently focused on specific aspects of the threats, while humans were influenced by lived experiences, adherence to formal procedures, and personal assumptions.
The findings suggest that LLMs can enhance the reliability of school-based threat assessments as part of a human-LLM hybrid team, or as the sole assessor in under-resourced or rural schools that lack trained human experts. This study contributes to the fields of behavioral economics, decision science, violence prevention, and serves as a framework for comparing the assessment abilities of LLMs to those of humans.
Freakonomics/People I (Mostly) Admire Interview (2021): Daniel Kahneman on Why Our Judgment is Flawed — and What to Do About It
David Riedman, PhD is the creator of the K-12 School Shooting Database, Chief Data Officer at a global risk management firm, and a tenure-track professor. Listen to my weekly podcast—Back to School Shootings—or my recent interviews on Freakonomics Radio and the New England Journal of Medicine.
