Talk by Prof. James Soland
How Measurement is Affecting Evaluation Results: Evidence from 100 Randomized Control Trials
February 23, 2026, 14:30 h
Andreasstrasse 15, 8050 Zurich, AND 4.55/57 (4th floor)
There is much evidence that measurement decisions, such as how an assessment is designed and scored, can affect program evaluation results, including from randomized control trials (RCTs). While many studies on how scoring decisions affect evaluation results suggest that substantial bias can be introduced into treatment effect estimates based on seemingly trivial measurement choices, the majority of that research relies on Monte Carlo simulation. In this set of papers, we examine how sensitive results of actual RCTs are to measurement choices like how to score the dependent variable. We conduct these analyses using item level data from over 100 actual RCTs. Initial results suggest that, in many cases, treatment effect estimates are highly sensitive to decisions like whether to use sum scores or not, and that many basic measurement assumptions (e.g., whether the outcome is unidimensional) are not met. We further show that participating in a treatment, even when blinded, can change how individuals interpret survey measures.