Statistical Considerations in Medical Decision Models

Abstract

The influence of sampling error on decision-analytic models was investigated to determine how these errors affect model reliability. Formulas were developed to relate statistical error in the probability decision threshold and gain in expected utility to the error in the data samples upon which such models are based. The formulas were validated in a simulation experiment and then applied to a hypothetical decision model and to the clinical problem of immediate surgery versus continued observation in suspected acute appendicitis. The results of this analysis show that modest statistical error affecting any variable in a decision model may be amplified into a substantially larger error in both the probability decision threshold and the gain in utility predicted by the model. In addition, when errors are present simultaneously in several variables, they may compound to unexpectedly large magnitudes, rendering the model unreliable over a wide range of disease probability. The interpretation of the results of a decision analysis should be viewed along a continuum that takes into account both the magnitude of the gain or loss in expected utility predicted by the model and a quantitative measure of the reliability of this prediction. Whenever possible, a determination of statistical error should be an integral part of any formal decision analysis.