Comparison of model predictions with measurements: A novel model-assessment method

Abstract

Frequently, scientific findings are aggregated using mathematical models. Because models are simplifications of the complex reality, it is necessary to assess whether they capture the relevant features of reality for a given application. An ideal assessment method should (1) account for the stochastic nature of observations and model predictions, (2) set a correct null hypothesis, (3) treat model predictions and observations interchangeably, and (4) provide quantitatively interpretable statistics relative to precision and accuracy. Current assessment methods show deficiencies in regards to at least one of these characteristics. The method being proposed is based on linear structural relationships. Unlike ordinary least-squares, where the projections from the observations to the regression line are parallel to the y-axis and inverse regression where they are parallel to the x-axis, the generalized projection regression method (GePReM) projects the observations on a regression line in a direction determined by the ratio of the precision of the observations to that of the mathematical model predictions. Estimation and testing issues arise when the model is expressed in the common slope-intercept format. A polar transformation circumvents these issues. The parameter for the angle between the regression line and the horizontal axis has symmetrical confidence intervals and is equivariant to the exchange of X and Y. The null hypothesis for the equivalence test is that the model predictions are not equivalent to the observations. Information size is calculated as the simple ratio of the variance of the true values of the observations and of the computer model predictions divided by their respective precision. This information size plays a critical role in determining the number of observations required and the size of the zone of practical tolerance for the equivalence tests. The terminology used in the comparison of measurement methods is adapted to that of model assessment based on the equivalence tests on the relative precision, regression slope, and mean bias. Two examples are presented, with complete details of the calculations required for parameter estimation, equivalence tests, and confidence intervals. The assessment method proposed is an alternative to other assessment methods available. Further research is required to establish the relative benefits and performance of this proposed method compared with others available in the literature.