A simulation study of the model evaluation criterion mmre
Top Cited Papers
- 17 November 2003
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 29 (11), 985-995
- https://doi.org/10.1109/tse.2003.1245300
Abstract
The mean magnitude of relative error, MMRE, is probably the most widely used evaluation criterion for assessing the performance of competing software prediction models. One purpose of MMRE is to assist us to select the best model. In this paper, we have performed a simulation study demonstrating that MMRE does not always select the best model. Our findings cast some doubt on the conclusions of any study of competing software prediction models that use MMRE as a basis of model comparison. We therefore recommend not using MMRE to evaluate and compare prediction models. At present, we do not have any universal replacement for MMRE. Meanwhile, we therefore recommend using a combination of theoretical justification of the models that are proposed together with other metrics proposed in this paper.Keywords
This publication has 35 references indexed in Scilit:
- Software cost estimation with incomplete dataIEEE Transactions on Software Engineering, 2001
- Using public domain metrics to estimate software development effortPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2001
- On the problem of the software cost functionInformation and Software Technology, 2001
- An Empirical Study of Analogy-based Software Effort EstimationEmpirical Software Engineering, 1999
- Software Metrics Data Analysis—Exploring the Relative Performance of Some Commonly Used Modeling TechniquesEmpirical Software Engineering, 1999
- The predictive validity criterion for evaluating binary classifiersPublished by Institute of Electrical and Electronics Engineers (IEEE) ,1998
- A comparison of software effort estimation techniques: Using function points with neural networks, case-based reasoning and regression modelsJournal of Systems and Software, 1997
- Experience with the accuracy of software maintenance task effort prediction modelsIEEE Transactions on Software Engineering, 1995
- Parametric estimation of programming effort: An object-oriented modelJournal of Systems and Software, 1991
- Calibrating a software cost estimation model: why and howInformation and Software Technology, 1987