A critique of software defect prediction models
- 1 January 1999
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 25 (5), 675-689
- https://doi.org/10.1109/32.815326
Abstract
Many organizations want to predict the number of defects (faults) in software systems, before they are deployed, to gauge the likely delivered quality and maintenance effort. To help in this numerous software metrics and statistical models have been developed, with a correspondingly large literature. We provide a critical review of this literature and the state-of-the-art. Most of the wide range of prediction models use size and complexity metrics to predict defects. Others are based on testing data, the "quality" of the development process, or take a multivariate approach. The authors of the models have often made heroic contributions to a subject otherwise bereft of empirical studies. However, there are a number of serious theoretical and practical problems in many studies. The models are weak because of their inability to cope with the, as yet, unknown relationship between defects and failures. There are fundamental statistical and data quality problems that undermine model validity. More significantly many prediction models tend to model only part of the underlying problem and seriously misspecify it. To illustrate these points the Goldilock's Conjecture, that there is an optimum module size, is used to show the considerable problems inherent in current defect prediction approaches. Careful and considered analysis of past and new results shows that the conjecture lacks support and that some models are misleading. We recommend holistic models for software defect prediction, using Bayesian belief networks, as alternative approaches to the single-issue models used at present. We also argue for research into a theory of "software decomposition" in order to test hypotheses about defect introduction and help construct a better science of software engineering.Keywords
This publication has 37 references indexed in Scilit:
- How software process improvement helped MotorolaIEEE Software, 1997
- Reexamining the fault density component size connectionIEEE Software, 1997
- On the use of testability measures for dependability assessmentIEEE Transactions on Software Engineering, 1996
- Software testability: the new verificationIEEE Software, 1995
- A relationship between software coverage metrics and reliabilitySoftware Testing, Verification and Reliability, 1994
- Software engineering metrics. Vol. 1: Measures and validations. Martin Shepperd, Published by McGraw‐Hill Book Company Europe, Maidenhead, U.K., 1993. ISBN 0‐07‐707410‐6, 302 pages. Price: £35.00, hard coverSoftware Testing, Verification and Reliability, 1994
- An investigation of the maintenance and support characteristics of commercial softwareSoftware Quality Journal, 1992
- Prediction and control of ADA software defectsJournal of Systems and Software, 1990
- Predicting software development errors using software complexity metricsIEEE Journal on Selected Areas in Communications, 1990
- A critique of cyclomatic complexity as a software metricSoftware Engineering Journal, 1988