The effect of code coverage on fault detection under different testing profiles
- 15 May 2005
- journal article
- Published by Association for Computing Machinery (ACM) in ACM SIGSOFT Software Engineering Notes
- Vol. 30 (4), 1-7
- https://doi.org/10.1145/1082983.1083288
Abstract
Software testing is a key procedure to ensure high quality and reliability of software programs. The key issue in software testing is the selection and evaluation of different test cases. Code coverage has been proposed to be an estimator for testing effectiveness, but it remains a controversial topic which lacks of support from empirical data. In this study, we hypothesize that the estimation of code coverage on testing effectiveness varies under different testing profiles. To evaluate the performance of code coverage, we employ coverage testing and mutation testing in our experiment to investigate the relationship between code coverage and fault detection capability under different testing profiles. From our experimental data, code coverage is simply a moderate indicator for the capability of fault detection on the whole test set. However, it is clearly a good estimator for the fault detection of exceptional test cases, but a poor one for test cases in normal operations. For other testing profiles, such as functional testing and random testing, the correlation between code coverage and fault coverage is higher in functional test than in random testing, although these different testing profiles are complementary in the whole test set. The effects of different coverage metrics are also addressed in our experiment.Keywords
This publication has 13 references indexed in Scilit:
- Effect of test set size and block coverage on the fault detection effectivenessPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Software reliability growth with test coverageIEEE Transactions on Reliability, 2002
- On the correlation between code coverage and software reliabilityPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Code coverage, what does it mean in terms of quality?Published by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- Effect of code coverage on software reliability measurementIEEE Transactions on Reliability, 2001
- Using simulation for assessing the real impact of test-coverage on defect-coverageIEEE Transactions on Reliability, 2000
- An empirical evaluation of weak mutationIEEE Transactions on Software Engineering, 1994
- A coverage analysis tool for the effectiveness of software testingIEEE Transactions on Reliability, 1994
- An experimental evaluation of software redundancy as a strategy for improving reliabilityIEEE Transactions on Software Engineering, 1991
- The cost of data flow testing: an empirical studyIEEE Transactions on Software Engineering, 1990