Measuring effectiveness of sample-based product-line testing
- 7 April 2020
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM SIGPLAN Notices
- Vol. 53 (9), 119-133
- https://doi.org/10.1145/3393934.3278130
Abstract
Recent research on quality assurance (QA) of configurable software systems (e.g., software product lines) proposes different analysis strategies to cope with the inherent complexity caused by the well-known combinatorial-explosion problem. Those strategies aim at improving efficiency of QA techniques like software testing as compared to brute-force configuration-by-configuration analysis. Sampling constitutes one of the most established strategies, defining criteria for selecting a drastically reduced, yet sufficiently diverse subset of software configurations considered during QA. However, finding generally accepted measures for assessing the impact of sample-based analysis on the effectiveness of QA techniques is still an open issue. We address this problem by lifting concepts from single-software mutation testing to configurable software. Our framework incorporates a rich collection of mutation operators for product lines implemented in C to measure mutation scores of samples, including a novel family-based technique for product-line mutation detection. Our experimental results gained from applying our tool implementation to a collection of subject systems confirms the widely-accepted assumption that pairwise sampling constitutes the most reasonable efficiency/effectiveness trade-off for sample-based product-line testing.Keywords
Funding Information
- Deutsche Forschungsgemeinschaft (LO 2198/2-1,SCHA 1635/10-1)
- Conselho Nacional de Desenvolvimento Científico e Tecnológico (307190/2015-3)
- Hessisches Ministerium für Wissenschaft und Kunst (Software-Factory 4.0)
- Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (175956,117875)
- Fundação de Amparo à Pesquisa do Estado de Alagoas (14/2016 60030 000435/2017)
This publication has 39 references indexed in Scilit:
- Automated Statistical Testing for Embedded SystemsPublished by Taylor & Francis Ltd ,2017
- On strategies for testing software product lines: A systematic literature reviewInformation and Software Technology, 2014
- A Classification and Survey of Analysis Strategies for Software Product LinesACM Computing Surveys, 2014
- Overcoming the Equivalent Mutant Problem: A Systematic Literature Review and a Comparative Experiment of Second Order MutationIEEE Transactions on Software Engineering, 2013
- Configuration coverage in the analysis of large-scale system softwareACM SIGOPS Operating Systems Review, 2012
- A mutation carol: Past, present and futureInformation and Software Technology, 2011
- Software product line testing – A systematic mapping studyInformation and Software Technology, 2011
- Automated analysis of feature models 20 years later: A literature reviewInformation Systems, 2010
- Evaluating improvements to a meta-heuristic search for constrained interaction testingEmpirical Software Engineering, 2010
- Software fault interactions and implications for software testingIEEE Transactions on Software Engineering, 2004