Studying the Characteristics of a "Good" GUI Test Suite
- 1 November 2006
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in 2006 17th International Symposium on Software Reliability Engineering
- p. 159-168
- https://doi.org/10.1109/issre.2006.45
Abstract
The widespread deployment of graphical-user interfaces (GUIs) has increased the overall complexity of testing. A GUI test designer needs to perform the daunting task of adequately testing the GUI, which typically has very large input interaction spaces, while considering tradeoffs between GUI test suite characteristics such as the number of test cases (each modeled as a sequence of events), their lengths, and the event composition of each test case. There are no published empirical studies on GUI testing that a GUI test designer may reference to make decisions about these characteristics. Consequently, in practice, very few GUI testers know how to design their test suites. This paper takes the first step towards assisting in GUI test design by presenting an empirical study that evaluates the effect of these characteristics on testing cost and fault detection effectiveness. The results show that two factors significantly effect the fault-detection effectiveness of a test suite: (1) the diversity of states in which an event executes and (2) the event coverage of the suite. Test designers need to improve the diversity of states in which each event executes by developing a large number of short test cases to detect the majority of "shallow" faults, which are artifacts of modern GUI design. Additional resources should be used to develop a small number of long test cases to detect a small number of "deep" faultsKeywords
This publication has 12 references indexed in Scilit:
- Studying the fault-detection effectiveness of GUI test cases for rapidly evolving softwareIEEE Transactions on Software Engineering, 2005
- Using simulation to empirically investigate test coverage criteria based on statechartPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2004
- On test suite composition and cost-effective regression testingACM Transactions on Software Engineering and Methodology, 2004
- Firewall regression testing of GUI sequences and their interactionsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2004
- Bayesian analysis of software cost and quality modelsPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2002
- The distribution of faults in a large industrial software systemACM SIGSOFT Software Engineering Notes, 2002
- Coverage criteria for GUI testingACM SIGSOFT Software Engineering Notes, 2001
- Hierarchical GUI test case generation using automated planningIEEE Transactions on Software Engineering, 2001
- A critique of software defect prediction modelsIEEE Transactions on Software Engineering, 1999
- Generating oracles from your favorite temporal logic specificationsPublished by Association for Computing Machinery (ACM) ,1996