The Truth, The Whole Truth, and Nothing But the Truth
- 13 October 2016
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Programming Languages and Systems
- Vol. 38 (4), 1-20
- https://doi.org/10.1145/2983574
Abstract
An unsound claim can misdirect a field, encouraging the pursuit of unworthy ideas and the abandonment of promising ideas. An inadequate description of a claim can make it difficult to reason about the claim, for example, to determine whether the claim is sound. Many practitioners will acknowledge the threat of unsound claims or inadequate descriptions of claims to their field. We believe that this situation is exacerbated, and even encouraged, by the lack of a systematic approach to exploring, exposing, and addressing the source of unsound claims and poor exposition. This article proposes a framework that identifies three sins of reasoning that lead to unsound claims and two sins of exposition that lead to poorly described claims and evaluations. Sins of exposition obfuscate the objective of determining whether or not a claim is sound, while sins of reasoning lead directly to unsound claims. Our framework provides practitioners with a principled way of critiquing the integrity of their own work and the work of others. We hope that this will help individuals conduct better science and encourage a cultural shift in our research community to identify and promulgate sound claims.Keywords
This publication has 23 references indexed in Scilit:
- Architectural Simulators Considered HarmfulIEEE Micro, 2015
- Should software conferences respect software?Published by Association for Computing Machinery (ACM) ,2013
- Artifact evaluation for software conferencesACM SIGPLAN Notices, 2013
- The Speedup‐Test: a statistical methodology for programme speedup analysis and computationConcurrency and Computation: Practice and Experience, 2012
- Repeatability and workability evaluation of SIGMOD 2011ACM SIGMOD Record, 2011
- Paper bricksACM SIGMOD Record, 2011
- Repeatability & workability evaluation of SIGMOD 2009ACM SIGMOD Record, 2010
- Wake up and smell the coffeeCommunications of the ACM, 2008
- The five orders of ignoranceCommunications of the ACM, 2000
- The computer scientist as toolsmith IICommunications of the ACM, 1996