The Limited Value of Non-Replicable Field Experiments in Contexts With Low Temporal Validity

Abstract

My first publication as a grad student was a field experiment using Twitter “bots” to socially sanction users engaged in racist harassment. The ascendant paradigm in quantitative social science emphasizes the need for research to be “internally valid,” with a preference for randomized control trials like the one I conducted. That research project was well received, both by the political science discipline and the public, but I no longer believe that one-off field experiments are as valuable a tool for studying online behavior as I once did. The problem is that the knowledge they generate decays too rapidly (alternatively, that the realms in which it can be applied are too few) because the object of study changes too rapidly. I have been developing the concept of “temporal validity” as form of “external validity” that is particularly relevant to the study of social media. I suggest two avenues for producing more temporally valid research: (1) faster, more transparent publication (adapting the CS model of conference proceedings); (2) a “hollowing out” of empirical research, replacing medium-scale experimentation like my own work with either purely descriptive work (ethnographic or large-scale) or with massive, collaborative, replicable experimentation.