Threats to validity in software engineering research: A critical reflection

Context: In the contemporary body of software engineering literature, some recurrent shortcomings characterize how threats to validity (TTV) are considered in studies. Objective: With this position paper, we aim to open a discourse on the current use of TTV sections. The goal of our position is to jointly reflect and systematically improve how we, as a research community, consider TTV in our studies. Methods: Based on our personal experience as researchers, authors, reviewers, and editors, we critically reflect on the treatment of TTV in current empirical software engineering literature. Results: We discuss the key shortcomings of TTV consideration, including the failure to acknowledge different types of validity categorizations and the tendency to treat threats just as an afterthought. For each identified problem, we propose a vision for an improved state, intending to catalyze thoughtful engagement and improvements the way our community addresses TTV. Conclusion: We posit there is an urgent need to reconsider how we approach, document, and evaluate TTV in software engineering research.