From Researcher Degrees of Freedom to Questionable Research Practices
Researcher Degrees of Freedom
Throughout the course of model development and analysis researchers make numerous decisions. These alternative decisions are termed ‘researcher degrees of freedom’ (Wicherts et al. 2016; Simmons, Nelson, and Simonsohn 2011). These decisions are usually subjective and arbitrary yet methodologically and substantively acceptable (Wicherts et al. 2016), however may collectively significantly influence the final ‘analysis strategy’ (sensu Hoffmann et al. 2021) or ‘specification’ (sensu Simonsohn, Simmons, and Nelson 2020), and consequently the research findings and conclusions drawn from that analysis (Desbureaux 2021).
Alternative decisions about model assumptions and implementation can lead researchers down very different analytical paths that result in ‘parallel’ or ‘adjacent’ worlds with different interpretations, and potentially conflicting conclusions originating from a single underlying dataset (Young and Stewart 2021; Liu, Althoff, and Heer 2020). The set of all (plausible or reasonable) possible decisions that a researcher could make over the entire analysis from beginning to end has been termed collectively as the ‘garden of forking paths’ (Gelman and Loken 2013) or the ‘multiverse’, which consists of multiple universes of plausible analysis strategies (Steegen et al. 2016).
Researcher degrees of freedom are not inherently problematic, but rather are a product of the uncertainty that besets model development and analysis, whether due to incomplete theoretical understanding of underlying ecological processes, or because modelling is an act of simplification and abstraction; there is always some degree of choice about which ecological processes and phenomena should be included in the model, how they should be represented (Babel, Vinck, and Karssenberg 2019; Getz et al. 2017). Further uncertainty and multiplicity of analysis strategies stems from uncertainty in data preprocessing, model, parameter, and method uncertainty (Hoffmann et al. 2021).
Questionable Research Practices
When researchers are faced with ambiguous decisions where there is no single decision or strategy acceptable according to scientific standards and consensus, we are likely to select the decision that results in the most ‘favourable’ finding with “convincing self-justification” (Simmons, Nelson, and Simonsohn 2011; Hoffmann et al. 2021). Researchers routinely execute and explore alternative analysis pathways in service of identifying the most favourable (i.e. publishable) result, selectively reporting only one or several preferred results (Liu, Althoff, and Heer 2020; Simmons, Nelson, and Simonsohn 2011).
Importantly, as Gelman and Loken note (Gelman and Loken 2013) this behaviour is not a consciously nefarious ‘fishing expedition’, rather, because researchers are motivated actors in a ‘publish or perish’ culture who are susceptible to cognitive biases and are willing to change their modelling assumptions when the results conflict with desired outcomes – sincerely and self-convincingly believing that the more favourable model is superior (Young and Stewart 2021).
Thus researcher degrees of freedom precipitate questionable research practices when they remain both undisclosed and exploited opportunistically (Bakker et al. 2018). Undisclosed and opportunistic use of researcher degrees of freedom artificially increases the probability of false positive results, inflates effect size estimates and ultimately reduces the replicability of published findings while encouraging overconfidence in the precision of results and exascerbating the risk of accepting and propagating false facts (Wicherts et al. 2016; Hoffmann et al. 2021; Roettger 2019; Nissen et al. 2016). Emerging evidence suggests that rates of QRPs in ecology and evolutionary biology are similar to rates observed in other disciplines beset by reproducibility crises (Fraser et al. 2018).
Preregistration mitigates against researcher degrees of freedom
Preregistration is an open-science practice adopted in scientific disciplines that have begun to confront the ‘reproducibility crisis’, and aims to improve research transparency and mitigate questionable research practices within a study by distinguishing between what is a genuine a priori planned analysis, and what is not (Parker, Fraser, and Nakagawa 2019). Preregistration requires the methods and analysis plan of a future study are registered in a secure and publicly accessible platform prior to any data collection, and after submission it can no longer be altered.
Despite enthusiastic uptake of preregistration in some scientific disciplines, there has been little uptake of pre-registration in ecology and conservation, and many modellers and researchers engaged in non-hypothesis testing research both within and outside of ecology have eschewed preregistration on the grounds that preregistration and existing templates are irrelevant because they are focused on null-hypothesis significance testing (e.g. McDermott et al. 2021; Prosperi et al. 2019).
Instead of throwing out the baby with the bathwater, this guide is premised on the idea that preregistration can and should be used in model-based research in ecology and conservation, but its particular form should reflect the norms and practice of the research context in which it is applied, in order to adequately restrict researcher degrees of freedom that may uniquely emerge within a domain-specific methodology. The work presented in this guide aims to translate preregistration into model-based research within ecology and conservation by proposing an expanded view of preregistration we term Adaptive Preregistration. We also present a template for preregistering model-based research in ecology and conservation.