Challenges to Good RDM

Challenges to Good RDM Practices

The challenges to good RDM practices are multifaceted, but deeply engrained in the reward systems of academia, the funding models of funding agencies, and the publishing model of journal publishers.

Begining at the undergraduate level students are often awarded on ‘finding the right answer’ over identifying error and bias. In an undergraduate lab where sample sizes are small and conditions are error prone, epmhasizing the ‘right answer’ sets the building blocks for a culture of practice that is not first and foremost geared to reducing questionable research practices, encouraging reproducible research, and championing good research data management practices.

Questionable research practices (QRPs) derive from researcher degrees of freedom and not clearly documenting the seperation between exploratory and confirmatory research.

An example of the former includes p-hacking, where data are worked until statistical significance is achieved. This may be through how the data are manipulated during cleaning, how thresholds are set, deciding to collect more data after the fact etc.

An example of the latter includes HARKing – hypothesizing after the results are known. When the data collected do not support the hypothesis being tested, alternative hypotheses are tested on the data and potentially reported on as the original point of inquiry.

At the graduate level and above in one’s academic career, the reward system continues to emphasize outputs through publication counts, h-indexes, and impact factors – that is, how much are you publishing, and how much is it being cited in highly cited journals. As evidenced by the few journals that employ data editors, the data – and their veracity as demonstrated through good RDM practices – themselves are not the primary deciding factor on publication, but rather the novelty and statistical significance of what is reported in the article. The focus on the latter is generally to the detriment of process making measuring bias and reproducibility extremely challenging. See for example Do Pressures to Publish Increase Scientists’ Bias? An Empirical Support from US States Data available at https://doi.org/10.1371/journal.pone.0010271.

This is coupled with novel and statistically significant findings being favoured by publishers in general. Reporting on replication studies and publishing non-significant findings is both challenging and not as highly rewarded either by publication or ensuing citation. But the value of these activities to scientific research is significant. This practice is not equal across all fields with some disciplines being more prone to the issues than others. See for example “Positive” Results Increase Down the Hierarchy of the Sciences available at https://doi.org/10.1371/journal.pone.0010068.

At the funging stage, replication studies are generally not favoured when compared to novel pursuits with the potential of significant near term impact. This is perhaps most evident in medical fields, where many reseach questions are never addressed or significant replication never undertaken because research funds are targetted in the direction of novel pursuits with impact that is more easily reported on. See for example How a now-retracted study got published in the first place, leading to a $3.8 million NIH grant reported on by Retraction Watch or the recent issues highlighted in Alzheimer research, as reported in Science, Blots on a field.

These challenges all manifest in poor documentation (documentation of decision making processes takes time, reducing number of scholarly outputs), and increased study bias (less time spent checking cognitive biases, increase in p-hacking and HARKing related practices to find significance, less attention to influence of systemic biases).

---

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License

Built and rendered courtesy of RMarkdown from RStudio