Michael Kane

Yale University

# Takeaways

## How big a problem is lack of reproducibility?

### E. Shanil et al. (2014) put an upper bound on the irreproducibility of the final analysis published from a set of 37 publications at 35%

-"Reanalyses of randomized clinical trial data." Jama

## But it's worse than that...

\begin{aligned} \mathbb{P} [ & \text{irreproducible trial}] \\ & \leq 1 - \mathbb{P} [\text{reproducible study design}] * \mathbb{P}[\text{reproducible data analysis}] \\ & = 1-(1-0.35)^2 \\ & = 0.57. \end{aligned}
\begin{aligned} \mathbb{P} [ & \text{irreproducible trial}] \\ & \leq 1 - \mathbb{P} [\text{reproducible study design}] * \mathbb{P}[\text{reproducible data analysis}] \\ & = 1-(1-0.35)^2 \\ & = 0.57. \end{aligned}

## and even worse than that...

"We completed an electronic search of MEDLINE from inception to March 9, 2014, to identify all published studies that completed a reanalysis of individual patient data from previously published RCTs addressing the same hypothesis as the original RCT."

"We identified 37 eligible reanalyses in 36 published articles, 5 of which were performed by entirely independent authors."

By Michael Kane

• 464