## Defining Reproducibility in Clinical Trials (Challenges and Opportunities)

Michael Kane

Yale University

# Takeaways

### Lack of reproducibility is currently a big problem in clinical trials.

### The most significant obstacles to adoption are social.

### Reproducible research provides better trials and new opportunities to produce findings more quickly.

## How big a problem is lack of reproducibility?

### E. Shanil et al. (2014) put an upper bound on the irreproducibility of the final analysis published from a set of 37 publications at 35%

-"Reanalyses of randomized clinical trial data." *Jama*

## But it's worse than that...

\begin{aligned}
\mathbb{P} [ & \text{irreproducible trial}] \\
& \leq 1 - \mathbb{P} [\text{reproducible study design}] * \mathbb{P}[\text{reproducible data analysis}] \\
& = 1-(1-0.35)^2 \\
& = 0.57.
\end{aligned}

$\begin{aligned}
\mathbb{P} [ & \text{irreproducible trial}] \\
& \leq 1 - \mathbb{P} [\text{reproducible study design}] * \mathbb{P}[\text{reproducible data analysis}] \\
& = 1-(1-0.35)^2 \\
& = 0.57.
\end{aligned}$

### A trial consists of a study design and data analysis...

## and even worse than that...

"We completed an electronic search of MEDLINE from inception to March 9, 2014, to identify all published studies that completed a reanalysis of individual patient data from previously published RCTs addressing the same hypothesis as the original RCT."

"We identified 37 eligible reanalyses in 36 published articles, *5 of which were performed by entirely independent authors.*"

## and even worse than that...

### "2 [studies were] based on publicly available data and 2 on data that were provided on request; data availability was unclear for 1."

## and even worse than that...

### The supplemental material was in pdf.

## and even worse than that...

### The study did not include the analyses trying to reproduce the results.

## What can we draw from the paper?

### Authors find a "lower-upper" bound of 35% irreproducibility.

### Availability of data is separated from the ability to analyze it.

### Clinicians have a very different conception of reproducibility.

### Why is reproducibility difficult for statisticians in medicine?

### We are often not the PI.

### Clinicians don't understand the extra effort needed for reproducibility.

### The extra effort is not budgeted for.

## Why is reproducibility difficult for clinicians in medicine?

### They are often not aware of what we mean by reproducible.

### They barely understand what we are doing to begin with.

### A fully reproducible analysis may increase liability (Baggerly Coombs 2009).

## What do we get from reproducible CT's

### - More sophisticated inclusion/exclusion criteria based on similar trials.

### - Better prognostic factors (cancer naivety as an example).

### - Better contextualization of trial results - trial comparison goes from a scarcity to abundancy problem.

### - Better understanding of histological heterogeneity.

# Thanks

#### deck

By Michael Kane