The Scientific Process and the Need for Better Data
Julie Frearson

The Scientific Process and the Need for Better Data

20160718-scipolimage-294x152-eurekahomeHow a Reproducibility Network—and CROs—might help us produce better drugs for tough diseases like AD.

Alzheimer’s disease (AD) is inarguably the largest unmet need in the US today. While there are diseases, such as heart disease, with higher prevalence and greater mortality, AD is the largest disease without any meaningful treatment or cure. We don’t know how to prevent AD, and as our population ages the prevalence of this progressive, neurodegenerative condition will likely increase as well.

Multiple political leaders have advocated for applying more resources to researching treatments for AD. During her lifetime, Former First Lady Nancy Reagan advocated strenuously in support of funding for novel therapeutic approaches—including embryonic stem cell research—while Democratic nominee Hillary Clinton wants to see a doubling of the US government spending on AD. Vice Presidential nominee Tim Kaine advocated for further research to find a cure for the disease during his acceptance speech at the 2016 Democratic National Convention in Philadelphia last week. Most people know someone afflicted with AD and the prospect of being afflicted or caring for someone with AD concerns almost everyone. Nancy Reagan’s “Long Goodbye” to her husband perhaps best captured the fear we have that we, or someone we love, will be afflicted.

Our reproducibility problem

Unfortunately, we in the world of CNS Drug Discovery have had limited success in identifying novel ways to treat the symptoms. And despite years of effort across hundreds of labs, there is no cure available. Part of this is due to the challenges applying preclinical discoveries into meaningful outcomes for patients.

The development of new drugs typically begins with academic researchers discovering a biological pathway or molecules that might be a good disease target, and then publishing their work in scientific journals. Replicating those results can be challenging. In fact, findings reported last year by Boston University suggest that as much as US$28 billion is spent yearly on preclinical research that can’t be reproduced.

Published results that are difficult to reproduce in life-science-based academic research have been highlighted as a major concern for research policy makers, scientific press and the pharmaceutical healthcare industry. Key academic discoveries are particularly vulnerable to invalidation as they transition into more industrialized, high stringency evaluation for drug discovery. The pharmaceutical and biotechnology sectors rely heavily on discoveries made by curiosity-led academic endeavor to develop disease biology understanding and provide the confidence for the industry to pursue first-in-class medicines and diagnostics that address genuine, unmet need.

A network solution

As such, a reputable, stakeholder-approved and independent mechanism developed by the US government for the pressure testing of key discoveries is sorely needed to focus industry efforts on validated phenomena and thereby reduce attrition in the earliest phase of drug discovery-target validation. We feel the federal government is positioned to influence the quality of academic research and help provide mechanisms whereby key impact discoveries can be rapidly and independently validated and enriched through development of a reproducibility network.

Such a network could, for instance:

  1. Develop new granting policies which actively encourage principal investigators to submit their discoveries for reproduction in accredited third party organisations- and reward them for doing so.
  2. Develop a new scheme to identify and qualify a number of organizations to act as independent agents for reproduction of scientific studies to industry standards. Such organizations would be required to be independent (i.e., not participating in internal IP-generating activities), have expertise in disease biology aligned with key areas of unmet need (likely fulfilled by a combination of large organisations with breadth plus smaller niche organizations with domain-specific expertise), deep drug discovery and development expertise, and demonstrable quality management processes in place.
  3. Develop a new scheme to identify organizations capable of monitoring life science publications for their potential impact on healthcare. Such organisations would require the IT, statistics expertise and infrastructure to establish and curate the necessary databases and develop systems to prioritise publications for reproduction. These systems should also be a capable of recording the output of said reproduction studies and making the results available in the public domain.
  4. Develop publication polices in conjunction with scientific journal publications to enable the rapid, open access publication of reproducibility studies whether the outcome is positive or negative.
  5. Launch a new granting scheme which enables PIs or third party organisations to secure funding to support reproducibility studies. Such studies will be conducted in one of a group of verified reproducibility organisations. This scheme could be leveraged in key disease areas by co-operative funding from patient and disease foundations. 

A fully enabled reproducibility network would need to have clear funding channels to support projects, key reproducibility advocates to manage and promote the objectives of the enterprise, plus a number of reproducibility centers with the operational and management expertise to conduct reproducibility studies. And a functional network would most likely need to include organizations with fundamental cell biology, pharmacology, chemistry and antibody generation capabilities plus centers of excellence with domain specific expertise in key platforms (genomics, proteomics) and disease areas.

The role of CROs

Independent laboratories, such as those at CROs, have a valuable role to play here by offering industry-standard quality management infrastructure and methods, non-biased approaches to studies, staff trained in conducting high-quality projects efficiently, broad therapeutic expertise, fundamental platforms in cell biology, pharmacology and chemistry, and experience in working with academic laboratories.

The problem of irreproducibility in life science research significantly erodes its potential value to future healthcare. While we must protect the fundamental practices of academic research, being curiosity-driven with the freedom to follow new ideas, it is clear that additional systems need to be put in place to smooth out the ‘lack of experimental rigor’ and ‘speed to publish’ issues in evidence.

Our belief is that the federal government has a role to play in working with the NIH and equivalent bodies to create a life science reproducibility platform which will provide a route to rapid assimilation, prioritization, validation and publication of key discoveries in a quality managed, statistically powered and independent manner. Such a platform has the potential to become one of the key channels the pharmaceutical industry uses to derive data to support novel target validation concepts. Furthermore, it could also influence the conduct of the next generation of academic investigators. Future investigators will be judged not only on publication rate and quality but also on how often their discoveries feature in federally funded reproducibility programs.

The AD literature is littered with hundreds of novel potential treatments based on findings in a single lab. Translating those findings into treatments that have clinical utility has been unsuccessful and even translating those findings into something that works across labs is unsuccessful more often than not. A reproducibility network might help improve our success in the clinic, and CROs are uniquely positioned to help make such a network successful.

How to Cite: 

Frearson, Julie. Hodgson, Robert. The Scientific Process and the Need for Better Data. Eureka blog. Aug. 4, 2016. Available: