Breast Cancer Symposium - Enhancing Clinical Care Through Collaboration

Recommendations for Evaluating Clinical Validity and Clinical Utility of Molecular Diagnostic Tests

Gary H. Lyman, MD, MPH, FASCO, FRCP (Edin)

There are several molecular assays available in clinical oncology practice that are used for either prognostic classification of patients or the selection of targeted therapeutic interventions. Despite the understandable excitement around these assays and potential therapies, a universally agreed upon approach to defining ther clinical validity and utility in actual practice has been lacking. Although defining the analytic validity of a molecular diagnostic test to determine that it yields consistent results is often established, demonstration that the test accurately predicts outcome or discriminates patients with different outcomes (clinical validity), or that the test leads to improved patient outcomes (clinical utility) is often unclear.

Establishing Clinical Validity and Utility

Work from the Evaluation of Genomic Applications in Practice and Prevention Working Group has been foundational in defining many of the important methodologic issues and how they can be implemented (see Figure 1).1,2 Initially convened by the National Cancer Institute (NCI) and the European Organisation for Research and Treatment of Cancer, an international committee previously developed and recently updated a checklist of Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK), which has been adopted by multiple major medical journals.3, 4 The need for complete and transparent reporting of tumor marker research has recently been emphasized.5 ASCO guideline recommendations concerning the use of tumor markers in patients with breast cancer and other malignancies have previously been published and are currently being updated.6,7

Although randomized controlled trials remain the gold standard of comparative effectiveness research, they are often not available or feasible, thus necessitating the consideration of evidence from well-designed cohort and other study designs to establish the clinical validity and utility of genomic and molecular assays.8 To further investigate the appropriate role of such assays in clinical decision making and therapeutics, NCI funded seven research programs to conduct comparative effectiveness research into genomic and precision medicine.9 The results from these initiatives have recently been presented and are in various phases of publication.10

Evidence Guidance Document Recommendations

In the meantime, given the large number of molecular and genomic tests under development, there is a critical need to further define appropriate and optimal methods for demonstrating when and how to use molecular diagnostic tests to improve patient outcomes in oncology. An Evidence Guidance Document (EGD) has recently been developed under the auspices of the Center for Medical Technology Policy in an effort to provide specific recommendations for designing studies to evaluate the clinical utility of molecular diagnostic tests in order to clarify the evidence requirements of a range of stakeholders including patients, clinicians, researchers, payers, industry, and regulators.11 It is anticipated that by meeting or exceeding the evidence standards presented in this EGD, molecular diagnostic tests are more likely to be properly developed, validated, and used to improve patient outcomes. Complete recommendations and discussion are available online,11 and the final recommendations of the EGD are briefly summarized below.

  1. It is important to ensure that analytic validity has been properly established using standard reporting guidelines (e.g., BRISQ, STARD, and REMARK) prior to performing clinical validity studies.
  2. Studies of clinical validity must specify the target patient population intended to benefit from the action and/or the clinical decision to be guided by the test result. 
  3. Validation should be performed on different data from those used to develop the model, preferably from patients recruited from other centers, and must include individuals for whom the use is intended.
  4. Clinical validation studies should report the strength of an association between the assay and relevant patient outcomes using appropriate measures. These may include sensitivity, specificity, and predictive value. Receiver operator characteristic curves may be useful in defining optimal cut-points for continuous outcomes.
  5. Regression methods may be used to model the relationship between the test result and continuous or time-to-event outcome.
  6. For tests for aiding clinical decision, outcomes on each treatment should be compared as a function of the test result.
  7. To evaluate clinical utility of a molecular assay, all clinical decisions or actions to be based on the test must be specified in advance as well as outcomes assessing both benefits and harms. Process measures such as changes in physician behavior are typically insufficient to qualify as study endpoints.
  8. Optimally, clinical utility should be assessed through a randomized controlled trial evaluating the effectiveness of the decision compared with an appropriate control for both patients who are marker-positive and patients who are marker-negative. 
  9. Properly conducted prospective-retrospective studies of appropriately designed, powered, and conducted clinical trials with banked biospecimens may represent adequate evidence of clinical utility.
  10. Clinical utility can be established in single-arm studies if all the following are met:
    • The test is being developed with a drug that has already been approved by the U.S. Food and Drug Administration (FDA).
    • Adequate archived tissue samples are not available to conduct a prospective-retrospective trial.
    • It is feasible to use response as an endpoint.
    • There exist comparable response data in a comparative cohort.
  11. Prospective cohort studies and other selected data may be acceptable options provided that the reasons for using such data are addressed, efforts to minimize confounding are documented, and good research practices are followed.
  12. Decision-analytic modeling using separate sources of evidence of relevant measures of benefits and harms can be used when there is no direct evidence of clinical utility when clinical validity has been established and plausible evidence of clinical utility based on modeling is found. Summary measures such as clinical events, life expectancy, and quality-adjusted life years represent appropriate modeling outcome measures.

Cost Considerations

Finally, the cost of molecular diagnostics and targeted therapeutics is an ever-present reminder of the continuing and unsustainable rise in the costs of modern cancer care.12,13 Nevertheless, costs are not currently considered in the review or approval of diagnostic tests or therapies by the FDA and are rarely addressed in clinical practice guidelines. In the future, however, evaluation of the economic validity and overall value of molecular diagnostics and therapeutics must be an essential part of comparative effectiveness research in oncology and will be an unavoidable part of the discourse between patients, providers, payers, and regulators.

About the Author: Dr. Lyman is professor of medicine in the Division of Medical Oncology, Department of Internal Medicine, at Duke University School of Medicine and the Duke Cancer Institute. He serves as director of the Comparative Effectiveness and Outcomes Research Program in Oncology. Dr. Lyman is also senior fellow in the Duke Center for Clinical Health Policy Research. He is a Fellow of the American Society of Clinical Oncology and is currently a member of the ASCO Board of Directors. Dr. Lyman will speak on the “Comparative Effectiveness and Cancer Therapies” during General Session VII: Health Policy and Recent Controversies, Saturday at 1:00 PM (PDT).