Published 6/1/2015
David S. Jevsevar, MD, MBA

AAOS to Add Systematic Reviews to Evidence-based Products

Beginning this year, the AAOS Evidence-based Quality and Value Committee (EBQV) will convene workgroups to produce and publish systematic reviews as part of its evidence-based program for orthopaedics.

Since 2007, the AAOS has created and published clinical practice guidelines (CPGs) as part of this program. To date, the AAOS has published 17 CPGs, highlighting the efficacy of treatments and the current state of associated research literature. As defined by the Institute of Medicine (IOM), CPGs are “systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances.”

Typically, CPGs are initiated for care of a disease or injury with high cost, high volume, and/or high variation in care. At their core, the AAOS CPGs are a series of questions addressing areas of interest regarding diagnosis and treatment of patients with orthopaedic problems, which are then answered through systematic reviews. AAOS CPGs have had as few as three systematic reviews and as many as 30 or more.

What is a systematic review?
A systematic review is a thorough, comprehensive, and explicit way of assessing the medical literature on a particular topic or topics. Systematic reviews involve several steps including the following:

  • asking an answerable question(s) (often the most difficult step)
  • identifying databases to search and developing an explicit search strategy
  • selecting titles, abstracts, and manuscripts based on explicit inclusion and exclusion criteria to identify relevant work
  • assessing the quality of the studies
  • summarizing the evidence
  • interpreting the findings of the review
  • conducting a peer review of the final product

In contrast, narrative reviews are not considered evidence (research) even though they may be evidence-based. Unless otherwise specified, narrative reviews typically lack systematic search protocols or explicit criteria for selecting and appraising evidence.

Instead, experts gather evidence and synthesize findings when producing a narrative review. When evidence is lacking, the authors of narrative reviews make recommendations based on their opinions and experience. This is again in contrast to systematic reviews, in which the authors recommend additional research if evidence is lacking.

One misconception about systematic reviews is that they always include a meta-analysis. A meta-analysis is a statistical approach that combines the data from different studies derived from a systematic review. The meta-analysis usually helps to make greater inferences from the available data.

Every meta-analysis should be based on an underlying systematic review, but not every systematic review includes a meta-analysis. Systematic reviews that include meta-analysis can be referred to as quantitative systematic reviews, and those based on literature analysis in which direct data comparisons are not possible are called qualitative systematic reviews.

As with primary research evidence, systematic reviews are subject to various flaws that affect their strength and findings. The author’s conflicts of interest should be explicitly defined and managed to avoid the potential for bias. The research questions developed by the team need to be clinically important and applicable to patient care. Search strategies need to be clearly described and followed, with strict definitions for inclusion and exclusion criteria.

The inclusion of small studies generally increases the number of potential studies available, but it can lead to “small study effects” that give the small studies a disproportionate impact on data analysis. All evidence needs to be graded with a defined quality grading methodology. Relying solely on lower levels of literature evidence decreases the strength on the final interpretation of the data. Combining dissimilar studies in a meta-analysis can lead to such heterogeneity (a measure of difference between studies) that the results become more difficult to interpret.

AAOS and systematic reviews
The AAOS will use the IOM’s “Standards of Systematic Reviews” as a guideline for its process. There are many reasons for adding systematic reviews to the AAOS evidence-based portfolio. Although CPGs, by their nature, are large and slow moving due to the number of recommendations, systematic reviews should be developed more expeditiously to help members address current “hot topics” of care.

In addition, systematic reviews will help to fill in the gaps where an issue is not big enough for a CPG, but is still important for quality, safety, and/or cost/reimbursement issues. Systematic reviews will also be developed as needed to support the AAOS appropriate use criteria (AUC), performance measures, and patient safety programs. Nominations for topics can come from members, AAOS committees, or affiliated Specialty Societies.

The AAOS maintains its commitment to a robust quality program, which will continue to include its evidence-based CPGs and AUCs, as well as safety initiatives and performance measures. The systematic review program will serve to fill the need for relatively quick, high-level evidence analyses on important orthopaedic topics.

David S. Jevsevar, MD, MBA, chairs the AAOS Evidence-based Quality and Value Committee.


  1. Khan KS, Kunz R, Kleijnen J, Antes G: Five steps to conducting a systematic review J R Soc Med 2003;96(3):118–121.