We will be performing site maintenance on our learning platform at learn.aaos.org on Sunday, January 29th at 12 AM EST. The site will be down for up to 5 hours. We apologize for the inconvenience.


Published 4/1/2011
Annie C. Hayashi

Making the case for Level I evidence

Case series no longer sufficient to push field forward

Although the quality of orthopaedic clinical research continues to improve, most studies still lack the appropriate level of evidence needed to address the growing musculoskeletal burden in the United States.

More than 35 million Americans report some type of musculoskeletal disorder, reported Theodore Miclau, MD at the Orthopaedic Research Society (ORS) sponsored Clinical Research Forum. Orthopaedic procedures, such as hip and knee replacements, enable many to continue their level of activity and quality of life.

“The overall challenge is how to deal with the increasing cost of musculoskeletal care, which is becoming a larger percentage of the gross domestic product,” said Dr. Miclau.

“The pressure on orthopaedists is going to continue to increase as we perform more high-cost procedures—especially in the areas of hip, knee, and spine,” said Kristy L. Weber, MD. Orthopaedic surgeons will need to justify these costs with evidence-based medicine (EBM), according to Dr. Weber.

Why EBM?
For three decades, major advances in orthopaedics were made based on evidence from case series, according to James G. Wright, MD. These included total hip replacement pioneered by Sir John Charnley in the 1960s, internal fixation and instrumentation for the spine in the latter 1950s, and treatment of club foot pioneered by Ignacio V. Ponseti, MD, in the late 1940s.

“These were the parachutes of orthopaedics. Randomized controlled trials (RCT) weren’t needed to determine if parachutes were effective,” Dr. Wright said. “But what have the similar major jumps in patient outcomes been since that time?

“We have developed new treatments,” he continued. “But we have not seen the same paradigm shift in patient outcomes as we did in those earlier studies. We need alternatives to case series to demonstrate effectiveness.” That alternative paradigm is EBM.

The power of evidence
“Evidence can change practice. This is probably the most powerful rationale for going in this direction,” Dr. Weber said. “If we really apply the best evidence to a particular situation, then we can use treatments that have been shown to be effective in the literature but aren’t commonly used.”

An example of a well-designed RCT demonstrates the impact of EBM. In 2010, The Journal of Bone and Joint Surgery—American (JBJS–A) published the results of an RCT comparing surgical versus nonsurgical treatment of acute Achilles tendon rupture. The multicenter Canadian study demonstrated that accelerated rehabilitation treatment with early weight bearing and range of motion was as effective as surgical treatment. Outcomes for the patients who received the conservative treatment were virtually the same as those for patients who underwent surgical repair. In fact, more complications occurred in the surgical group.

Dr. Wright offered an example from pediatric oncology that showed how evidence-based studies could dramatically change and improve outcomes. Thirty years ago, approximately 10 percent of children diagnosed with malignancies survived. Because pediatric oncologists relied on their personal experiences, they did not have any objective, systematic approach to evaluate the efficacy of one treatment versus another.

Frustrated with the low rates of survivorship, the oncologists formed the “Children’s Oncology Group” and conducted RCTs. The survival of children with malignancies increased from 10 percent to 70 percent and was attributed almost completely to RCTs.

The current state of orthopaedic clinical research
The quality of orthopaedic clinical research has improved over the last 30 years. For example, in JBJS-A, the average level of evidence for published studies rose from 3.72 in 1975 to 2.90 in 2005.

Despite this improvement, case studies—Level IV evidence—still accounted for nearly half (48 percent) of published studies in 2005; the percentage of Level IV evidence was only slightly less than Levels I, II, and III combined (Fig. 1).

According to Charles M. Turkelson, PhD, who heads the AAOS research department, the quality of orthopaedic literature is compromised by Level IV case series subject to regression to the mean. Regression to the mean can also affect controlled studies.

Dr. Turkelson explained the impact of regression to the mean in this way: “People don’t feel the same every day. Sometimes they feel better and other times they’re worse. They go to the doctor when they feel worse. As the natural part of this cycle, they will feel better even if they don’t receive any treatment. This is known as regression to the mean. It’s not surprising that most case series find that treatment is effective. This is simply part of the cycle of their symptoms.”

He cited a study published in The Lancet on outcomes after vertebroplasty. The authors considered this an open-label (nonblinded), randomized trial of vertebroplasty versus conservative treatment.

The two groups, however, differed in 60 percent (3 of 5) of their outcomes at baseline. “This is not something that could happen in an RCT. The researchers may have attempted to randomize but the randomization did not succeed,” he said.

The results showed greater improvement from baseline in the vertebroplasty group versus the control group. “It is exactly what you would expect from regression to the mean. At no time did the authors test to see whether there was a difference between the two groups. It only tested for differences from baseline,” Dr. Turkelson explained.

Even by the author’s own definition, no clinically important differences were found between the groups. Although this study is frequently cited as demonstrating the effectiveness of vertebroplasty, that assessment is inaccurate, according to Dr. Turkelson.

“Considerations about study quality are not mere academic exercises. They have real world consequences. Good quality research means good quality care,” he concluded.

Annie C. Hayashi is the ORS development and communications manager. She can be reached at hayashi@ors.org

Editor’s Note: This is the first of a three-part series on the ORS-sponsored Clinical Research Forum.