AAOS Board grapples with how to meet an expressed member need

Should the AAOS perform technology assessment (TA) for its members, and if so, on what, in what format, and to what end? Those questions were at the heart of a recent Board workshop, held April 18–21, 2007. According to several surveys, members want the AAOS to offer some form of technology assessment. But first, according to AAOS President James H. Beaty, MD, both the Board and the membership need to reach an understanding of the issues, impact, and role of TA. He outlined the following goals for the workshop:

• Educating Board members on TA
• Understanding the methodology behind TA, including the concept of systematic reviews
• Understanding the role of TA from a payor perspective
• Reviewing the legal issues and conflicts of interest related to TA
• Evaluating the impact of TA on industry relationships

All things to all people?
Addressing TA is difficult, partly because there are so many ways to do it. As a result, “technology assessment” means different things to different people. As workshop attendees learned, gauging the goals of an AAOS technology assessment program is challenging. For example, providing information that helps orthopaedic surgeons decide whether they should adopt a new device or procedure, or helping them discuss direct-to-consumer advertising with patients might be reasonable goals of a TA program.

Also, the term “technology assessment” does not imply that all assessments would be limited to orthopaedic devices. Medical drugs and procedures could also be considered. Comparing specific products would be difficult, and AAOS technology assessments might not mention the name of specific manufacturers. Rather, they could address classes of technologies such as the gender-specific knee, or minimally invasive surgery for a certain indication. Although the AAOS probably would not undertake laboratory evaluations of devices, it could implement assessments based on information in the published medical literature.

History of TA at AAOS
According to Joshua J. Jacobs, MD, chair of Council on Research, Quality Assessment, and Technology, and James S. Breivis, MD, a member of the Evidence-based Practice Committee, the AAOS Committee on Health Care Delivery first addressed the TA issue in the early 1990s. The committee’s White Paper included recommendations to identify possible topics for assessment and appoint work groups to review the new technologies.

Formal consideration of technology assessment, however, wasn’t undertaken until 2003, when then-president James H. Herndon, MD, asked the Council on Research to form an ad hoc project team “to consider the prospect of implementing a program to provide the Fellowship with an assessment of the evidence surrounding new technologies in orthopaedic surgery.”

This team, headed by John Kirkpatrick, MD, acknowledged the potential value of performing TA at AAOS, but noted that “its value to the membership cannot be determined without a needs assessment of the members.” Subsequently, AAOS conducted three surveys (the 2010 Member Needs Assessment in 2004, the NOLC Technology Assessment Survey and the Member Technology Assessment Survey, both completed in 2005) that asked about the members’ interest in TA. In each case, there was strong support for TA at AAOS. In fact, most of those responding to the 2005 member TA survey indicated that they would be willing to have their dues increased $35 to $50 per year to receive these technology reports.

Other influences also support AAOS consideration of technology assessment, explained board member Kevin J. Bozic, MD, MBA. These include the increasing demand for newer, more expensive technologies, increased public scrutiny of which technologies are used, and increased concern of the impact of new technologies on healthcare costs and patient outcomes.

In December 2005, the Board of Directors established a Technology Assessment Project Team. The Team produced a White Paper that highlighted the importance of using transparent, systematic processes in TA. It also distinguished between “technology overviews,” which summarize published information on a technology, and “systematic literature reviews,” which attempt to reach conclusions about the effectiveness of a technology (see sidebar).

A payor’s perspective
Richard A. Justman, MD, national medical director at UnitedHealthcare, provided an overview of TA from a private payor’s standpoint. At UnitedHealthcare, the Technology Assessment Committee, composed of physicians in multiple medical and surgical specialties and subspecialties, is responsible for reviewing medical technologies. In preparing assessments, the Committee uses outside vendors and does not review unpublished data, professional society abstracts, or meeting proceedings.

According to Dr. Justman, payors have several reasons for reviewing technologies, including the following:

• Scientific evidence may vary from patterns of practice.
• Some technologies work, but are useful for only a small population.
• Some treatments work for some diseases, but not others.
• The utility of some effective technologies is undefined.
• Some treatments are the first to work effectively for a specific condition.
• Some treatments are no better than established treatments, but have substantially greater costs.

“We must identify the ‘hot’ topics,” said Dr. Justman, “topics that physicians and consumers want to know about now. We must identify and discuss issues of controversy and look at cost-effectiveness data.”

Sean Tunis, MD, MSc, who previously served as director of the office of clinical standards and quality and chief medical officer at the Centers for Medicare and Medicaid Services (CMS), summarized Medicare’s use of technology assessment and noted the opportunity areas where AAOS might work with CMS. Dr. Tunis noted AAOS is well-positioned to engage in this project.

“TA can not only help doctors sort through complex science for multiple products, it could also identify knowledge gaps and generate evidence needed by patients and clinicians,” said Dr. Tunis.

The role of specialty societies
AAOS is not the only professional society to consider TA; both the American College of Cardiology (ACC) and the American Academy of Neurology (AAN) have viable, ongoing TA programs. Janis Miyasaki, MD, co-chair of the AAN technology and therapeutics subcommittee, expressed her belief that “TA is crucial for physicians, patients, and those who fund health care. Evidence-based medicine enables individual practitioners to counsel their patients with confidence and policymakers to make decisions based on science, not marketing.”

At AAN, the assessment process begins by developing a clinical question, searching literature for evidence, evaluating the validity of the evidence, applying the information, and evaluating the effectiveness of the technology in question. The AAN performs these activities in-house. Its TA activities have come under some legal challenges, and some surgeons have expressed their concerns as well. The AAN legal department reviews all guidelines and assessments prior to final approval.

Richard Tarr, president and executive director of InMotion Musculoskeletal Institute, provided the industry perspective on TA. He noted that industry’s ultimate goal is to turn technology into treatments that will improve patient care.

In his view, industry would have a positive view of TA efforts by AAOS, and a strong collaboration between industry and orthopaedic surgeons—conducted within appropriate ethical standards—will help to improve patient care through technologic advances.

Charles Turkelson, PhD, is director of the AAOS department of research and scientific affairs. He can be reached at turkelson@aaos.org

Systematic TA
The significance of a systematic approach in TA is most obvious in the final result. Using a systematic approach to either a technology overview or a literature review does not result in a traditional review article. Traditional reviews have several shortcomings, including a lack of transparency, a lack of rigor in preparation, and the potential for bias. Performing TA in a systematic manner addresses these shortcomings.

Systematic reviews and systematic sum-maries share the following common steps:

• framing specific and evidence-based questions
• including and excluding studies according to predefined criteria
• conducting a comprehensive literature search
• evaluating the quality of studies (assigning a level of evidence to each study)
• presenting the relevant data in a consistent format

Systematic reviews can be narrative (qualitative) or meta-analytic (quantitative). By definition, systematic literature summaries would never employ meta-analysis.

Systematic literature summaries, although relatively uncommon, are suitable when data are sparse. They provide an overview of the “state-of-the-art,” but do not arrive at conclusions about the effectiveness of a technology. Rather, the data are allowed to “speak for themselves.” If AAOS undertakes systematic literature summaries, it may call them technology overviews or highlights rather than technology assessments because “assessments” implies evaluation.

When properly conducted, “systematic reviews” are the methodologic “gold standard” of evidence-based medicine. Systematic reviews make every effort to come to conclusions about the effectiveness of a technology. These conclusions, however, stay close to the data and do not incorporate clinical opinion or judgment. Such reviews are likely to lead to conclusions only when a substantial amount of data are available and, therefore, may not be appropriate for newer technologies.