A lot of buzzwords, abbreviations, and criticisms exist in medicine today. Quality, value, and even standards of care are moving targets based on evolving healthcare trends. Increasingly, we hear about appropriate use criteria (AUC) and clinical practice guidelines (CPGs) as tools that summarize evidence and fill gaps when evidence is unavailable.

The AAOS Committee on Evidence-based Quality and Value (EBQV) sets the methods for CPGs and AUC, based on the best standard methods, yet the average AAOS member may be unaware of what AUC are, how they are produced, and how they can be used in day-to-day practice.

Kevin G. Shea, MD, chair of the AAOS Committee on EBQV, and Jayson N. Murray, MA, director of the Department of Clinical Quality and Value at AAOS, weigh in on the AUC process, how AUC are made, and the evidence behind them.

Dr. Reznik: How did AUC come about?

Dr. Shea: The goal was to collect evidence and assemble the best information as part of the Academy’s initiative to improve orthopaedic quality—to hold ourselves to a high standard before others create standards for us. The primary goals of AUC are twofold. The first is to increase the use of those procedures supported by the best available evidence, especially those that are currently underused, and secondly, to decrease the procedures that have had patterns of overuse, without evidential support. These goals address two areas that may represent an unnecessary increase in medical costs and provide an opportunity to improve medical care for our patients. In this way, some additional caution is given for more invasive procedures that have much less or lower levels of evidential backing. At times, our patients demand some of these treatments, and good AUC can help us during the shared decision-making process.

Dr. Reznik: How are AUC developed?

Dr. Shea: After a literature review for a given orthopaedic topic, the evidence is evaluated. The level of evidence also is considered. A writing panel organizes patient vignettes, based on common patient factors for the orthopaedic topic under study along with a list of applicable interventions. Then a voting panel independently rates the appropriateness of each procedure.

Dr. Reznik: Once the votes are in, are the AUC discussed in a forum?

Mr. Murray: Yes. AUC were developed as a bridge to connect the gaps in the evidence using a multidisciplinary panel of experts and a well-defined Delphi process (Fig. 1). Initial votes are cast as to treatment options (Fig. 2). After voting, an in-person meeting—a key component in the AUC creation process—is held at AAOS headquarters in Rosemont, Ill. The majority of the in-person meeting is spent discussing the ins and outs of each of the representative panel member’s opinions on the AUC. All fellows on the committee provide perspectives on why they voted the way they did for each of the patient vignettes.

Dr. Shea: More specifically, the following occurs during the in-person meeting:

• The results of the round of ratings, especially in areas of disagreement, are discussed.
• The patient vignettes are revised as necessary.
• As warranted, the procedures for any patient vignettes are re-rated.
• This method is well tested, and a modified two-round Delphi process is derived from the Rand/UCLA Appropriateness Method. It is meant to accurately find the “true” value of the expert opinion created by the panel.
• Disagreements are vetted, and afterward, a re-vote is taken based on the discussions.

Dr. Reznik: How often is there disagreement?

Dr. Shea: On average, the group initially disagrees on 20 percent to 30 percent of the total voting items. After the in-person discussion, the disagreement drops to around 1 percent to 3 percent. A nearly unanimous agreement is a result of the Delphi process.

The discussion includes the exact evidence behind a procedure and various nuances that may apply to any given patient vignette. This is especially true when the voting results in disagreement. Lastly, for completeness and full transparency, any disagreements are listed within the AUC app and the hardcopy AUC. There is no formal outside peer-review period for AUC, mainly because of the inherent difficulty in reviewing a large number of patient vignettes. This may be something to consider in the future.

Dr. Reznik: Are there any other AAOS committees involved?

Mr. Murray: Once an AUC has been approved by the Committee on EBQV, it goes to the Council on Research and Quality (CORQ) for approval and finally to the AAOS Board of Directors. At the EBQV and CORQ levels, we request approval through email votes. If an AUC passes, it moves to the subsequent stage of approval. If not, we hold a conference call to have an open discussion with EBQV or CORQ members regarding their concerns. For a conference call vote, a consensus to approve is required, and all “no” votes are recorded. The additional review periods within the approval process are the real reasons why an additional peer-review period is absent from this product.

Dr. Reznik: Is it difficult to obtain good evidence?

Dr. Shea: Although the weakness of evidence has always been a concern, this is becoming increasingly less of an issue. It’s true that when we started guidelines in 2006–2007, a dearth of evidence existed for the topics we chose. It has been hard to shake some members’ historical perspective of orthopaedics as an evidence-starved profession, but if you look at most of our guidelines published after 2013, you’ll find that many of our recommendations are based on high-quality literature, and this literature is expanding exponentially year by year. This is particularly evident with the AAOS CPGs on carpal tunnel syndrome, anterior cruciate ligament injury, osteoarthritis (OA) of the hip, and surgical management of OA of the knee.

Dr. Reznik:Have there been legal issues with prior AUC?

Mr. Murray: At the time of writing, there have been no legal issues involving the use of AUC. Quality products are educational tools to guide qualified physicians. The ultimate judgment for any treatment must be made by the clinical circumstances and resources particular to the locality or institution. Still, this is an area we can always improve upon; it is a concern of our fellows, and we are aware of it.

Dr. Reznik: Has any more thought been given to the use of the term “appropriate use” in AUC?

Dr. Shea: A concern that was recently brought up at a CORQ meeting is that the opposite of “appropriate use” is “not appropriate use.” That could have a negative connotation. It may be an unintended malpractice risk even though we have proper disclaimers. The Committee on EBQV consciously changed the rating language of “not appropriate” to “rarely appropriate” to acknowledge the need for leeway for those procedures that are rated on the low end for various patient vignettes. We understand that a lack of evidence for rare procedures may be a function of the rarity, not the effectiveness. Still, our goal is to move toward increasingly better care, and the language we choose may affect that goal. The overall language we choose also can make a difference in the meaning and purpose of this work. That is something to consider as we view this as a dynamic process that we will continually improve upon.

Dr. Reznik: Thank you, Dr. Shea and Mr. Murray. There is a real need to know more about everything we as orthopaedic surgeons do. In some situations, the best we can do is an expert opinion. At the same time, big data primarily based on billing codes are being used to fill a gap in the real clinical evidence for care. Our members know that if our Academy is not in the lead, we will fall behind as others rush to dictate how we practice the art of orthopaedic surgery. It is very helpful to understand that the AUC process is well conceived and that the work product is informed by the literature and clinical vignettes, as well as rigorous debate. Hopefully, our members will find the tablet or phone app version of AUC both useful and good clinical back-up to make better choices that help us nudge our patients in the best direction possible.

Future challenges for the AUC criteria process

The AAOS Committee on Evidence-based Quality and Value (EBQV) recognizes that creation of appropriate use criteria (AUC) is an evolving process and that subsequent methods may work in different ways. Below are some future challenges for the AUC process.

1. Insurance coverage and AUC: Literature reviews and vignettes are created to test ideas. The goal is for the Academy to own the process and hence create a significant benefit to members. Yet they are in the beginning stages, and one size does not fit all. Insurers should be aware that these are not tools for blanket denial authority. Rather, they are intended to nudge behavior in a more evidence-based direction, as patients may at times present with conflicting issues that prevent the first choice of treatment and force lesser alternatives. For example, nonsteroidal anti-inflammatory drugs are the first line of treatment for many injuries and inflammatory processes, except in patients with Barrett’s esophagus or peptic ulcer disease. We also know deep vein thrombosis (DVT) prophylaxis is contraindicated in patients with low platelet counts. It is surprising how often even the most straightforward contraindications are left out of some treatment guidelines. In medicine, as people age and considerations for nonorthopaedic diseases expand, examples like this are becoming more and more common.
2. AUC and clinical practice guidelines (CPGs): AUC offer more individual patient perspectives on treatment recommendations and help to fill evidence gaps with multidisciplinary expert opinion. Dovetailing them with CPGs as they are developed will always be a concern.
3. Evaluating AUC from other orthopaedic specialties or organizations to identify possible differences and potential conflicts: In time, the Committee on EBQV will need to conduct a broader survey on which orthopaedic and outside subspecialties have developed AUC similar to those of AAOS. We could add conflict analysis in the future. For example, the North American Spine Society, Scoliosis Research Society, and Musculoskeletal Tumor Society have started down this path, and there is potential for overlap. Other outside related specialties, such as rheumatology, neurosurgery, and physiatry, may have started their own AUC as well. This type of conflict has been particularly evident in the past when the American College of Chest Physician guidelines for DVT and pulmonary embolism prophylaxis have been broadly applied to joint replacements.
4. Physician behavior after AUC: Few studies have examined changes in physician behavior in response to AUC and CPGs. As the literature matures, it would be useful to cover the topic, perhaps with some reader survey data or anecdotes. The Committee on EBQV is currently analyzing Centers for Medicare & Medicaid data to identify the impact of CPGs by assessing procedure-use trends (both in increasing and decreasing specific treatment options) in areas addressed by our recommendations.
5. AUC and the electronic medical record: Are we determining how AUC should be used as decision aids or care pathways? Can they be incorporated in electronic charts?

6. AUC and innovation: There is a risk that strict use or interpretation of AUC will stifle innovation. If we are all using the same guides exactly the same way, there may be little room or appetite for new ideas or innovations. We may be in danger of having the best 10-year-old medicine possible. I was reminded of this possibility recently, when a peer review denied an indicated procedure. In this case, the guide the reviewer was using was last updated 10 years ago and contained no new data. Needless to say, technology and data moved in a new direction over 10 years, so the denial eventually was overturned with some effort and much frustration. This would not have been necessary if the guideline used had been more up to date.

Alan M. Reznik, MD, MBA, FAAOS, specializes in sports medicine and arthroscopic surgery and serves on the AAOS Now Editorial Board, AAOS Communications Cabinet, and CORQ. Dr. Reznik is chief medical officer of Connecticut Orthopaedic Specialists, associate professor of orthopaedics at Yale University School of Medicine, and a consultant.