Orthopaedists provide feedback during open forum at Annual Meeting
During the 2017 Annual Meeting, orthopaedic surgeons and other stakeholders listened intently as representatives from U.S. News & World Report described the publication's plan to publish individual hip and knee surgeon performance analyses. These score cards are expected to be made available online at USNEWS.com sometime this fall.

The event was moderated by AAOS President William J. Maloney, MD. Panel members included Gerald R. Williams Jr, MD; Antonia F. Chen, MD, MBA; Thomas S. Muzzonigro, MD; Kevin Shea, MD; and Adolph J. Yates Jr, MD.

U.S. News & World Report

"We are here today not to endorse a specific ratings system, but to listen to what U.S. News & World Report plans to do and provide feedback," explained Dr. Maloney.

Ben Harder, chief of health analysis, and Geoff Dougherty, MPH, senior health services researcher, U.S. News & World Report, presented the methodology and preliminary results of the publication's analysis. "Physician quality evaluations are widespread on the Internet and they are not going away. It's no longer a question of whether surgeons will be publicly measured, but rather by whom and by what methods," Mr. Harder told attendees. "Our purpose is to provide patients with decision support. We believe that, with your input and constructive feedback, we can make what's available to patients in the public domain better reflect the quality of care that you and your colleagues provide."

Draft methodology
Mr. Dougherty noted that although their methodology for evaluating hip and knee surgeon performance remains under development, the framework is based on the Donabedian model, which focuses on structure (eg, provider characteristics), process, and outcomes.

Data for the preliminary analysis were obtained from Medicare claims data (Part A Standard Analytic Files, which allow tracking of patients across multiple encounters) and Doximity physician profiles (see sidebar). Surgeons who had performed at least five primary total hip arthroplasty (THA) and/or total knee arthroplasty (TKA) procedures for the treatment of osteoarthritis in Medicare beneficiaries between October 2010 and October 2015 were included in the analysis. Surgeons also needed to operate at acute care or surgical specialty hospitals that had a volume greater than 15 cases over 3 years. Cases that involved concurrent bilateral total joint replacement procedures or concurrent partial procedures were excluded.

U.S. News & World Report,

Quality indicators included procedure volume, specialization (the ratio of index procedures to all major procedures), training, the U.S. News hospital quality score for the index procedure, and transfusion use.

"It's important to note that this is a very tight distillation of the indicators," said Mr. Dougherty. "A variety of others were tested empirically and were rejected."

Derived from the Centers for Medicare & Medicaid Services (CMS) claims data, risk-adjusted outcomes included discharge disposition status; extended hospital length of stay (LOS); 5-year revision and 30-day readmission rates; and CMS risk-standardized complication rates, including surgical site infections. The researchers also adjusted for the following:

• socioeconomic status, using Medicare/Medicaid dual eligibility as proxy
• comorbidities (Elixhauser Comorbidity index, 29 risk predictors)
• age
• sex
• Medicare entitlement reason (aged, disabled)
• transfer status
• year of admission

Confirmatory factor analysis was used to combine all of the indicators into a single summary quality score for each surgeon. Statistical inference was used to test for positive outlier (ie, high performing) status.

"The benefit of [confirmatory factor analysis] is two-fold," said Mr. Dougherty. "First of all, it enables us to account for and eliminate a lot of the measurement errors across all these indicators. Secondly, indicators that are more reliable will receive a higher weight in our final score."

Preliminary results
Results for approximately 13,000 hip procedures and 16,000 knee procedures were included in the preliminary analysis. Most of the surgeons were board certified and approximately 4 percent were fellowship trained.

A strong relationship was found between high volume and high performance for both hip and knee surgeons. Specialization, hospital score, and surgeon level of training were also important indicators of high performance in both cohorts. On average, surgeons operating at teaching hospitals had higher quality scores than those operating elsewhere. Surgeons who perform technically complex cases, serve on American Association of Hip and Knee Surgeons (AAHKS) and AAOS boards of directors, and who have a high percentage of returning patients for contralateral hip or knee replacement also had high-quality scores.

Approximately 2,500 surgeons were tentatively identified in the model as statistically high performing in either hip replacement, knee replacement, or both operations. "All other surgeons who perform either or both of those surgeries will be identified as 'unrated.' Importantly, we will not publish negative results and we will also not publish rankings," said Mr. Harder.

Orthopaedic surgeon feedback
Mr. Harder and Mr. Dougherty admitted that their work at this point has limitations, including the inability to address functional outcomes or patient experiences. In addition, CMS does not routinely include Medicare HMO cases in the Standard Analytical File, and so very few cases from HMOs appear in the file. Another drawback is that surgeons who do not treat Medicare patients—including some who may be high volume in the index procedure, which according to the model is associated with high performance—will not be rated.

When asked how U.S. News & World Report acquires individual surgeon revision rates, Mr. Harder responded, "A surgeon's risk-adjusted revision rate is based on the number of his/her index cases who underwent subsequent revision during our study period, whether or not the same surgeon performed the revision. In CMS data, we can identify which surgeon (by NPI [National Provider Identifier]) was the operating physician in an index case that preceded a revision operation—performed either by the same surgeon or any other surgeon—in the same Medicare beneficiary. It should be noted that ICD-9 data do not indicate laterality, which is a limitation of this measure."

Dr. Maloney, however, stressed that this method is not reliable, as the currently available data from CMS are based on ICD-9 codes. "A given patient could have a primary hip one year and a revision hip 2 years later. It would be difficult to know if the index operation was the hip revised," he said.

William Jiranek, MD, FACS, past president of AAHKS, added that using the CMS Administrative database to attribute cases and outcomes to surgeons presents a fundamental problem. "CMS has difficulty separating the cases and outcomes of individual surgeons from the collective tax identification numbers of their groups or institutions. How U.S. News & World Report can improve that using the same administrative database is unclear," he said.

"It appears that U.S. News & World Report has done a good job with the data available; however, the data is imperfect," agreed Richard Iorio, MD, of NYU Langone Medical Center, Hospital for Joint Diseases. "I am concerned that some surgeons will be missed, miscategorized, and possibly damaged by the process. I don't think it is ready for primetime."

Orthopaedic surgeons also voiced concerns about how risk factor adjustment may affect patient access.

"The U.S. News & World Report model does not account for the different patient populations that different surgeons treat, and there may be substantial differences in the risk profile and comorbid conditions among these different populations. Without the ability to risk stratify patients adequately, surgeons caring for the most complex and challenging patients will not be fairly assessed," said Dr. Jiranek. "This may create access problems as surgeons seek to avoid higher risk patients who could negatively impact their ratings. It is not clear how U.S. News & World Report improves upon currently used inadequate risk adjustment methods."

"Ratings of physicians and surgeons have, for the most part, been plagued by poor methodology and bias," added Thomas P. Sculco, MD, past president of The Knee Society. "It is important that any process by U.S. News & World Report be transparent, accurate, and recognize the diversity of patients cared for, which may influence measures like LOS and the need for postdischarge rehabilitation. The potential downside of not providing care to lower socioeconomic populations and patients with more comorbidities is a risk if it is not reflected in these rating methodologies," he said.

At the conclusion of the meeting, Dr. Maloney assured attendees that the leadership of AAOS and other stakeholder societies would continue to provide Mr. Harder and Mr. Dougherty with feedback as they work to further develop the ratings methodology.

"U.S. News & World Report is a highly respected source of information used by thousands of patients seeking healthcare information," said Kevin L. Garvin, MD, president of The Hip Society. "We look forward to providing the input necessary to create an accurate and objective compilation of highly skilled hip surgeons."

Adolph V. Lombardi Jr, MD, FACS, president of The Knee Society, echoed those sentiments. "The Knee Society appreciates the time and the outreach on behalf of U.S. News & World Report to present their methodology to the adult arthroplasty community in an open forum," he said. "Together with our orthopaedic sister organizations, The Knee Society will participate in any future planned joint efforts to ensure our patients have access to the information—and care—they deserve."

Maureen Leahy is assistant managing editor of AAOS Now.

Update your Doximity profile
Information used in the U.S. News & World Report ratings includes data on orthopaedic fellowship training sourced from Doximity—reportedly the largest network of physicians and healthcare professionals in the country. However, an initial survey of Doximity suggests that the compliance rate of physicians reporting their adult reconstruction training to Doximity is quite low—less than 50 percent, according to Dr. Jiranek. Orthopaedic surgeons are therefore urged to update their Doximity profiles at www.doximity.com

AAOS has provided U.S. News & World Report access to its online fellowship list as a resource for their research. Orthopaedic surgeons who wish to search fellowship listings can visit the Membership/Career Development page.