The ProPublica “Surgeon Scorecard” has been available online since July 15. If any AAOS members haven’t yet checked out this tool, it might be a good idea to look yourself up. The scorecard details complication rates for individual surgeons who perform any of eight different procedures, five of which are orthopaedic (hip and knee replacement, cervical spine fusion, and anterior and posterior column lumbar fusion).

ProPublica, founded in 2007, describes itself as “an independent, nonprofit newsroom that produces investigative journalism in the public interest.” Although it accepts advertising, major funding comes from philanthropic organizations.

To create the Surgeon Scorecard, ProPublica analyzed Medicare billing records for nearly 17,000 surgeons and identified 66,569 complications over 5 years (2009–2013). Complications were reported if the patient was readmitted to the hospital within 30 days of the index procedure. The cost of these readmissions was estimated at $645.3 million.

Patients were assigned a “health score” based on age and coded comorbidities such as diabetes and obesity. Scores were also stratified by hospital quality. However, many experts have commented on the low level of risk stratification applied in the “adjusted rates.” In an interview with Medscape, AAOS President David D. Teuscher, MD, worried that adjustments to complication rates could not account for all patient differences. He specifically cited physical deformity and prior surgery as risk factors.

Media reaction and traction
Media coverage of the Surgeon Scorecard has, thus far, been mixed. Gannett-owned USA Today compiled stories from Gannett-owned local newspapers. These local stories usually mentioned high- and low-risk area surgeons—mostly orthopaedists—by name. Many identified the wide variation in “adjusted complication rates” among surgeons practicing in the same hospital. They chalked differences up to “genuine doctor skill, a strict adherence to best practices with procedures, volume of surgeries, and direct follow-up and personal contact.”

The reader comments on these articles were fascinating. Most complained that the list didn’t include the surgery that they were considering. Others noted that the scorecard did not grade surgeons on more specific issues. For orthopaedic surgeons, these included “well-positioned components.” Actually, malpositioned implants were among the complications assessed, but only if the problem led to readmission within 30 days.

The likelihood of finding these surgeon-specific complications in the database would vary by the type of surgery. For example, malpositioned total knee implants may not be revised for months or years. However, a malpositioned medial pedicle screw causing a new, severe, postoperative radiculopathy may very well require removal within 30 days. In both cases, if revision occurred during the index hospitalization, it would not be in the data. The only complication included from the index hospitalization was death.

As Jeffrey Parks, MD, a general surgeon in Cleveland, noted in his blog, “Why is ‘return to OR’ not there? What about post hemorrhage and need for transfusion? What about a surgeon who all too regularly whacks a common bile duct and transfers the patient immediately to a tertiary care center where it is promptly repaired, and the patient never gets readmitted?”

In Forbes, urologist Benjamin Davies, MD, examined the Surgeon Scorecard results for radical prostatectomy. He noted that the most common complications leading to readmission were “digestive system complications.” Not included in the data: complete removal of the cancer or preservation of erectile function. He asks readers whether they would trade erectile function for a period of constipation. (The preponderance of direct-to-consumer advertising for products that address both of those issues may make his question a moot point.)

Critiques
Obviously the Surgeon Scorecard has some limitations. But, as a first effort, is it a worthwhile or premature effort? Either way, will it make a difference?

Many critics cited the limited dataset (Medicare only; no Medicaid or private payer data) employed. Some noted that the small datasets lead to wide confidence intervals that, for many listed surgeons, extended across risk categories. Almost every spine surgeon I looked up was “medium risk.”

Studies using administrative databases built from billing codes are increasing in the orthopaedic literature and elsewhere. But billing codes can be manipulated or misapplied in ways that have a marked impact on the data retrieved from hospital records.

Errors or code misapplications may result from deliberate up- or downcoding or miscoding due to poor documentation in the patient record. The order in which codes are applied may also affect how subsequent computer analyses “see” the cause-and-effect of the patient’s stay. For example, one hospital may aggressively code complications and comorbidities to maximize reimbursements. Another hospital, with a different payer mix or competitive milieu, may code conservatively to “look better” than its competitors.

Different databases may lead to very different conclusions. Using hospital-only databases can be problematic when measuring complications, such as surgical site infections, that occur both before and after discharge. For answering clinical questions, data derived from billing codes are less satisfactory than data abstracted from charts.

Additionally, the ProPublica methodology does not account for “patient factors such as their social support system, physician factors such as willingness to accept risk, and factors effecting access to care such as the presence of observation units or care in the emergency department,” according to Peter Pronovost, MD, director of the Armstrong Institute for Patient Safety and Quality, Johns Hopkins Medicine.

Most comments on the scorecard, however, have been positive. They acknowledge the limitations but point out the lack of any available data for most patients.

As physicians, we are all familiar with the whispers within hospitals about subpar physicians. Many of us have told colleagues or family members, “If I get in an accident, don’t let Dr. A take care of me.” Shouldn’t patients also be forewarned?

More importantly, Dr. A probably does not see himself as subpar. Our system for collecting and acting on outcomes and complications data remains rudimentary.

Regardless of what we think about the scorecard, it’s out there. And public reporting measures will only increase as time goes on.

The road forward
As a profession, orthopaedic surgery must continue to engage its members, the public, and other stakeholders to improve safety and outcomes. We should be generating (and sharing) better quality data. Eventually, quality reporting systems must include long-term functional outcomes data. A safe surgery is only helpful to the patient if it was a useful surgery. Any model employing 30-day readmissions data or other process variables will favor marginally indicated surgeries over interventions in more profoundly affected individuals.

We must find ways to risk stratify the data so that the total hip revisionologist is not penalized. As Dr. Teuscher has said, “The danger in these ratings is that the most vulnerable patients, who are the most difficult to get a good result on, the ones that need the most to be fixed, would not have access to care.”

Maybe our discomfort with the ProPublica Surgeon Scorecard comes from the fact that we were not involved in its methodology and we understand the limitations of its data source. But its publication should stimulate us to move our own reporting and safety mechanisms into high gear. That’s one reason the AAOS Now Forum next year will focus on risk assessment.

Sure, ProPublica’s report card has its limits. But, if my cousin in California needed a cholecystectomy, I might suggest she look up her surgeon before scheduling the procedure.

Additional Information:

• ProPublica Surgeon Scorecard
• Loews R: Online Scorecard Shows Surgeons’ Complication Rates. Medscape Medical News July 15, 2015.
• Surgeon Scorecard: What Gannetts Local Newspapers Found. USA Today July 14, 2015.
• Unger L: Going under the knife? Check on your surgeon. The Courier-Journal July 20, 2015.
• Parks J: Why the Surgeon Scorecard is a journalistic low point for ProPublica. KevinMD.com July 22, 2015.
• Davies B: ProPublica’s Surgeon Scorecard: Clickbait? Or Serious Data? Forbes July 14, 2015.
• Pongpirul K, Robinson C: Hospital manipulations in the DRG system. A systematic scoping review. Asian Biomedicine 7(3),2013:301–310.
• Grauer JN, Leopold SS: Large database studies–What they can do, what they cannot do, and which ones we will publish. Clin Orthop Relat Res 473(5),2015: 153–159.
• Bannow T: Most Central Oregon docs see moderate complication rates. The Bulletin, July 15, 2015.
• Society of Thoracic Surgeons: STS Public Reporting Online.