PRO data, which can be collected electronically, can have a significant impact on health services delivery.
Courtesy of Thinkstock


Published 2/1/2017
Joseph P. DeAngelis, MD, MBA; Neha Agrawal, MPH

PROs in Practice

An end user's perspective
In the first article of this two-part series on the use of patient-reported outcomes (PROs) (see "Collecting and Using Patient-Reported Outcomes in Everyday Practice," AAOS Now, January 2017), Charles A. Goldfarb, MD, from Washington University in St. Louis, shared his experiences with implementing the Patient-Reported Outcomes Measurement Information System (PROMIS®). In this article, we talk with Bradley J. Nelson, MD, from TRIA Orthopaedics in Minneapolis about his experiences with collecting PRO data and its impact on health services delivery.

Dr. DeAngelis: In what type of setting do you practice? Do you have a concentration within orthopaedics? How long have you been in practice?

Dr. Nelson: I am a sports medicine orthopaedic surgeon at the University of Minnesota in Minneapolis. I have been in practice for about 17 years. My primary practice site is TRIA Orthopaedic Center, a full service orthopaedic center that includes an ambulatory surgery center, imaging, and rehabilitation. Approximately one half of the orthopaedic surgeons at TRIA work for the University of Minnesota; the remainder are employed by a large health system.

Dr. DeAngelis: How long have you and your team been collecting PROs? What system do you use and what types of data do you collect? What have you learned?

Dr. Nelson: We have been collecting PRO data at TRIA since the center opened about 10 years ago. It started with a couple of diagnoses and now we collect PROs on six unique diagnoses.

We started collecting information on our high-volume procedures; some are high cost and others are not. The diagnoses include anterior cruciate ligament (ACL) reconstruction, rotator cuff repair, carpal tunnel syndrome, total knee arthroplasty, total hip arthroplasty, and bunion surgery.

We learned quite a bit by collecting PRO data. For example, the PRO data on carpal tunnel showed there is not a lot of variability in outcomes or cost for the procedure. As a result, we discontinued collecting PRO data on carpal tunnel procedures. If procedures do not have significant variability in either outcome or cost, there may be no value in continuing to collect that data for the purposes of quality improvement. We are now trying to use and analyze the PRO data we collected on carpal tunnel procedures for journal articles.

Dr. DeAngelis: That's great insight, because I think there is a lot of criticism when it comes to standardizing outcomes measures. Have you a seen a difference between carpal tunnel syndrome, rotator cuff, ACL, and total joint arthroplasty?

Dr. Nelson: Outcomes for rotator cuff repair and ACL reconstruction are more variable than the other procedures; it actually gets uncomfortable when you use outcome measures for quality control purposes. If there are 10 sports medicine orthopaedic surgeons and eight of them have similar outcomes, you have to decide what to do with the outlier data from the other two surgeons.

Dr. DeAngelis: That makes sense. What patient-reported outcome measures (PROMs) have you used for ACL patients?

Dr. Nelson: We use the Knee injury and Osteoarthritis Outcome Score (KOOS), Marx's activity score, and the single assessment numeric evaluation (SANE). We do not use PROMIS; we use  diagnosis-specific outcome measures.

Dr. DeAngelis: Are you collecting disease-specific outcome measure data on everyone who has a procedure or may have a procedure? How do you decide who is included in the measure?

Dr. Nelson: Currently, we collect PRO data on patients going into the operating room because it's the easiest location to ensure nearly 100 percent completion. The program is successful because there is very little burden on the orthopaedic surgeon. Patients complete their outcome measures electronically in the preoperative waiting area, and complete their follow-up measure data via email. We're trying to develop an external or remote link to the presurgery questionnaire so the patient can complete it at home prior to their procedure.

Dr. DeAngelis: Many orthopaedic surgeons may worry that they would be personally responsible for collecting PRO data if close to 90 percent completion rates are required. Can you address this?

Dr. Nelson: At our facility, the medical assistant—not the orthopaedic surgeon—is responsible for collecting the data. For example, if a 1- or 2-year outcome measure is required from the patient, the medical assistant gives the patient an iPad to complete the questionnaire. The nurses are also familiar with the process and will often provide the iPad to the patient. We have nearly 100 percent collection of presurgery data for each single year. However, the completion rate for follow-up visits decreases to approximately 70 percent at 1 year and 50 percent at 2 years. This completion rate is adequate for quality improvement work but usually does not meet criteria for scholarly publication. For that, we would want about an 80 percent completion rate.

Dr. DeAngelis: Who decided on which measures to use and how was that decision made?

Dr. Nelson: Our Research and Outcomes group looked at the various diagnoses and tried to select a diagnosis that has high volume, high variability, or high cost. We also included total knee arthroplasty and total hip arthroplasty in our diagnoses selection. Although the variability is not high, the cost of those procedures is. In addition, the state of Minnesota requires outcomes collection for total knee arthroplasty. We also believe that payers would be interested in the data on high-volume, high-cost, or high-variability procedures. For example, payers are probably less interested in carpal tunnel surgery data because it is relatively low cost and there is low variability. But if you are doing total knee replacement in the outpatient setting, a payer will want to know your quality data and PROs in particular.

Dr. DeAngelis: You acknowledged that ACL reconstructions and rotator cuff repairs are associated with high variability. What are you doing with this information and what conversations are being generated?

Dr. Nelson: At the beginning we gave providers their individual data, as well as other providers' data (de-identified) from the institution. This enabled our orthopaedic surgeons to compare their PRO data to that from orthopaedic surgeons in the rest of the organization. Now we identify the data for all orthopaedic surgeons, with the goal of facilitating discussion. Since our outcome data are not very different among surgeons, most of our discussions are focused on cost. For example, surgeons are starting to understand the effect of allograft and implants on cost.

Dr. DeAngelis: How has this understanding changed practice? Have you performed fewer allografts in the surgery center?

Dr. Nelson: I think orthopaedic surgeons will change their practice patterns based on their own patients' outcomes and cost. We are adjusting our practice based on the data and have performed fewer allografts.

Dr. DeAngelis: Has there been any dialogue about making a change in vendor relations or materials acquisition?

Dr. Nelson: That's a really good question. It's difficult to get orthopaedic surgeons to do that—they often choose implants based on their comfort level and training. We are just beginning to get more detailed data on cost, which we hope will enable surgeons to make more informed choices.

Dr. DeAngelis: Your group has been collecting PRO data for more than 10 years now. Overall, has it been a positive experience? Is there any lingering animosity because of the changes that collecting PRO data have prompted?

Dr. Nelson: We have had numerous discussions regarding the role of PROs in our practice. Overall, I think the experience has been positive and we have learned a lot about the process. There is not much animosity, but orthopaedic surgeons are competitive by nature and they want their patients to do well.

Dr. DeAngelis: We have spent a lot of time talking about physician behavior and interaction. What about patients? What has their engagement been like in this process?

Dr. Nelson: I'd say patient engagement at our institution has been mixed. Patients who complete many surveys can get fatigued. They do want to know their data, however, and so we are trying to improve on getting patients their own PROs. Although we publish our overall PRO data for each diagnosis on our website, we want to reach a point where we can provide the patients their individual data in real time. We need to provide the patient with something of value for completing outcomes and probably the most important value is the actual number. The patient could then put that number in perspective with the institutional, state, regional, or national average.

Dr. DeAngelis: Has physician-patient communication changed because of these measures?

Dr. Nelson: Not yet. I expect this communication will change when we provide the patient real-time data. Our surgeons are concerned that much of the patient visit would focus on a number and what that number means. We have some work to do in developing the best way to discuss the PRO data with the patient. In addition, we do not have the resources to collect PROs on every patient at every visit. This means we have to explain to patients why we are collecting the data for their particular problem. The hope is that by explaining our rationale we develop patient "buy in," which will improve our ability to capture follow-up data.

Dr. DeAngelis: Based on your experiences, do you have any advice for orthopaedic surgeons?

Dr. Nelson: It is helpful to think carefully about the reasons for collecting PRO data and what diagnoses are most important to monitor. In our system it was not possible to collect data on every patient, so we tried to choose a diagnosis where a change in surgeon behavior would likely have the most impact. I think starting with a diagnosis that has high variability in terms of outcome and cost makes the most sense. Orthopaedic surgeons could also benefit from guidance from the AAOS, including access to national data on variability.

It is also important to identify the most efficient data collection points within the patient workflow. Although PROs are not physically collected by the orthopaedic surgeon, it is critical that the surgeon establishes the culture for PRO collection; otherwise, it will be difficult to get the organization on board.

Dr. DeAngelis: In light of impending value payment models, what should orthopaedic surgeons be doing right now with respect to PROs?

Dr. Nelson: Orthopaedic surgeons need to get comfortable with the idea of collecting PROs in their practice because we will probably be required to provide this data in the near future. Although surgeons may not know the exact PRO data that will be required, developing a small data collection project now can help establish the culture and assist in process development.

Joseph P. DeAngelis, MD, MBA, is a member-at-large of the AAOS Performance Measures Committee. Neha Agrawal, MPH is the Quality & Performance Measurement Specialist in the AAOS Research, Quality & Scientific Affairs Department.