Most outcome assessments require a substantial amount of time to complete but rarely yield new clinically relevant data. As a result, orthopaedic surgeons seldom use them when making diagnosis or treatment decisions. Similarly, researchers often do not incorporate these tools in clinical studies because they add costs that slow the development of new treatment options.


Published 2/1/2012
Jay D. Lenn

Measuring Outcomes and Realities

OREF grant recipient hopes to develop an instrument that works on both levels

Better patient care depends not only on objective clinical data, but also on outcome measurements based on patients’ perceptions of their well-being. In most cases, outcome assessments are derived from quality-of-life questionnaires completed by patients. Although these questionnaires can provide relevant information, they are not easily integrated into normal clinical or research activities.

Paul M. Saluan, MD

Paul M. Saluan, MD, a pediatric orthopaedic surgeon at the Cleveland Clinic, hopes to develop a dual-purpose pediatric orthopaedic outcomes questionnaire that is simultaneously useful in clinical studies and clinical practice. His work is being supported by a 2009 Orthopaedic Research and Education Foundation (OREF)/Pediatric Orthopaedic Society of North America (POSNA) Research Grant.

The grant is available to POSNA members and supports new investigators by providing seed and start-up funding for promising, clinically relevant research projects.

Defining success
According to Dr. Saluan, accurately judging the success of a treatment depends on patient input. For example, he first saw a 17-year-old soccer player for a second opinion, 1 year after surgical reconstruction of an anterior cruciate ligament tear. By clinical standards, the surgery was successful—no pain, no instability, no swelling, and excellent range of motion. But the patient reported that something didn’t feel right and he wasn’t able to go back to soccer.

“That child by all objective standards did great,” explains Dr. Saluan. “In reality, he was not happy. His knee was not working right.”

Dr. Saluan later found that the placement of the graft, while providing front-to-back stability, didn’t allow good horizontal control and stability for the planting and pivoting necessary for soccer. As a result, for this patient, the initial surgery was not a success.

Too many questions, too little time
A number of questionnaires are used to measure quality of life in adolescent patients with musculoskeletal problems, but these instruments have significant limitations for most clinicians. “The questionnaires currently being used are functional,” says Dr. Saluan, “but in real time they’re difficult to complete as a part of the exam, while sitting in the office with the patient looking at the questions for the first time.”

These instruments often rely on information from proxies such as parents or guardians, lack sufficient specificity, include clinically irrelevant questions, and do not identify the location of the problem. Some instruments try to cover too broad an age range, which also presents problems.

Dr. Saluan hopes to develop and validate a questionnaire for youths aged 11 to 18 years who have pain or injury in their upper or lower extremities. To achieve the goal of creating a tool that is relevant to both clinical research and clinical practice, his team established the following criteria:

  • Is the instrument valid? Does it measure what it is intended to measure? Does it elicit relevant responses?
  • Is it reliable, or does it measure a factor consistently?
  • Is the instrument sensitive to change, so that it captures the effect of the orthopaedic
  • Does it function as a screening tool? Does it attribute a patient’s complaint to a particular body part?
  • Is it brief enough to be completed as part of a typical office visit?
  • Does it primarily depend on the report of the patient? If it requires a proxy, is this person needed only for limited

Asking the right questions
The investigators developed a three-stage process for creating the new patient-reporting instrument. “We tried to identify the most appropriate questions so that we would end up with a highly validated instrument that gives us the most information,” explains Dr. Saluan. “That required a lot of trial and error, examining what’s been done before, and then testing and retesting.”

In the first stage, the researchers developed a draft instrument based on currently published questionnaires, documented conversations between doctors and patients, and held focus groups with experts. The draft document included 50 questions—a far longer instrument than the ultimate goal.

The second stage was designed to assess the validity, reliability, and sensitivity of the instrument. The draft instrument was tested with 1,000 patients who were in treatment for new orthopaedic problems. The patients completed the questionnaire during the initial office visit. A smaller number of patients completed it twice more, before and after treatment.

The final stage was to refine the draft to a more practical instrument—a 20-item questionnaire with approximately the same validity, reliability, and sensitivity as the longer version. This final draft was also tested with in-office, pretreatment and post-treatment feedback loops.

Dr. Saluan predicts that the final product will take a patient 5 to 10 minutes to complete, have real-time clinical relevance, and be applicable for adolescents with a range of extremity problems. He adds, “This project, as we move it forward, has the potential to redefine how we evaluate our patients.”

Jay D. Lenn is a contributing writer for OREF. He can be reached at