Computerized adaptive testing (CAT) may be more effective than traditional testing systems in capturing patient-reported outcomes, according to a prospective clinical study presented by Charles L. Saltzman, MD, at the 2012 annual meeting of the American Foot & Ankle Society (AOFAS). The study, which won the Leonard Goldner Award for outstanding research, compared outcome measurement tools in patients undergoing elective foot and ankle surgery.

“Accurately measuring, reporting, and comparing outcomes is essential for improving healthcare delivery,” wrote the authors. “Yet the ability to evaluate and document effective medical interventions depends on the availability of valid, reliable, and efficient outcome measures for research and clinical use.”

The authors also noted that current health status scales face the following three challenges:

• Patient fatigue—the length of time it takes for patients to complete the scale
• Floor/ceiling effects—coverage of the full range of the physical function trait
• Validity/reliability—the precision of the instrument’s measurements

Many existing patient-reported outcome scales cover a broad range but yield imprecise data, or are precise only within a narrow range, noted the authors. In contrast, CAT, which is based on item response theory, enables “dynamic administration of fewer and more appropriate items (from the full test battery) based on a person’s previous response, resulting in more precise measures than traditional short forms.”

Gathering data
The study involved 287 patients (40 percent male; average age, 47 years) who were being treated at an orthopaedic foot clinic and considering or scheduling foot and/or ankle surgery at the time of survey. Patients were asked to complete a web-based survey that captured demographic information and comorbidities. They also completed the following three testing tools, which were administered in random order:

• the Foot and Function Index (FFI)
• the sport module from the Foot and Ankle Ability Measure (spFAAM)
• the Patient Reported Outcomes Measurement Information System (PROMIS)-based Lower Extremity Physical Function Computerized Adaptive Test (LE CAT)

PROMIS, funded by the National Institutes of Health (NIH)Roadmap, is an attempt to develop instruments that can provide both precision and range. The LE CAT has a 79-item bank from which items are drawn based on a person’s previous responses.

Researchers calculated the average time (in seconds) that patients took to complete each of the instruments. They then selected items within each instrument and set them to a common scale to facilitate interpretation. They also calculated person and item measures and standard errors to examine instrument precision and coverage.

Analyzing results
The researchers found that participants completed the LE CAT much faster than the other two tests—in just 66 seconds, compared to 130 seconds to complete the spFAAM, and 239 seconds to complete FFI. The LE CAT test was also shorter than the other instruments, using a mean of 6 items, compared to the 8 items of the spFAAM and the 23 items of the FFI.

“We believe that the significant reduction in the LE CAT’s testing time is an important advantage,” reported the authors. “In our experience, testing time is a major impediment to collecting PROs in orthopaedic patients.”

Correlations among the three instruments found strong relationships, indicating that they were measuring similar constructs. Researchers also indentified that all three instruments were fairly precise at the midrange of the trait, and that they all were less precise at the upper physical function trait level (ceiling effect). But at the lower range (floor effect), the LE CAT was more precise than either the spFAAM or the FFI.

Drawing conclusions
The authors noted that the PROMIS-based LE CAT yielded “equivalent or superior” results compared with the other two scales—and required an average of just 6 items to “produce and maintain a precision level that adequately covered the physical function trait.”

The authors advocate careful consideration of the amount of time required to administer a test (with an eye toward minimizing patient test burden), while also taking instrument content coverage and precision into account. “Of the three scales tested, the LE CAT was either equivalent or clearly superior to the two legacy scales in generating an accurate point estimate of a foot and ankle patient’s lower extremity physical function,” they noted.

“The LE CAT can reduce the burden on patients, researchers, and providers, reduce sample sizes and study costs, all with very high precision,” stated the researchers. “This study substantiates the LE CAT as a superior health status measure for evaluating physical function in foot and ankle patients.”

The authors acknowledged, however, that more research needs to be conducted to validate the widespread adoption of the PROMIS-based LE CAT for measuring patient-reported outcomes.

For more information on the NIH-funded PROMIS measures, visit www.nihpromis.org

Coauthors with Dr. Saltzman (Tornier, Zimmer, Saunders/Mosby-Elsevier, Clinical Orthopaedics and Related Research, Foot and Ankle International) are Man Hung, PhD (nothing to disclose); Florian Nickisch, MD (Smith & Nephew, Techniques in Foot and Ankle Surgery, AOFAS); Timothy C. Beals, MD (GE Healthcare, Foot and Ankle International, AOFAS Research Committee); Tom Greene, PhD (nothing to disclose); and Daniel Clegg, MD (no information available).

Jennie McKee is a staff writer for AAOS Now. She can be reached at mckee@aaos.org

Bottom Line

• This study aimed to determine whether computerized adaptive testing (CAT) provides similar or better precision, coverage, and test administration time compared to two outcome scales currently being used.
• Compared to the Foot and Function Index and the Foot and Ankle Ability Measure sport module, the Patient Reported Outcomes Measurement Information System (PROMIS)-based Lower Extremity Physical Function CAT tool took less time to complete, used fewer questions, and was as accurate or better in estimating a foot and ankle patient’s lower extremity physical function.
• The authors recommend broader use of the PROMIS-based CAT tools with foot and ankle patients to improve precision and reduce the burden on patients in reporting outcome measures.