AAOS evidence-based medicine (EBM) products, such as clinical practice guidelines (CPGs) and appropriate use criteria (AUC), are created by and for clinicians of multiple disciplines to summarize the best available evidence for various procedures across multiple orthopaedic disease topics. The essential foundation for all evidence-based products is a quality systematic literature review, which must be meticulously constructed in the following sequential stages:

• a priori development of questions to guide a focused literature search
• abstract review
• full-text review
• evaluation of study quality
• data extraction
• data analysis
• table/chart construction

Organization and data management are paramount to ensuring an accurate and robust literature review throughout these many stages of development. Additionally, for EBM products such as AUC, which contain treatment recommendations for a large number of patient scenarios, ensuring user-friendly access to the interpretation of the results is important.

The AAOS department of research and scientific affairs, along with an external web developer (WebAuthor), has created the following web-based tools to organize and track the development of EBM products:

• PEER (Presentation and Evaluation of Evidence-based Research)—Provides an electronic structure for A through Z development of systematic literature reviews
• AUC ballot tool—Electronic voting, organization, and analysis of appropriateness ratings for a large set of patient scenarios/treatments
• AUC web-based application—User-friendly access to treatment recommendations for a multitude of patient scenarios associated with an orthopaedic condition

The initial goal of creating the web-based tools was to increase the efficiency of development and dissemination of the AAOS EBM products. However, these tools are applicable to any healthcare organization, academic facility, or private researcher developing a systematic literature review, CPG, and/or AUC.

How PEER works
PEER is a secure web-based systematic literature review development tool that can help organize and simplify all aspects of the process (Fig. 1). For example, PEER can do the following:

• Create unique entries for each reference/abstract that is returned from the literature search.
• Provide a platform for systematic review developers to evaluate abstracts and recall full-text articles with a single click; if the abstract is available online, it can be linked directly to the article.
• Track and search for various markers, including articles marked as “included” or “excluded,” author, or year.
• Record the strength of a given study using quality evaluation forms for diagnostic, intervention, and prognostic studies; create quality tables and export them as Word or PDF documents.
• Extract, organize, and save all relevant data.
• Generate a bibliography of all included and/or excluded studies and export it to a Word document with one click.

PEER is especially helpful when multiple projects are ongoing. Once a full-text article has been loaded into the system for one project, the system scans all other projects and automatically attaches the PDF to any matching references. Its web-based functionality facilitates easy collaboration among reviewers who may be working on the same project in different locations. A built-in question tracking system ensures that systematic review methodologists can easily collaborate with content expert clinicians and a statistician to ensure correct interpretation of articles.

Security encryption and multiple daily off-site backups ensure against data compromise. Progress is continuously measured and displayed in user-friendly graphs and charts, which are exportable as Excel and Word documents.

After the systematic literature review is completed, the extracted data from each of the individual studies included in the review can be made public to complement the review. This function is also a good resource for researchers and educational institutions (Fig. 2).

Voting on AUC
Creating AUC can be burdensome for the voting panel, which must rate the appropriateness of a multitude of treatment options for hundreds of patient scenarios. Combining patient scenarios and treatment options may result in up to 6,000 voting items for one AUC topic. To ease the burden of the voting process, AAOS staff have designed a web-based balloting system with the following features:

• Electronic ballots (Fig. 3) are secured behind a unique login and password for each user. Ballots are saved after each rating is completed, and all ratings are backed up via multiple servers.
• A filtering function enables voting panel members to filter by one or more patient indications (eg, fracture type), creating logical chunks for rating. Filters may also be used to show only those ratings that have not been completed.
• A progress bar shows in real-time the percent of ratings not yet completed.
• A bulk update feature allows voting panel members to rate the appropriateness of treatment for similar patient scenarios all at once, instead of individually.
• A stats section affords the administrator, manager, or moderator an overview of whose ratings have been completed, each voting panel member’s rating, and the group’s median rating for each patient scenario. It also shows whether there was agreement or disagreement among the voting panel members.

After all ratings are in, they are exportable to Excel or Word format for further analysis or presentation. Once the final ratings have been submitted, the AUC Ballot calculates the median scores and agreement ratings and imports the data into the AUC Web-Based Application. Administrators have the ability to test the AUC Application before making it public.

Clinical use of AUC
The over-arching goal of AUC is to prevent underuse of appropriate treatments for patients and prevent overuse of rarely appropriate treatments for patients who have a specific orthopaedic disease. Practical uses for AUC may include the following:

• aiding in clinician-patient decision-making and discussions
• filling in the gaps and catalyzing new research in areas where research is currently lacking
• producing educational and quality improvement tools

The AUC app (www.aaos.org/aucapp) provides access to treatment recommendations. Clinicians can enter the patient indications of interest, print the treatment recommendations or save them as a PDF, and access the evidence relevant to the disease topic of interest electronically via PEER.

These tools were developed for AAOS and have been very helpful in streamlining the systematic review, AUC development, and AUC implementation processes for both clinician volunteers and staff.

Recently, AAOS has had inquiries from other organizations looking for help in managing these tasks. Readers interested in learning more about using one or more of the tools mentioned in this article for their own development of systematic literature reviews and/or AUC should contact the authors.

Jayson Murray, MA, is manager of the AAOS EBM unit and can be reached at jmurray@aaos.org; Ryan Pezold, MA, is a research analyst in the AAOS EBM unit and can be reached at pezold@aaos.org