E-mail this article to a friend  Download this article in PDF format

Evaluations go electronic

By J. Lawrence Marsh, MD

“The times, they are a-changin’”

In 1963, the AAOS administered the first annual Orthopaedic In-Training Examination (OITE) to evaluate orthopaedic residents and residency programs. That year, 1,118 residents from 171 training centers took the exam, which consisted of 150 multiple-choice questions. Not only was the AAOS the only medical specialty society to begin producing an annual examination for residents, the exam was a first in medical education.

The Orthopaedic Self-Assessment Examination (OSAE) was introduced in 1975. Since then, the AAOS has established 10 new subcommittees to develop additional special interest examinations, but the basic process of exam development and the format of the exam have remained the same. New technologies, increased educational demands, and macroeconomic forces, however, are changing the way the Evaluation Committee and its subcommittees produce assessment, home-study, and learning programs.

Streamlining the development and scoring process
During the past year, the entire exam development process has become electronic. Instead of a 500-page printed field test book, we now have Web-based booklets, arranged by content domain on a secure Web site for the committee. In addition, reviewers get the booklets on a flash drive so they can access their questions in multiple formats and return their comments as electronic files.

The shift to content domain electronic files improves workflow by enabling reviewers to return sections as they are completed. Review comments are compiled and converted to a portable document format (PDF) on the secure Web site. This also enables the posting of early results and updating of review comments.

Lastly, to accommodate a longitudinal perspective of programs’ involvement with the examination program, OITE score reports for the previous 4 years are now posted as PDF files in the program director’s profile site. Individual OSAE score reports are posted in the member’s profile site. This change enables program directors and members to directly access to their own score reports.

Introducing electronic testing
As the Academy updated the exam development process (Fig. 1), the exam taking process also changed. Instead of getting a question book, figures book, and answer sheet, residents who take the 2009 OITE will get a DVD. Because the DVD is not Web-based, it eliminates the need for residents to travel to a test center to take the exam.

Fig. 1 Howard Mevis, AAOS director of electronic media, evaluation programs, course operations, and practice management group, watches as Evaluation Committee member A. Samuel Flemister Jr., MD, tests the new electronic OITE.

The newly created examination software mimics the paper-and-pencil test taking experience as closely as possible, with several new features. It enables the inclusion of a higher level of imaging information for questions that include magnetic resonance or computed tomography axial, coronal, and sagittal views (Fig. 2). Video provides greater detail for anatomy, clinical examination findings, and surgical procedure questions.

Fig. 2 The new OITE exam on DVD includes images and video, making it a valuable educational product for residents.

Best of all, the software maintains test security and serves as an examination proctor. Once residents begin taking the test, they cannot venture out to the Internet or open another software program because the OITE software will shut down automatically if they do. The examination proctor will need to restart the program.

In addition, residents keep the DVD as an enduring educational product that provides them with the correct answer and their actual answer from the exam. By graduation, residents will have a library of more than 1,300 examination questions searchable by content domain. The integrated note feature will enable residents to add discussion notes, additional references, and comments from clinical experiences.

Looking ahead
The changes to the process are both environmentally and fiscally responsible. As more committee work is done electronically, AAOS fellows will have more time for patient care. These permanent changes to the development process and the OITE are just the beginning of several new Academy initiatives under the exams program.

Maintenance of Certification™ requirements from the American Board of Orthopaedic Surgery, for example, have increased the profile of assessing medical knowledge on multiple choice exams. An Academy project team is developing a new online examination program that enables learners to choose the content of their exam based on their self-identified educational needs. In addition, this question-based program will create an online learning community where orthopaedic surgeons will be able to share their knowledge, additional information from the literature, and post case information to increase the value of home study education. Utilizing Web 2.0 technology, these concepts create opportunities for all participants to be learners and teachers. Watch for more information on this program later this year.

J. Lawrence Marsh, MD, is chair of the AAOS Evaluation Committee. He can be reached at j-marsh@uiowa.edu

Did you know…?

  • More than 4,000 orthopaedic residents at more than 200 programs take the OITE every year
  • The switch to an electronic examination development process will save more than 65,000 sheets of paper annually
  • The new development process drastically reduces staff time and eliminates the need for temporary help to manage mailings
  • The new examination process eliminated the need for producing 10,000 books (question books and figure books)
  • The first Subcommittee on Examinations and Evaluations was established in 1963 under the Committee on Graduate Medical Education

AAOS Now
June 2009 Issue
http://www.aaos.org/news/aaosnow/jun09/clinical4.asp