The American Board of Orthopaedic Surgery (ABOS) administers computer-based written examinations for both the initial board certification process and also as part of the recertification process. The recertification examination is taken at the end of a Diplomate's 10-year Maintenance of Certification (MOC) cycle. Producing the final product of a scored examination requires a great deal of expertise, time, effort, and expense; this article provides an overview of the process of creating a psychometrically valid multiple-choice examination. The ABOS redesigned the computer-based written examinations in recent years to ensure that they are relevant to the orthopaedic surgeon's practice. The general orthopedic questions have been removed from all practice profiled examinations (PPE). As of 2018, PPEs are available for hand, sports, spine, adult reconstruction, trauma, pediatric orthopaedics, and foot and ankle.

Building a blueprint
The first step in creating a valid examination is developing an examination "blueprint." Blueprints for ABOS examination content are based on a job task analysis performed by expert panels along with information derived from the ABOS Case List Database, a collection of case lists that are submitted for certification and recertification.

This process ensures that the examination content accurately represents what is being done by the practicing orthopaedic surgeon and by what a subject matter expert panel considers critical information. The blueprint is periodically reviewed and modified to accurately reflect changing trends in practice. Blueprints for all ABOS examinations are available on the ABOS website (www.abos.org).

The ABOS solicited assistance from the AAOS and the appropriate subspecialty societies to recruit subject matter experts to write the questions and review the job tasks, competencies, and knowledge base for each of the new ABOS examinations. These groups have developed blueprints for the new examinations and reviewed them with a larger group of subject matter experts. The blueprints are now finalized and have been used to construct the ABOS recertification exams.

Item writing
The ABOS contracts with the National Board of Medical Examiners (NBME) in the test development process. A critically important step is the item-writing process during which examination questions are written, edited, and placed into a final form. Currently, the ABOS has a bank of more than 3,000 questions.

New questions are developed by volunteers on the Question Writing Task Force (QWTF). The QWTF has approximately 40 members, all experienced orthopaedic surgeons from a variety of practice settings and subspecialties. Each member of the QWTF submits 8 to 12 new questions annually. In addition to images and/or diagrams, each question is accompanied by at least two appropriate references.

Although some members of the QWTF have as much as 20 years of question writing experience, new volunteers are added each year. The ABOS encourages Diplomates to submit their applications to join the QWTF—or any volunteer position—through the password protected portal on the ABOS website.

New questions are reviewed by professional editors at the NBME prior to a 2-day QWTF meeting in Philadelphia each spring. At that meeting, specialty orthopaedic surgeons discuss the new questions in their content area. Questions approved through this peer review will be added to the bank of questions.

Each new question accepted, along with questions that have performed well on previous examinations, is designated for use on one or more examinations, including the Part I Certifying Examination and the practice-profiled recertification examinations.

Field testing
Once the Part I Examination is created, it is edited by the ABOS Field Test Task Force (FTTF), during a face-to-face meeting. The FTTF includes approximately 20 ABOS Diplomates. The ABOS Written Examination Committee does the next level of editing.

The chair of the ABOS Written Examination Committee and the ABOS Executive Medical Director perform the final editing. At this point, the questions on the examination have gone through five levels of editing. In addition, all questions in the bank are carefully reviewed and edited on a rolling 3- to 5-year schedule to ensure that each question remains current, applicable, and accurate.

The ABOS is offering three new Practiced-Profiled Examinations for Recertification in 2018 (pediatric orthopaedics, orthopaedic trauma, and foot and ankle surgery). (For more on these exams, see "MOC: What's New and What Do You Need to Know?" AAOS Now, June 2017.)

Taking the test
All ABOS computer-based examinations are administered locally at Prometric Testing Centers. Diplomates choose the location and date that is best for them during the examination period. Diplomates cannot register for the ABOS recertification examination unless they have submitted an application and case list, paid the examination fee, and received a scheduling permit. The sooner they have a scheduling permit, the more likely they are to find an examination time at a Prometric Testing Center that fits their schedule.

By completing the application as early as possible, Diplomates can have up to three opportunities to pass the examination by the end of their MOC cycle. The ABOS highly recommends that Diplomates review the examination tutorial on their own computer so they can familiarize themselves with the user interface employed for all ABOS computerized examinations at Prometric. The examination tutorial and examination blueprints are on the ABOS website.

After the examinations are administered, psychometricians at the NBME summarize the statistical performance of each examination question. A Key Validation Subcommittee reviews the data and poorly performing questions are deleted before scoring. The psychometricians also analyze the degree of difficulty of each question and of the overall examination itself to ensure that the examination is valid, reliable, and produces scoring that is scalable.

With this information, the ABOS Written Examination Committee can set the examination's cut score, also known as the pass/fail point. Thus a Diplomate has the same statistical likelihood of passing or failing no matter what year he or she takes the exam. The ABOS does not set an overall pass rate, only a cut score that can be scaled across numerous years. In recent years, the pass rate for the computer-based recertification examination has ranged from 95 percent to 98 percent.

Examination development and testing requires many steps. With the quality control measures taken by the ABOS staff, a month or more may pass before a Diplomate's results are posted to the ABOS website.

The ABOS is grateful to the many Diplomates who volunteer their time and effort to support this process. The public perception of ABOS board certification and participation in ABOS MOC is an attestation of quality and competence and a real value to our profession and to our patients.
For more information on registration dates, fees, and requirements for eligibility, visit the ABOS website at www.abos.org or call the ABOS office staff at (919) 929-7103.

James R. Roberson, MD, is the ABOS president; Michael S. Bednar, MD, co-chairs the ABOS Written Examination Committee, and David F. Martin, MD, is the executive medical director of the ABOS.