Douglas W. Lundy, MD, MBA, and J. Lawrence Marsh, MD

The American Board of Orthopaedic Surgery (ABOS) is dedicated to upholding the standards of our profession and ensuring the highest quality of orthopaedic practice possible. The ABOS Oral Examination for certification and recertification is one of the most successful and high-profile examples of practice-based exams in medical certification. During 4 days at the Palmer House in Chicago, nearly 1,000 candidates are questioned on cases selected from their practice.

Although the location hasn’t changed for decades, the exam itself has evolved to remain a relevant, valid assessment of the candidates’ actual practices. Recent changes to the exam process reflect the widespread adoption by hospitals and practices of electronic medical records and images and efforts by the ABOS to improve the reliability of the exam. Maximizing reliability makes the exam as fair as possible for the candidates.

A digital platform
Over the last several years, the ABOS has moved away from paper charts and actual radiographs to a completely digital platform, similar to the actual practice environment for most candidates. Gone are the days when candidates were required to carry massive loads of patient documents and films to the Oral Examination. In 2014, for the first time, the entire examination was in a digital format; all patient records and images were uploaded before candidates arrived at the test site.

The deadline for document upload is roughly 7 weeks prior to the exam. This deadline is required to ensure adequate time to organize the documents so that the risk of technical failure is essentially eliminated. In addition, all documents are loaded onto all of the computers in the exam halls. This time-consuming process provides the redundancy necessary in the event of a single system’s crash, but it demands that uploads are finalized well before the test. This policy is applied evenly to all candidates.

Improved reliability
The reliability of the exam is improved with increased numbers of data points, and the ABOS has made several changes to leverage this fact. For example, beginning this year (2015), all candidates will be required to upload 12 cases, instead of the 10 cases needed in 2014. Because each case is evaluated by two examiners on six facets (Data Gathering, Diagnosis and Interpretive Skill, Treatment Plan, Technical Skill, Outcomes, and Applied Knowledge), these additional two cases will provide 24 more data points for each candidate. The additional two cases will also prevent the need to examine a candidate on the same case twice.

In 2014, the examination process was changed to use four separate 25-minute panels instead of the previous three 35-minute sessions. Each candidate was evaluated by two additional examiners, which increased the number of data points and therefore the reliability of the examination process. This process will continue this year.

Digital scoring
Another change in 2015 will be the introduction of a digital score sheet. This will enable the ABOS to capture more data concerning the delivery of the examination and to upload the scoring data into the system more accurately. In addition, an electronic timer will be used for the first time. The timer will be visible on the examiners’ computers so that they can more effectively manage time during the examination.

Examiner improvements
Exam reliability depends on adjusting the raw scores to account for the relative severity or leniency of the examiners. To accomplish this, examiners must be statistically linked to each other across the candidate pool. This process is enhanced by having examiners work with multiple other examiners.

Beginning last year, the examiner pairings assigned to a candidate were derived from a more robust method of examiner rotations than previously used. This change allowed for increased statistical linking of examiners because the examiner pairs switched with every candidate. The examiners rotated in opposite directions while candidates remained in the same booth throughout the examination.

Enhanced examiner training will also improve the reliability of the exam. Although all examiners are linked statistically, the more they use the scoring rubric in a similar fashion, the better the exam and the higher the reliability. Every year, prior to the exam, all examiners are required to complete online training modules and participate in a scoring linking exercise. In addition, the performance of all examiners is assessed statistically and the results are used to improve performance. Only the examiners who perform within acceptable standards are invited to return in future years.

These changes enable the ABOS to improve the Oral Examination and the quality of the certification process. The Oral Examination is a highly effective method to evaluate a physician’s surgical practice and has an important role in improving the practice of orthopaedic surgery. This process depends on the efforts of the volunteer examiners who provide this professional service.

The ABOS is always interested in recruiting new examiners. Criteria to be considered for this opportunity are as follows:

• Examiners must be ABOS diplomates.
• They must have an ABOS recertification certificate that is less than 10 years old.
• They must be in the active operative practice of orthopaedic surgery.
• They must abide by the ABOS Conflict of Interest Policy for Oral Examiners and all of the orientation and learning elements required by ABOS.
• They must be approved by the ABOS Oral Examination Committee.

The ABOS recognizes the requirements expected from diplomates and advocates for effective certification with the least amount of administrative burden. Most comments from candidates and examiners about these changes have been very favorable.

Readers who are interested in becoming oral examiners for the ABOS should submit a request and CV to Shepard R. Hurwitz, MD, executive director of the ABOS at shurwitz@abos.org

Douglas W. Lundy, MD, MBA, chairs the ABOS Oral Examination Committee; J. Lawrence Marsh, MD, is president-elect of the ABOS.