Serving on the AAOS Evaluation Committee has been one of the most important professional roles I have ever had in my AAOS and academic careers. I began writing questions for the Orthopaedic In-training Examination (OITE) in 1999 when the Central Evaluation Committee was led by Stephen A. Albanese, MD. Four years later, I took over the chairmanship of a specialty evaluation subcommittee, and I then went on to chair the Central Evaluation Committee (2010–2013).

The roster of orthopaedic surgeons who have served on the Evaluation Committee, been involved with test question writing, and gone on to leadership roles at the Academy or in academic medicine is rich—and much too long to list. Some individuals have spent more than a decade on the Committee, while others have spent a shorter time. But all have recognized the value of the experience.

“I have been involved in the AAOS for the past 30 years in many different positions,” said former AAOS President Joseph D. Zuckerman, MD. “However, it all started in 1985 with my appointment to the Committee on Hip and Knee Evaluation. It was a wonderful experience that only made me want to be more involved.”

Similarly, Paul Tornetta III, MD, who chaired the Evaluation Committee for 6 years, stated, “Serving on the Evaluation Committee not only kept me up to date in all areas of orthopaedics, but allowed me to get to know academic surgeons in other subspecialty areas and form stronger networks for other activities. It was some of the most valuable work that I have done.”

My own path to the Evaluation Committee started at the 1997 AAOS Basic Education Course. The course faculty (James C. Farmer, MD; Frederick G. Lippert III, MD; John F. Sarwark, MD; and John Murnaghan, MD) stressed the importance of test question writing in orthopaedic education. I realized that I needed to learn about and get involved in the Academy’s test question writing programs.

The beginnings of the OITE
More important than the individuals involved with OITE question writing, the OITE itself has a robust history. Carroll B. Larson, MD, who chaired the Examination and Evaluation Subcommittee in 1962, proposed three goals for the OITE: assess the quality of training programs, determine some national standards for orthopaedic residents, and measure resident knowledge against the norm.

When it was launched in 1963, the OITE was the first in-training exam of any specialty. The original OITE was designed without an answer key for the following reasons:

• It would create a cognitive dissonance among examinees if there were multiple correct answers.
• Giving the answer key would remove residents’ motivation to use the examination as a study guide.
• Providing an answer key would remove the possibility that the question could be used on a future In-training Examination itself or on the Part I examination by the American Board of Orthopaedic Surgery (ABOS).

The 1960s and 1970s were rich decades for the OITE. Henry J. Mankin, MD, spearheaded a 1976 study on “The Effect of Native Language on Performance on the Orthopaedic In-training Examination,” which was published in The Journal of Bone and Joint Surgery. Dr. Mankin’s study became a major impetus behind the development of score reports with separate cohorts, including one comprising those who took the exam outside of the United States.

For years, there was no policy on how the exam should be administered. During my tenure as chair of the Evaluation Committee, an examination testing environment and scoring policy was established, the window during which the exam itself could be administered was tightened, and a signed certification that the OITE itself was proctored and appropriately administered was instituted.

Leaps forward in testing
Until recently, in-house support for the examination by the AAOS was mostly administrative and did not include either oversight of the psychometrics of test questions or scoring of the examination. Today, the AAOS has internal expertise in educational theory and test writing, under the direction of Ms. Laura Hruska, examinations manager. The scoring of the examination, too, has also been brought in-house. In 2009, the OITE went from paper to electronic disk format, largely the result of the leadership by J. Lawrence “Larry” Marsh, MD, and Ms. Hruska.

The OITE took another quantum leap forward in 2012, when it moved to a web-based format. The shift pushed the Academy’s Information Technology support unit to the limit, prompting the development of special processes. The data that are captured from this web-based format is so detailed that we can even tell how much time is spent per item.

Technology has also shifted the way in which the OITE is developed, improving processes such as blueprinting, writing, test item peer reviewing (formally called the field test), panel reviewing (formally called the field test meeting), item analysis (new every year), and scoring (new every year).

Originally, question writing and the job of reviewing test questions was largely performed individually and en masse during 3-day meetings in the basement of the AAOS headquarters building. Just about everything, even editing, was done by hand. Referencing was done the old-fashioned way (by actually looking facts up in textbooks or other sources). The tediousness of this cannot be overestimated.

For years, committee members received a 500-page paper book to review test items; now committee members review them online. The panel review has also radically changed—from long weekends in Rosemont to 2- to 3-hour web conferences. To respect the practice and family commitments of Evaluation Committee members, most travel has been eliminated. This change has introduced individuals to a new discipline of premeeting preparation and question review and has ensured protected time for web-conferences.

Unique to any other in-training examination, the AAOS OITE is created de novo every year. This enables residents to keep their tests as study aids during the year.

The greatest lessons
Question writing taught me an appreciation for educational theory and also kept me on the lookout for testable facts—those that not only are quantitative, but also can be used to discern various levels of understanding. It gave me a keen eye for interesting cases in my own practice that could be made into problem-solving, or Tax 3, questions.

An important byproduct of question writing is the opportunity to learn how to work in a committee format and to do critical peer review face-to-face. Another advantage is the interaction with the wonderful Academy infrastructure and staff. Many of the staff members I worked with—including Mr. Howard Mevis, director of the electronic media, evaluations programs, course operations, and program management; Ms. Julie Minnice, examinations manager; Ms. Anna Scheer, senior examinations coordinator; and Ms. Marcie Lampert, senior examinations coordinator—have been at the AAOS for years and “trained” many a question writer.

The OITE and the Evaluation Committee are treasured assets of the AAOS’ educational arena. “Writing multiple choice questions is a core educational and academic activity in orthopaedic surgery,” said Dr. Marsh. “Those who dedicate themselves to the question-writing process will often find future opportunities as leaders of AAOS subcommittees or question writers for the ABOS.”

“Serving on the Evaluation Committee is hard work, but the educational products are important tools for orthopaedic residents and practicing orthopaedic surgeons,” agreed Dr. Albanese. “The test question writing skills that are acquired on this committee are essential for those orthopaedic surgeons with career goals that include leadership in orthopaedic education and Maintenance of Certification examinations.”

Finally, in the words of Frederick M. Azar, MD, AAOS first vice-president, “The Evaluation Committee is an excellent place to start volunteering for the AAOS. The work is quite gratifying, and it is a great venue to interact with fellow colleagues and our superb AAOS staff. I would recommend it to anyone who is interested in orthopaedic education.”

Craig S. Roberts, MD, MBA, recently completed his term as chair of the AAOS Central Evaluation Committee. He can be reached at craig.roberts@louisville.edu.