The Orthopaedic In-Training Examination (OITE) is a formative—and formidable—assessment tool. It not only measures a resident’s knowledge as compared to his or her peers, it also enables residency programs and program directors to compare their programs with other residency programs, with results stratified by content domain and training year.

Because the OITE is an educational and assessment tool, not a standardized certifying examination, it should not be used in any of the following ways:

• as a single criterion for promoting a resident to the next year of training
• as a single method of determining whether a resident should be retained in a training program
• as a means of obtaining admission into an orthopaedic surgery residency program
• as part of the application process for a fellowship or any other position

Each year, a new OITE is constructed. The 33 members of the Central Evaluation Committee, with occasional assistance from specialty evaluation committees, are responsible for developing the test. Here’s how the system works.

Developing a blueprint
The first part of test construction involves “blueprinting” the examination, which involves establishing the content domains and subtopic mappings. The 275-item test is broken into 11 content domains (Table 1). Some areas overlap, and the number of test items in the final test will vary slightly. In addition, miscellaneous questions fall within the domains of Professionalism, Communication, Practice-Based Learning, and Systems-Based Practice.

Each committee member receives a question assignment. Typically, a committee member writes about 20 questions, so that the total number of questions submitted is about double the number used. This is done because many questions will be eliminated during the peer-review process. Guest writers from other groups such as the Diversity Advisory Board and the Patient Safety Committee also contribute questions.

Writing the questions
Every new committee member and item writer completes an orientation program (test writing workshop) during the AAOS Annual Meeting. The program covers test-item construction, question taxonomy, and performance metrics of individual items. Writers practice spotting problematic question constructions and quickly learn that writing good test items requires significant skill. A checklist helps to avoid poor item constructs such as multiple true/false questions, redundancy, and verboseness.

Questions are categorized by three taxonomies: recall, interpretation, and problem solving. Recall questions require the examinee to have memorized a fact. Interpretation questions ask the test taker to read an imaging study or a histologic slide. Problem solving tests the examinee on his or her ability to integrate facts and apply them to clinical decision-making. The focus is to increase the proportion of questions involving interpretation and clinical problem solving.

The question items are then compiled for the peer-review process. Committee members “take” the examination and their responses and comments are then reviewed by small groups of item writers.

Once all the comments are in, the test is divided into domains. Small working groups review and discuss each item in detail, rating question quality, flagging poor items, and rewriting problematic questions if possible. Problem-solving questions that cover essential content within the fundamentals of orthopaedic surgery and musculoskeletal care get priority.

Questions are chosen to match the blueprint matrix and mapping by content domain. Items selected for the final examination undergo a medical edit to ensure quality item writing and appropriateness for each answer choice. After the medical edit, the committee chair conducts a final review and the OITE is programmed and tested for web-based delivery. Each examinee receives questions in a random order.

Preparing for the future
The OITE tests the knowledge that residents have acquired, enabling them to chart a course of study for the next residency year. Each test site is required to proctor the examination to simulate the “high-stakes” experience residents will have when they complete Part 1 of the American Board of Orthopaedic Surgery examination.

In 2013, the OITE test day is Nov. 9. Approximately 4,800 orthopaedic residents in more than 200 programs in the United States and 15 other countries will take the OITE. There’s no better way that they can prepare for their futures.

Montri D. Wongworawat, MD, chairs the Central Evaluation Committee. He can be reached at dwongworawat@llu.edu