Test Development, Scoring and Reporting Processes

Certification and Article-Based Continuing Certification (ABCC) examinations

• Committee members develop a content outline that specifies both the topics to be covered and the number (percent) of items devoted to each topic.
• The outline is posted on the ABPN website for examinees to review.
• The outline is periodically reviewed and updated.
• Committee members ensure that test items are coded properly when outlines are updated.

Article-Based Continuing Certification (ABCC) examinations

• Committee members are given an assignment to locate a determined number of articles within specific topic categories.
• Articles are uploaded to our data storage software along with a sheet of information pertaining to the article.
• ABPN staff review the submissions, input the data into our database, and organize the materials for easy access for the committee members.
• Committee members review the articles of their partner in each topic to provide an alternate opinion on the article’s utility in our program.
• Committee members meet to discuss the articles and choose the selections that will be the focus of question writing.
• ABPN staff takes notes during the meeting and records which are articles that have been chosen for usage, which are to be reviewed again at a future meeting, and which will not be in contention for use.

Certification examinations

• ABPN test development staff perform bank analysis and identify areas where more items are needed.
• Committee members receive assignments and submit 20 items per year per committee to ABPN.
• ABPN editors edit items and return them to authors for revision if needed.
• Committee members review a section of the draft items ahead of the meeting.
• Committee members meet to review items.
• Staff facilitate the meeting and track the committee’s recommended changes.
• After the meeting, staff make all item revisions. Final versions of the items are saved in the item bank.
• Staff compile feedback on any items that were flagged as rewrite or delete. These items are sent back to the original writer with feedback provided.

Article-Based Continuing Certification (ABCC) examinations

• Committee members receive assignments and submit 16 items per year per committee to ABPN.
• ABPN editors edit items and return them to authors for revision if needed.
• Each committee member assigned a topic and given a task to review all submitted items in that topic. They review the questions blindly (no keys) and use the articles, to simulate the experience of a test taker.
• Committee members meet to review items. During the meeting, the top five questions per article are chosen, which progress to the live exam.
• Staff facilitate the meeting and track the committee’s recommended changes.
• After the meeting, staff make all item revisions. Final versions of the items are saved in the item bank.
• Staff compile feedback on any items that were flagged as rewrite or delete.

The items on ABPN’s article-based continuing certification (ABCC) examinations are carefully designed to measure meaningful and plausible testing points (e.g., diagnosis, management, etc.), without the influence of assumptions, bias, or stereotypes. When examinees select the correct (keyed) response, they are given credit because they are demonstrating what the examination item is designed to measure. ABPN examination committees encourage thoughtful consideration of patient characteristics, while at the same time striving to promote diversity and present patients who reflect the populations served by the examinees.

Many test items in this examination contain descriptions of patients. Characteristics of a patient such as age, sex, gender identity, race, ethnicity, sexual orientation, disability, socioeconomic status, native language, country of origin, and/or occupation are sometimes mentioned within case vignettes in test items. Some patient characteristics may be important inputs into the diagnostic reasoning process. Inclusion of some characteristics may increase the clinical verisimilitude of the patient cases. Their inclusion, however, as in actual clinical practice, may lead to incorrect conclusions and misdiagnoses. Among the latter are characteristics that could potentially be associated with harmful patient stereotypes.

In the context of item creation, race is considered a social construct not linked to biology or susceptibility to disease. This is similarly true of ethnicity and culture, heritage, or even country of origin. Ancestry, if known, may be biologically important, and thus may be relevant to factors relating to health and disease. In addition, when and if these characteristics are included in items, they should be considered based on patient self-report, not the assumption of the physician.

Based on the thinking above, some of the question stems can be brief and if the reference to a patient is general, patient characteristics are likely omitted unless directly relevant to the question at hand. On the other hand, if the question stem includes a portrayal of a specific patient (including vignettes for linked item sets), it is reasonable to at least include the patient’s age and sex.

Additional patient characteristics may be included for any of several reasons, including if they:

• are clinically relevant or could aid in distractor quality.
• are necessary for the examinee to better understand the context in which the patient is being seen (i.e., the item would be unreasonably difficult if excluded).
• add to the overall exam-level representativeness of the referenced patient population.
• increase the probability of detection, diagnosis, or recognition of an otherwise rare condition.
• do not contain negative stereotypes.

Certification examinations

• Staff perform resource bank analysis and identify where media needs are.
• Committee members receive script writing assignments that are used to create videos for the exam.
• Committee members review, revise, and finalize scripts.
• Staff work with a video production vendor to create the videos.
• The ABPN media editor processes all new media for item writing and potential exam inclusion.

Certification examinations

• Staff prepare draft exam, including pretest items and alternate items.
• Staff create a PDF of the draft exam, pretest, and alternate items to send to the committee ahead of the meeting.
• Committee members review a section of the draft exam ahead of the meeting.
• Committee members review the draft and select final items.
• Staff publish and proof final exam.
• Committee chair reviews and approves final exam.

Certification examinations

• All certification and continuing certification exams are administered under secure, standardized test conditions.
• Exam data are delivered electronically to the ABPN.

Article-Based Continuing Certification (ABCC) examinations

• Committee chairs review all finalized questions to determine if there are any changes.
• ABPN staff upload questions to our online exam server. Editing staff proof all items a final time before setting them to live status.
• Questions are activated as live on January 15 of a given year.

Certification examinations

• Psychometric staff conduct preliminary scoring to ensure that all data were transmitted accurately.
• Psychometric staff review the scoring results to verify that each question on the examination is psychometrically valid.
• Test development staff review all examinee comments to identify problematic items.
• Based on these analyses, items that do not meet certain statistical criteria and/or have generated notable examinee comments are referred to the committee chairs for review.
• Committee chairs review problematic items and come to an agreement on how to move forward.
• Items that are deemed to be flawed by the committee chairs are removed from scoring, and the exam is rescored.
• This phase typically takes 4-6 weeks, depending on the number of exam items and the number of exams being scored at the same time.

Article-Based Continuing Certification (ABCC) examinations

• ABPN staff review statistics and all diplomate feedback to track item performance and determine if any questions may have flaws.
• If a question is deemed to have possible flaws, the committee chairs will review the item and determine if the question should be taken offline for editing.
• If the item is deemed flawed, ABPN staff rescore the item, meaning the item will not count toward the diplomate’s score on the exam.

Certification examinations

• The ABPN utilizes a criterion-referenced approach to standard setting. That is, there are no preset pass/fail rates for any group of examinees (referred to as norm-referenced approaches). It is always possible for all examinees to pass an examination.
• Periodically (typically every 7-8 years) a standard setting study is conducted by the committee chairs and other subject matter experts who are not current test development committee members. During a 2-day meeting, they review all of the items on the examination and use well-established, evidence-based procedures to set the passing standard for the examination.
• After the initial standard has been set, in subsequent years statistical procedures are used to equate the pass/fail standard across different versions of the same test. This removes any advantage associated with taking a test with easier items or disadvantage associated with taking a test with harder items.
• This phase typically takes 2-4 weeks, depending on whether a standard setting study is conducted or whether equating is used.

Certification examinations

• Score reports are generated, and several quality control procedures are performed to verify the accuracy of the scores.
• Score reports are posted electronically in the Physician Portal (specialty certification examinations) or mailed to the delegate.
• This phase typically takes 2 weeks, depending on the number of examinees and the number of tests administered at the same time.

Certification and Article-Based Continuing Certification (ABCC) examinations

• Diplomates apply for committee membership via online application.
• Committee chairs review candidate applications and accept potential members into the pool of accepted candidates.
• Staff identify committee membership needs.
• Committee chairs choose potential members from the candidate pool to fill committee vacancies.
• Staff train new members on ABPN item writing style and requirements.