On this page
1. Introduction
2. Guidelines for Submitting Items
3. Writing an Effective Key Concept
4. Writing an Effective Stem
5. Writing A-Type Items
6. Avoiding Common Question Flaws
7. Writing Stems and Lead-ins
8. Conclusion
1. Introduction
The preparation of good examination questions is much more difficult than most critics think. The natural tendency of new question writers is to prepare questions that are extremely difficult and which test small bits of obscure, rare information. Questions of this nature perform very poorly. Likewise, questions that are too easy perform poorly because they don’t discriminate between test takers. This guide has been designed to help question writers prepare better questions and incorporate question theory as well as validity in question writing.
Please review this guide carefully prior to writing your question as it contains important information that should be taken into consideration during the question writing process.
2. Guidelines for Submitting Items
1. Review this entire booklet before beginning your assignment.
2. The items you are writing and reviewing are secure materials and should not be discussed with, or shown to, others. Please keep your items in a secure place, such as a locked file cabinet, when you are not working with them.
3. Writing an Effective Key Concept
The anchor to writing an acceptable question is having an effective key concept. The key concept is a statement of fact upon which the stem and distractors are based. The key concept should have a subject (i.e. the clinical problem) and action (i.e. the treatment). Key concepts are those that experts in the field agree should be common knowledge for NBMTM-certified individuals. The key concept should be concise and definitive.
4. Writing an Effective Stem
As a question writer for the National Board of Medication Therapy Management, your goal should be to provide the highest-quality test material possible to ensure that all examinees that become certified are knowledgeable in our field, not just test-wise. Well-written and appropriately edited examination material is critical for all examinees to demonstrate the depth and breadth of their knowledge.
Writing high-quality, effective test questions is difficult. In fact, many test questions we have been exposed to are NOT well written. This can include questions that appear on elementary school tests, as well as on recent continuing education examinations. When basic question writing principles are not used, test questions may be confusing or otherwise unclear, verbose, esoteric, or not as effective as they could be. These instructions can help you avoid these problems when writing questions for The National Board of Medication Therapy Management.
This Question Writing Guide provides instruction in effective question writing. It gives samples of questions that are well constructed as well as those that are not. Because the Examination Committee does not accept most flawed questions, the time you spend writing them is wasted. Therefore, the extra effort you put toward well-constructed questions not only expedites the review process, but also improves the quality of the examination and helps ensure the qualifications of the candidates who pass.
Application of Knowledge
What is application of knowledge?
Test questions can be written in two basic ways. By requiring:
(1) Mere recall of facts (1st order)
(2) Application of knowledge to answer the question (2nd order)
A question requiring application of knowledge (2nd order) provide a specific scenario and asks the examinees to apply their knowledge in interpreting this information and recalling appropriate facts to determine the correct answer. In contrast, recall questions (1st order) simply require knowledge of isolated facts to determine the correct answer. While a written exam will certainly include questions that require recall of facts, it also requires application of knowledge (2nd order questions).
How do you write a 2nd order question?
One of the most common approaches to writing this type of question involves composing a clinical vignette. The vignette should provide some or all of the following information: initial symptoms, underlying conditions, other appropriate clinical findings, and appropriate laboratory studies. The lead-in for this type of question would ask a question pertaining to diagnosis, management, related findings, or other information. This type of question requires an examinee to apply his or her knowledge, as would be done in a clinical situation. The largest flaw with a second order question is including information that is not required to answer the question. Information should be included if it is important to answer the question or provides clarifying information required to answer the question.
Questions that test application of knowledge should be structured to ask for a specific piece of information and are driven by the key concept. The STEM can be structured to ask for:
– Associated clinical findings
– Optimal treatment
– Implications of a given clinical decision
– Complications of a disease process or treatment
5. Writing A-Type Items
An A-type question utilizes the multiple-choice format, which involves a single best answer. Four answer choices should be provided (one correct answer and three distractors), with there being only one best answer as agreed upon amongst experts in the field.. While the distractors may be partially true, the single best answer must be the most appropriate response. Options should be lettered A, B, C, and D.
Basic Rules
- The key concept drives the STEM and options. Key concepts should involve situations that would be encountered in a typical practice. Avoid trivial, tricky, or unnecessarily complex key concepts.
- Focus the STEM on testing application of knowledge, not an isolated fact. Using a clinical vignette provides a good background for testing application of knowledge.
- Write a STEM that is focused, and poses a clear question. The examinee should be able to formulate an answer to the question without looking at the options. If the examinee must read every option to gather additional information before being able to answer the question, the item is not well focused and should be rewritten.
- Write options that are homogeneous, with all options falling into the same category as the correct answer (e.g., all diagnoses, treatments, associated conditions, etc.). All distractors should be viable; they should all be grammatically consistent, logically compatible, and approximately the same length. If the answers involve therapeutic options, generally they are listed from least to most invasive.
- Write the stem to contain all the information needed to answer the question. Options should be relatively short and should not contain additional background information, only the completion of the statement or question posed in the stem. The examinees should be able to formulate an answer the question without looking at the options.
- List options in logical or alphabetical order. For example, questions asking for procedures should start with least invasive and work down to most invasive procedures. Diagnoses can be listed alphabetically.
Do NOT write any items asking “Which of the following is true?” or “Each of the following is true EXCEPT” these items are almost always unfocused with options that are not homogeneous and will not be accepted by the board.
If your questions follow all of these rules, they most likely are well phrased and focused on an appropriate subject.
6. Avoiding Common Question Flaws
A number of question flaws can benefit test-wise examinees. People who are not candidates for this examination should not be able to answer the questions correctly on the basis of their construction alone. Typical item flaws include:
– Absolute terms in options, such as always, never, or only
– Subjective and nonspecific terms such as may, might, can, could, common(ly), frequent(ly), usually, sometimes, and rarely; these terms are too vague to be used for questions with one best answer and have different meanings to different people. Adding the term “most” to frequent or likely (e.g., The most likely diagnosis…) actually helps focus and clarify the item.
– Logical clues, or giving a subset of options that allows the examinee to rule out the distractors
– Repetition of words or phrases in the stem and options (this refers to the test taking jargon as “clanging”)
– Making the correct option the longest one. The most detailed option is usually the correct answer.
– Grammatical clues. An option that does not grammatically fit the stem is usually not the correct answer.
– Convergence. The correct answer includes elements that are common to the distractors.
– Unnecessary difficulty. Tricky or complicated stems; long, complicated options – remember options should be of the same length.
– Negative terms in stem or options, which require reverse thinking, should not be used. Use of negative terms in options, especially in association with a negatively phrased question (Each of the following EXCEPT) can be confusing and tricky. In many cases, options with negative terms are not homogeneous.
Use only one term or concept per option. Using “and” or “or” in the stem or some options is either asking more than one question or asking for more than one answer. In options, these terms often clue one option as correct or incorrect. If used in the stem, the question is testing more than one concept.
Avoid instructional information. The examinee should have the background knowledge necessary to answer the item; if the instruction is needed, the item may not be appropriate for the examination.
Do not use multiple true/false items, such as “Each of the following is true EXCEPT” items. In addition to the problems associated with negative items, asking multiple true-false statements is not the best way to test knowledge of examinees. In addition, the negative format requires the candidate to use reverse thinking twice, which makes an item unnecessarily tricky. Options for these items are almost always not focused and not parallel.
Avoid using statistical information or comparisons in options, as neither tests application of knowledge and both usually results in options not being homogeneous.
7. Writing Stems and Lead-ins
When composing questions using clinical vignettes, use realistic scenarios involving situations likely to develop in a typical practice. The stem should identify a patient and any pertinent information needed to answer the question. This information may include some or all of the following:
– Patient age
– Patient gender (use man, woman, boy, girl, infant) – and only use gender if it is pertinent to formulating the answer
– Presenting symptom(s) or underlying condition and duration
– Findings on physical examination
– Findings on laboratory evaluation
– Results of other diagnostic testing
– Issues arising during surgery
The stem should not include unnecessary information, however, a question may be more readable if it includes patient information, such as “A 45-year-old patient” even if age and gender are not important to the question.
The stem should not include information about culture, race, socioeconomics, or sexual preferences unless this information is absolutely necessary to answer the question.
When composing questions that do not contain vignettes, keep all information relevant. Some items may be short and not contain a vignette but still test application of knowledge. Some vignettes may be quite brief, while others may be more detailed. Occasionally, recall of information is important to test, as in anatomy questions.
The language used in the stem and options should be clear and concise. Avoid jargon and inflated diction. Test items should be written with the purpose of effectively measuring the examinees understanding and ability to apply principles.
If you are including an image to be assessed as part of your question, there is an opportunity to upload it as an attachment to your question in the database. It is imperative that you submit images of high resolution. One of the most frequent criticisms by examinees is that the image was unclear. The best way to avoid this is to submit images of high quality ensuring that all identifying patient information that is HIPPA protected is removed. If there is concern with the image during the review process, the author will be contacted.
Commonly used lead-ins are listed below:
Basic Science; Mechanisms
Which of the following is the most likely mechanism of action?
Which of the following is the most likely explanation for these findings?
Which of the following is the most likely additional finding?
Which of the following is the most likely site of the lesion?
Which structure is most likely to be involved?
Laboratory evaluation is most likely to show
This agent acts at the receptor for
Diagnosis
Which of the following is the most likely diagnosis?
These findings are most consistent with
Which of the following is the most likely associated condition?
Which of the following is the most likely site of the lesion?
This patient is at increased risk for (development of)
Management/Therapy
Which of the following is the most appropriate next step?
Which of the following is the treatment of choice?
Which of the following is the most likely outcome of [specify treatment]?
Which of the following is the most likely result of interaction between these drugs?
Which of the following is the most appropriate treatment? [e.g., pharmacotherapy]
Which of the following is the most appropriate management? [e.g., management other than pharmacotherapy or a mix of pharmacologic, surgical, or other types of therapy or observation]
Writing Options
Options consist of the correct answer and three distractors, or incorrect answers. Distractors should be plausible, with none standing out as obviously incorrect. In a well-constructed question, each distractor will be chosen by at least a few examinees. Potential sources of distractors include faulty reasoning and common misconceptions and errors. Distractors should not contain information that could give clues to the correct answer.
Well-written distractors should be:
– Homogeneous with the correct answer; all should be diagnosed, treatment options, laboratory studies or values, etc.
– Plausible to an uninformed examinee
– Incorrect or significantly inferior to the correct answer
– Similar to the correct answer in length and construction
- Ordered in a logical way, e.g. from least to most invasive, alphabetically
- Grammatically consistent with the stem
Well-written distractors should NOT:
– Give a clue to the correct answer
– Use ambiguous or non-specific terms such as never, frequently, almost, etc…
– Use “all of the above” or “none of the above”
– Be mutually exclusive
Distractors affect the difficulty of the item. For example, review the following option sets that accompany the same question:
- Which of the following companies manufactures sildenafil citrate (Viagra)?
- General Mills
- General Motors
- IBM
- Pfizer
- Which of the following companies manufactures sildenafil citrate (Viagra)?
- Eli-Lilly
- Glaxo-Wellcome
- Novartis
- Pfizer
In the first example, the options are quite different, with only one pharmaceutical company listed. Someone who knows very little about this subject could easily answer this correctly.
In the second example, the question becomes more difficult because of the homogeneity of the options. Someone with limited knowledge would find all of the options to be plausible.
8. Conclusion
The National Board of Medication Therapy Management thanks you for taking the time to participate in this very important process. The NBMTM also strongly suggests that you participate in question writing workshops that normally take place during the annual NBMTM meeting. The NBMTM thanks you for your work in writing examination items.
