Writing multiple-choice questions: stems, distractors and Bloom's levels

A multiple-choice question is quick to mark and hard to write well. A weak one lets pupils guess or follow clues, so a right answer tells you little. This article shows how to write the stem, how to build wrong answers from real misconceptions and how many options to use. It gives before-and-after examples from several subjects and one question for each Bloom level that multiple choice can reach.

Put the whole question in the stem

A good stem can be answered before the pupil reads the options. Rush, Rankin and White (2016) summarise the advice of item-writing experts: 'students should be able to formulate a projected correct response, based on the stem alone'. To test a stem, cover the options and ask whether a pupil who knows the topic could answer.

Before (Year 9 science): 'Which of the following is true?', followed by four statements on different plant topics. The stem sets no problem, so pupils must judge every statement before they can start. Rush and colleagues list unfocused questions like this among the flaws that make items harder than intended.

After: 'Which gas do plants take in for photosynthesis? A: oxygen. B: carbon dioxide. C: nitrogen.' A pupil who picks oxygen has the gases the wrong way round, which tells you what to reteach.

Ask for the right answer, not the wrong one. A stem with NOT or EXCEPT tests whether pupils can spot a false statement. Rush and colleagues note that 'learning objectives are more effectively assessed when students identify a correct answer rather than an incorrect answer'.

Before (Year 9 history): 'Which of these was NOT a long-term cause of the First World War?' After: 'Which of these was a long-term cause of the First World War? A: the alliance system. B: the assassination of Franz Ferdinand. C: the Treaty of Versailles.' B catches pupils who confuse the trigger with a long-term cause; C catches a chronology error.

Build wrong answers from real misconceptions

Distractors do most of the diagnostic work. Dylan Wiliam states the aim plainly: 'to have students with the right thinking and students with the wrong thinking give different answers'. He suggests starting 'from the partial or incomplete understandings that students have'.

Before (Year 8 maths): 'Work out 0.3 × 0.2. A: 0.06. B: 6. C: 600. D: 60.' Options B, C and D are unlikely to tempt anyone, so the question shows little more than who can multiply 3 by 2.

After: 'Work out 0.3 × 0.2. A: 0.06. B: 0.6. C: 0.5.' Option B catches pupils who miscount the decimal places, and C catches pupils who add instead of multiplying. Each wrong answer now points to an error you can fix.

Implausible distractors also weaken a question as a measure. Rush and colleagues analysed 1,925 exam questions written for third-year veterinary students. Items with implausible distractors were answered correctly about 95% of the time and did little to separate prepared from unprepared students. That was a university exam, but the logic carries over to Year 8: an option nobody picks does no work.

Plausible distractors help learning too. In two experiments, Little and colleagues (2012) found they can make students retrieve why each option is right or wrong.

Questions built this way are diagnostic: each wrong answer points to a specific misconception. Science education has used them for decades. Treagust (1988) described how to build diagnostic tests from misconceptions research, with examples on covalent bonding and on photosynthesis and respiration in plants. Class results made misconceptions easy to spot and address.

Use three options and leave out 'all of the above'

More options do not make a better question. Rodriguez (2005) pooled 80 years of research and concluded that three options are optimal. More three-option questions fit into the same testing time and cover more content, 'without detrimental effects on psychometric quality of test scores'. In Rush and colleagues' data, five-option items were no harder, and no better at separating students, than four-option items.

So write the key and the two most common wrong answers. If no second wrong answer comes to mind, set a short written question instead.

'All of the above' hands out clues. As Rush and colleagues explain, to choose it a pupil 'need only identify two correct answers'. To rule it out, they 'need only identify one false statement'. 'None of the above' has another problem: when it is the key, you only learn that the pupil rejected the options.

For a fuller list, Haladyna, Downing and Rodriguez (2002) drew 31 guidelines for classroom multiple-choice questions from 27 testing textbooks and 27 research studies and reviews.

Match each question to a level of Bloom's revised taxonomy

The revised taxonomy, edited by Anderson and Krathwohl in 2001, sorts thinking into six categories: Remember, Understand, Apply, Analyse, Evaluate and Create. Krathwohl's 2002 overview notes that the revision relaxed 'the requirement of a strict hierarchy', so the levels overlap. Use them to check that a quiz is not all recall. Butler (2018) found that assessment and learning research agree here: target 'specific cognitive processes that correspond to learning objectives'.

• Remember, retrieving knowledge from memory. Year 7 geography: 'What is the capital city of Australia? A: Sydney. B: Canberra. C: Melbourne.' Pupils who assume the biggest city is the capital choose A.
• Understand, working out meaning, which includes classifying. Year 8 science: 'Which of these is a chemical change? A: salt dissolving in water. B: a nail rusting. C: ice melting.' Pupils who think dissolving makes a new substance choose A.
• Apply, carrying out a procedure in a given situation. Year 8 French: 'Hier, il ___ au cinéma. A: est allé. B: a allé. C: va.' B catches pupils who use avoir with every verb in the past; C catches the present tense.
• Analyse, breaking material into parts and seeing how they relate. Year 8 maths: 'A bus carries 32 passengers 18 km at an average speed of 36 km/h. Which fact do you not need to work out the journey time? A: the number of passengers. B: the distance. C: the speed.'
• Evaluate, judging against criteria. Year 10 computing: 'A program should accept ages from 11 to 16. Which test data best checks the boundaries? A: 12, 13, 14. B: 10, 11, 16, 17. C: 11, 16.' The key is B; C tests only one side of each boundary.

Create means making something new, such as a plan or an argument. Multiple choice cannot reach it: choosing the best plan from three is Evaluate. For Create, set a short written or practical task.

Use a hinge question to decide what happens next

Wiliam calls a question that decides your next step a hinge question. It sits at the point in a lesson where pupils must grasp one idea before you move on. Every pupil answers at once, with fingers, ABCD cards or mini whiteboards. He advises that pupils should need 'no more than two minutes to respond'. You should be able to 'collect and interpret all the student responses in 30 seconds or less'.

Example (Year 7 English). Which sentence is punctuated correctly? A: The dog wagged it's tail. B: The dog wagged its tail. C: The dog wagged its' tail. Pupils who choose A think the apostrophe shows possession here, when it's always means it is or it has.

If most pupils choose B, move on. If few do, teach the idea again, starting from the error the answers revealed. If the class is split, have pupils explain their choice to a partner, then ask again.

Good hinge questions take time. Wiliam reports that groups of teachers 'typically take more than an hour to design one good question'. Write them once as a department and reuse them next year.

Try it on Monday

1. Take a quiz you already use. Cover the options and rewrite any stem a pupil could not answer on its own.
2. Turn every NOT or EXCEPT stem into a question that asks for the right answer.
3. Look through last week's books and note the two most common wrong answers for each question. Use them as your distractors.
4. Cut each question to three options. Remove 'all of the above' and any option no pupil would choose.
5. Label each question with its Bloom level. If every label says Remember, rewrite two questions at Apply or Analyse.
6. Plan one hinge question for the middle of your next lesson. Decide in advance what you will do if most, some or few pupils get it right.
7. Go through the answers straight away, including why each wrong option is wrong. In a laboratory study, feedback cut how often students later repeated a wrong option (Butler and Roediger, 2008).