What refers to here as effective question design is mostly effective multiple choice exam questions. A multiple choice question consists of three parts: a stem which is the base of the question, distractors, which are incorrect answers and the key, which is the correct answer. This is the terminology that is used throughout this page.
When designing effective, reliable questions it is interesting to consider the test from the student point of view. When students don't know the correct answer, they are likely to guess it, especially if incorrect answers do not incur negative points. Below are some 'guessing' strategies that students might use for multiple choice testing and some ideas for you how to make it difficult for the students to use those strategies through question design.
- Student rule 1: Pick the longest answer.
The longest answer is the one that stands out hence students are naturally drawn to it as a possible correct answer. When designing your questions make the questions equal length if possible and/ or don't make it a pattern in every question that the longest answer is the correct one.
- When in doubt pick 'c'.
Students might look for patterns in the answers. Make sure that correct answer choice letters are random.
- Avoid picking answers with 'always' and 'never' in it.
These are considered to be too definite hence students are more likely to go for a more tentative 'usually'. When designing your questions make sure this option is correct part of the time or avoid using always and never in the option choices.
- When you really don't know just guess.
All of the examples above illustrate an element of guessing, however, with a strategy in mind. When there are no rules to follow students will just choose whatever. To avoid that option use five alternatives instead of three or four to reduce guessing.
- Pick the scientific-sounding answer.
Students might be drawn towards an answer that sounds complicated, scientific and therefore 'correct'. To avoid this use scientific jargon throughout.
- Pick a word which you remember was related to the topic.
In other words, pick a word that rings a bell. When creating the distractors use terminology from the same area of the text as the right answer, but in distractors use those words incorrectly so the wrong answers are definitely wrong.
- Don't pick an answer which is too simple or obvious.
Students tend to suspicious of the obvious answers and tend to dismiss them as correct. Sometimes make the simple, obvious answer the correct one.
Testing higher order thinking skills
When designing test questions it is important to move away from understanding and comprehension questions towards higher order thinking skills such as analysing, evaluating and synthesising:
- Evaluation questions require the student to make a judgement. For example in medicine, a question might present the student with detailed information about a patient and requires them to make a diagnosis.
- Analysis and Synthesis can be assessed by presenting students with data, diagrams, images, multimedia etc that require analysis before a question can be answered. A Synthesis question might require students to compare two or more pieces of information. Questions in these categories could, for example, involve students in recognising unstated assumptions or separating useful information from irrelevant information.