Imagine an assessment committee reviewing a medical examination. The results are ready, and the discussion turns to which students should progress. Before reaching that decision, the committee needs to connect what the examination asked students to do with what their scores are being taken to mean.
Extended matching questions, or EMQs, organise several questions around a shared list of options. In a clinical set, different patient scenarios may present the same main complaint, with candidates choosing the most likely diagnosis for each. Carefully developed cases can require learners to interpret findings and distinguish between plausible explanations.
A systematic review of 19 studies examined the reporting of EMQ validity evidence. Reliability coefficients appeared in 14 studies, while item difficulty was reported in four and discrimination in five. None covered all nine indicators used by the reviewers. These findings concern published reporting; they cannot establish that unreported procedures never occurred. The review also included only English-language publications.
Validity concerns the evidence supporting a particular interpretation of assessment results. Reliability contributes information about score consistency within that argument. Downing describes five sources from which educators can seek supporting evidence:
- Content: coverage of the intended curriculum and level of thinking.
- Response process: sound administration and accurate handling of responses.
- Internal structure: item performance and consistency of scores.
- Relationships with other variables: expected links with relevant measures.
- Consequences: intended and unintended effects of decisions based on scores.
For an EMQ set, these ideas can be translated into concrete review work. Suppose a team wants to assess how students interpret causes of breathlessness. It could first specify the decision expected at the students’ stage of training. Reviewers could then examine whether each case provides enough information to support the keyed answer and whether the alternatives represent credible competing explanations.
NBME recommends clear lead-ins, plausible options of a consistent type, and removal of distracting wording or clues. For a diagnosis task, reviewers could check that every option is a diagnosis and that the clinical question is clear before the options are read.
The review meeting could also leave a useful record. The team could document the disagreement it resolved: perhaps the clinical information supported two answers, or a phrase revealed the answer without requiring interpretation. Keeping the original item, the revised version and the reason for the change would make the decision traceable.
After the examination, difficulty shows the proportion answering correctly, while discrimination describes the relationship between item performance and overall test performance. These statistics can prompt review of the key and wording. If stronger candidates repeatedly choose another option, reviewers could investigate whether it is also defensible.
A practical assessment record could bring together the intended use of the scores, the content plan, item-review decisions, scoring checks and unresolved concerns. When the committee makes a progression decision, that record would allow it to explain the evidence considered and the limits of its interpretation.
Read the full systematic review on validity evidence for extended matching questions.