A structured interview is one in which every candidate is asked the same questions, derived from a job analysis, in the same order, and scored on the same anchored rating scale by raters who score independently before they discuss. Structure is a property of the guide and the scoring rules — not of the interviewer's discipline on the day.
That distinction decides whether the interview is measurement or conversation. And the evidence is unusually specific about where the agreement goes missing: in a meta-analysis of 125 interrater reliability coefficients from a combined sample of 32,428, Huffcutt, Culbertson and Weyhrauch (2013) found panel interviews averaged .74, while interviews conducted separately by different interviewers averaged .44. Same construct, same candidates, half the error — and the only thing that changed was how the interview was administered and scored.
The interview is a selection procedure, and the law treats it as one
Teams routinely run a validated test and then a free-form interview, as though only the first one counts. Legally it does not work that way. The Uniform Guidelines on Employee Selection Procedures apply to any procedure "used as a basis for making employment decisions," and the EEOC's guidance on tests and selection procedures makes no exemption for the conversation at the end of the funnel.
There is direct evidence that structure travels well in court. Williamson and colleagues (1997, Journal of Applied Psychology) coded 17 aspects of interview structure across 130 federal cases — 84 disparate-treatment and 46 disparate-impact claims — and found most features related to verdicts favouring the defendant. The strongest composite was objective, job-related content, followed by standardised administration. The things that help you defend an interview are the same things that make it measure.
Step 1 — Derive the questions from a job analysis, not the job advert
The US Office of Personnel Management is explicit: questions should target competencies identified through job analysis and draw on job-related critical incidents — concrete episodes of notably good or poor performance collected from people who do the work.
Practically, that means three or four competencies, not nine. A 60-minute interview cannot carry nine constructs, and splitting attention across too many is one of the quieter reasons panel scores collapse into a halo.
- Collect 8–12 critical incidents per competency from incumbents and supervisors
- Keep incidents that distinguish strong from weak performance, discard ones everyone handles the same way
- Write one or two questions per competency directly from those incidents
Building that incident base and the scales that come out of it is slow, specialist work; it is the reason custom assessment design exists as a service at all, including at AssessAll.
Step 2 — Pick the question format per competency, not per interview
Two formats dominate, and they are not interchangeable.
Situational questions present a job-like dilemma and ask what the candidate would do. They work well where the behaviour is unlikely to be in the candidate's history — a campus hire, a first-time supervisor — because they do not require prior experience to answer.
Behaviour description questions ask what the candidate actually did in a past situation, on the premise that past behaviour predicts future behaviour. OPM notes that behaviour description questions show higher validity for highly complex jobs, which is the usual finding: as job complexity rises, hypothetical answers converge on what everybody knows they should say, and discrimination between candidates falls.
The operational rule: situational questions for roles where the candidate cannot have done the work yet, behaviour description for roles where they should have. Mixed panels for senior hires often need both, assigned deliberately to different competencies rather than scattered.
Step 3 — Write the anchored scale before you hear a single answer
This is the step most guides skip, and it is the one that produces the reliability. A rating scale with numbers but no behavioural anchors is an opinion form.
For each question, write what a 1, a 3 and a 5 sound like, in the language of the job:
- 5 — names the specific constraint, states the trade-off they accepted, and describes what they did when the first attempt failed
- 3 — describes a reasonable action without the trade-off or the follow-through
- 1 — restates the problem, or describes an action the role does not have authority to take
Anchors written after the interviews have started are rationalisations. Anchors written before are a measurement instrument. If the anchors are hard to write, the competency is probably too abstract to interview for — which is useful information before you have rejected 200 people on it.
Step 4 — Fix the probes, not just the lead questions
Levashina and colleagues' review of the structured interview literature treats probing and follow-up questions as a distinct component of structure, alongside question format, rating scale development and the handling of impression management. Their conclusion is candid: much is known about structured interviews, and much is still open — they close with 12 propositions and 19 research questions.
What is not in dispute is that unrestricted probing reintroduces the variance the guide was built to remove. The highest-structure designs specify the follow-ups in advance. A workable middle position:
- Write two permitted probes per question ("What was your specific contribution?", "What happened next?")
- Allow clarifying probes, prohibit leading or hint-carrying ones
- Cap probing time so one candidate does not get twice the airtime
Step 5 — Decide the administration format on purpose
This is where the .74 against .44 comes in. A panel that hears the same answer at the same time removes a whole source of error — different interviewers eliciting different responses. Huffcutt and colleagues also flagged something less comfortable: highly structured interviews conducted separately showed lower reliability than the structure level alone would predict. Structure on paper does not survive being split across rooms as well as practitioners assume.
- Default to a panel of two or three for decision-grade interviews
- If interviews must be separate, hold the question set and the scale absolutely fixed and expect to average more raters to reach the same precision
- Treat sequential interviews with different question sets as different instruments, not repeated measures of one
Step 6 — Score independently, then reconcile the gaps
OPM describes the standard sequence: each panel member rates responses individually, and the group then discusses significant discrepancies. The order matters. Discussion before independent scoring converts three ratings into one opinion with three signatures, and inflates apparent agreement while removing the information that agreement was supposed to provide.
Where an interview sits behind an earlier scored stage — an AI-graded scenario, a work sample, a proctored test with integrity bands, as in AssessAll's funnel — keep the interview rating independent of that score too. An interviewer who has read the test result is no longer an independent measurement.
Step 7 — Keep what a challenge would ask for
- The job analysis or critical-incident record behind each competency
- The question set, with version and date
- The anchored scale as it stood when the interviews were run
- Individual rater sheets, not just the reconciled score
- Pass rates by group at the interview stage, calculated separately from the rest of the funnel
How much this is worth, honestly
Older meta-analytic work put unstructured interview validity near .20 and the most structured designs near .57 (Huffcutt & Arthur, 1994), and those figures are still widely quoted. The current, more conservative estimate is better to plan against: Sackett and colleagues' 2022 re-analysis in the Journal of Applied Psychology puts the structured interview at a mean operational validity of .42, with an 80% credibility interval of .18 to .66. That interval is the real message. Structure is necessary but not sufficient — a well-built guide for the wrong competencies still lands at the bottom of that range.
The SIOP Principles are clear that validity is a property of an inference for a specified use, not a badge attached to a method. "We use structured interviews" is not a validity claim.
When a looser interview is the right call
Not every conversation should be scored. A realistic-job-preview discussion, a team meet-and-greet, or a candidate's own questions about the role are legitimate parts of hiring and do not need anchors — provided no rating leaves them and no one treats them as evidence. The failure mode is not the unstructured conversation; it is the unstructured conversation that quietly outvotes the scored stages.
Takeaway
The interview guide, not the interviewer, is the instrument — and the two steps that move reliability most are writing the anchors before the first interview and putting the raters in the same room. If you only have time for one change this quarter, write the anchored scales — it is the step that costs the least and does the most for rater agreement.