A job analysis is a documented study of what a job actually requires: the important work behaviours, the tasks that make them up, and the knowledge, skills and abilities needed to perform them on day one. It is the evidence that justifies what your assessment measures. Without one, a selection procedure has no defensible content.
Most hiring teams have never run one. They have a job description — a recruiting advertisement — and a job grade, a pay decision. Neither tells you what to measure.
The requirement list you are hiring against is probably stale
Lightcast's report The Speed of Skill Change (January 2025) found the skills requested for a given job changed by 32% on average between 2021 and 2024; for a quarter of jobs, 75% of skills changed. An earlier Lightcast analysis of about 15 million postings across 680 occupations put turnover at 37% of the top 20 skills over five years.
That churn collides with a habit: requirements get copied forward from last year's posting, increasingly with a drafting assistant smoothing the prose. The list grows longer and more fluent without anyone re-checking whether the work still looks like that.
Three things that are frequently confused:
- Job description — an outward-facing summary written to attract applicants. No evidentiary weight.
- Job evaluation — a scheme for sizing a role against others to set grades and pay. Answers "how big is this job?", not "what does it require?".
- Job analysis — a documented study of tasks and KSAOs, rated by people who know the work. The only one of the three that supports a claim about what an assessment measures.
What the written standards actually ask for
The US Uniform Guidelines on Employee Selection Procedures are the most-cited written standard here, and employers outside the US often adopt them as a template because nothing more detailed exists. Section 1607.14(C)(2) states plainly: "There should be a job analysis which includes an analysis of the important work behavior(s)", and the behaviours chosen for measurement "should be critical work behavior(s) and/or important".
Two requirements trip people up:
- Representativeness. The user must show "the behavior(s) demonstrated in the selection procedure are a representative sample of the behavior(s) of the job" (1607.14(C)(4)).
- Necessity at entry. Where you measure a knowledge, skill or ability rather than a work sample, you must show the attribute "is used in and is a necessary prerequisite to performance of critical or important work behavior(s)". The same section rules out content-validity arguments for traits and for skills learned on the job.
Section 1607.15 sets out what the written record must contain; the EEOC's guidance on tests and selection procedures is a shorter entry point. SIOP's Principles for the Validation and Use of Personnel Selection Procedures (2018) is broader but points the same way.
Five design decisions that decide whether it holds up
1. Write tasks, not abilities
This is the highest-leverage choice, and there is direct experimental evidence for it. Morgeson and colleagues surveyed 431 clerical incumbents, randomly assigned to task-phrased or ability-phrased versions of the same content (Journal of Applied Psychology, 2004). Ability statements were endorsed more often in 11 of 12 competency areas, averaging 0.80 extra endorsements per area (d = .52).
The decisive result came from 14 bogus items planted in the survey. Summed endorsement of bogus abilities averaged 2.04 against 1.00 for bogus tasks (d = .55). People affirm requirements that do not exist when those requirements are phrased as abilities. A check of 48 supervisors and analysts found no corresponding job differences (average d = .06), so the gap was self-presentation, not reality.
The authors also warn that looking at mean ratings instead of sums hides the inflation, which "can lead to a false sense of security about the accuracy of resultant job analysis data." Report both.
2. Rate importance, and treat difficulty ratings with suspicion
Dierdorff and Wilson's meta-analysis of 46 studies and 299 reliability estimates (Journal of Applied Psychology, 2003) found that which scale you use changes how much you can trust the answer. Uncorrected mean interrater reliabilities for task-level ratings:
- Importance — .771
- Frequency — .699
- Time spent — .665
- Difficulty — .632
Specificity matters too: task-level data reached .77 against .61 for generalised work activities, and .587 for the Position Analysis Questionnaire. Concrete statements about this job beat abstract statements about work in general.
3. Match the rater to the descriptor
From the same meta-analysis, for task-level data:
- Incumbents — .773 (k = 100)
- Trained job analysts — .631 (k = 10)
- Technical experts — .466 (k = 9)
Incumbents are the right source for what the job involves; once ratings are corrected for panel size, analysts come out ahead. Trained facilitation of an incumbent panel beats either alone.
4. Get enough raters before you get more items
Reliability is a function of panel size as much as instrument quality: corrected to 25 raters and 300 items, task interrater reliability sits at .727. California's state HR policy asks for four to ten SMEs for examination development, all past probation and drawn from strong performers, first-level supervisors or hiring managers. OPM names no number, saying only that "it's always advisable to gather as many SMEs as possible", because more raters capture more of the job and more views on criticality.
5. Write the linkage down
The link between each KSAO and the tasks it serves is the part auditors ask for and almost nobody has. Have SMEs rate how necessary each KSAO is for each important task, then record, per KSAO: the tasks it links to, the mean linkage rating, whether it is required at entry or learned on the job, and the method you will use to measure it. A KSAO with no strong task link does not enter the assessment, however much the hiring manager likes it.
A realistic one-week schedule
- Day 1 — Scope. Name the job, the population it covers, and the decision the assessment supports. Pull a starting task list from O*NET Online and your own documents. Recruit six to ten SMEs.
- Day 2 — Draft. Convert everything into task statements in the form "action + object + purpose". Cut anything you cannot observe. Plant a handful of bogus items.
- Day 3 — Rate. Survey SMEs on importance and frequency, task by task. Keep it under 45 minutes or completion collapses.
- Day 4 — Analyse. Compute means and sums, check agreement, inspect bogus-item endorsement, and drop tasks failing the importance threshold you set in advance.
- Day 5 — Link and specify. Run the KSAO linkage, then write the assessment plan: one method per surviving KSAO. Task statements convert directly into AI-graded scenario prompts and work-sample briefs — the form AssessAll's custom assessment work takes. The analysis is what makes a scenario defensible rather than merely realistic.
The sign-off checklist
- Tasks are phrased as observable behaviours, not abilities or traits.
- Bogus items were included and endorsement was checked.
- Both mean and summed ratings are reported.
- Importance ratings carry the weight; difficulty ratings do not drive decisions alone.
- Six or more SMEs rated, and their roles and tenure are documented.
- Every assessed KSAO has a documented task linkage.
- Entry-level requirements are separated from learn-on-the-job requirements.
- The report states its date and a review interval — 18 to 24 months, given 32% skill churn.
- The assessment blueprint maps section by section to retained tasks.
When a lighter approach is the right call
A full task-level analysis is not always proportionate. For a brand-new role with two people who understand it you have no panel; document the limitation and revisit in six months. Where a validated instrument already exists for a well-defined job family, 1607.7 permits reliance on other studies once the jobs are shown to be comparable — a job comparison, not a full analysis. Competency models stay useful for development conversations, where a mis-specified requirement costs far less than in selection. The caution is only not to let a development framework become a hiring filter without task evidence behind it.
The sequence is unglamorous: find out what the work is, write it down as behaviour, have the people who do it rate what counts, and only then decide what to measure. An assessment built in that order can be explained to a rejected candidate, a regulator and a hiring manager with the same document.