An AI-collaboration assessment measures whether a person can judge when to use an AI system, verify what it returns, and recognise the tasks where it quietly fails. It does not measure whether they can name models or recite prompt patterns. The measurable construct is decision quality under uncertain machine advice. Tool familiarity decays in months; judgment transfers across tools.
Most "AI skills" tests now in circulation get this backwards. They test the tool. The evidence says the variance that matters sits elsewhere.
The gain is not in the tool, and it is not automatic
The largest systematic review of human-AI collaboration to date covered 106 experiments across 74 papers, yielding 370 effect sizes. Its headline finding is uncomfortable: on average, human-AI combinations performed significantly worse than the better of human alone or AI alone (Hedges' g = -0.23, 95% CI -0.39 to -0.07). Combinations beat unaided humans comfortably (g = 0.64) — but that is a lower bar than most organisations think they are clearing. See Vaccaro, Almaatouq and Malone, *Nature Human Behaviour*, 2024.
The moderators are where the assessment design lives:
- Decision tasks — choosing from a fixed set of options — showed losses (g = -0.27, P = 0.002).
- Creation tasks — open-ended output — showed a positive but non-significant effect (g = 0.19, P = 0.180). The difference between the two task types was itself significant.
- When the human was the stronger party, combinations produced genuine synergy (g = 0.46).
- When the AI was the stronger party, combinations performed worse than the AI alone (g = -0.54).
Read that last pair again. What determines whether a person adds or destroys value is whether they correctly identify which side of the comparison they are on. That is a judgment, it varies between people, and it is assessable. The included experiments ran from January 2020 to June 2023, so the model capabilities are dated — but the finding is about human behaviour, not model vintage.
The same shape appears in the best-known field experiment on the question. In a preregistered study with Boston Consulting Group, 758 consultants were randomly assigned to work with or without GPT-4. On 18 tasks inside the model's capability, AI users completed 12.2% more tasks, 25.1% faster, at higher quality. On one task designed to sit outside it, AI users were 19% less likely to reach the correct answer (Dell'Acqua et al., HBS Working Paper 24-013). The authors call the boundary a "jagged technological frontier" precisely because it is invisible from inside the workflow. Finding the edge is the skill.
The benchmark that reversed itself
If you want a reason to be humble about measuring AI-assisted performance, look at what happened to the most careful attempt at it.
In July 2025, METR reported a randomised trial in which 16 experienced open-source developers worked through 246 real issues, randomly assigned to AI-allowed or AI-disallowed. They took 19% longer with AI available. They had forecast a 24% speedup beforehand, and after the fact still believed they had been sped up by 20% (METR, July 2025).
In February 2026, METR published a second round — 57 developers, 143 repositories, 800+ tasks — and estimated speedups rather than slowdowns (-18% time for returning developers, -4% for new recruits). It then declared its own new results unreliable. The reason is the interesting part: between 30% and 50% of developers reported withholding tasks from the experiment because they did not want to attempt them without AI, and others declined to participate at all on the same grounds (METR, February 2026).
So the measurement moved because the people being measured changed their behaviour around the instrument. Any assessment professional recognises that: it is range restriction produced by self-selection, and it is why a score collected under conditions candidates can shape is weak evidence. A test of "AI skills" administered as an open-book exercise on a candidate's own machine, with their own accounts and their own tool stack, has exactly this problem.
What a tool-knowledge test actually predicts
Three objections to the typical AI-skills test, in order of severity.
It has no defined construct. The *Standards for Educational and Psychological Testing* require that a test specify what it claims to measure before anyone reasons from the score. "AI proficiency" is not a construct; it is a topic area. Ask what inference the score supports and most of these instruments have no answer.
It has no criterion. The EEOC's guidance on employment tests is blunt that a selection procedure must relate to the job. Nobody has established that naming retrieval-augmented generation predicts performance in a role where AI is used. The honest position is that the predictive evidence does not exist yet.
It expires. The WEF's Future of Jobs Report 2025 projects that 39% of workers' existing skill sets will be transformed or become outdated between 2025 and 2030, with AI and big data the fastest-growing skill category. A score tied to a specific interface is a depreciating asset; a score tied to verification behaviour is not.
There is also a quieter point about compliance. Article 4 of the EU AI Act obliges providers and deployers to "take measures to support the development of AI literacy of their staff" — and then says explicitly that the obligation "does not require providers or deployers to guarantee any specific level of AI literacy of any individual" (European Commission AI Act Service Desk). The law asks for a training programme proportionate to context. It does not ask for a pass mark. Vendors selling certificates as the route to Article 4 compliance are selling something the text does not require.
Four things you can actually measure
Each of these is observable, scorable against a rubric, and tool-independent.
1. Task triage. Present a mixed set of short tasks and ask the candidate to sort them: delegate to AI, do unaided, or do with AI and check. Score against a key built from where the model in question actually succeeds and fails. This measures frontier awareness directly.
2. Verification behaviour. Supply AI output that contains a specific, findable defect — a transposed figure, a fabricated citation, a plausible-but-wrong clause — and score whether the candidate finds it and what they do next. Keep the defect rate unknown to the candidate so the task cannot be gamed by flagging everything.
3. Calibration. Ask for a confidence rating alongside each answer and score the gap between stated confidence and actual correctness. This is standard psychometric practice and it catches the exact failure mode METR's developers displayed: believing in a 20% speedup while running 19% slower.
4. Disclosure and traceability. Ask the candidate to show what they delegated, what they changed, and what they checked. Scored consistently, this separates people who can account for their process from those who cannot.
How to build one
- Write the construct statement first, in one sentence, naming the inference the score supports.
- Collect real failure cases from the work itself — defects that actually reached a reviewer — rather than inventing them.
- Fix the task set and administer it under controlled conditions, so the score is not shaped by what each candidate chose to attempt.
- Score with a behavioural rubric and calibrate your raters before they rate anything.
- Record the outcome you are trying to predict, and check the relationship after two or three hiring cycles. Without this step you have an exercise, not an assessment.
Scenario-based items of this kind are what AssessAll's AI-graded scenarios and custom assessment design are built for — a scored decision under uncertain advice, rather than a quiz about an interface.
When a tool-knowledge test is the right instrument
Two honest cases. Onboarding and policy: if your organisation has rules about what may be pasted into a model, people must know them, and a short knowledge check is the correct, cheap instrument. Training evaluation: a pre- and post-measure around an internal AI programme is a legitimate use of a knowledge test, because the inference is about whether the training landed, not about whether the person will perform.
What a knowledge test cannot do is carry a selection decision.
The takeaway
The evidence points one way: the gains from AI-assisted work do not come from knowing the tool, they come from knowing when the tool is wrong. Until an AI-skills test scores triage, verification and calibration against a real criterion, it is measuring the interface, and the interface will be different next year.