Pre/post measurement is evidence that training worked only if the measure means the same thing at both time points. Self-report scales usually do not: learners revise their internal standard as they learn, so real improvement can register as zero. One leadership programme, measured four legitimate ways, produced effects from d = .69 to d = 1.22.
That range is not a rounding problem. It is the entire finding, and it comes from the evaluation design rather than from the training. Below are the three designs most L&D teams actually use, what the evidence says each one measures, and when each is the right call.
Design 1: Self-report pre/post
Ask participants to rate their own capability on a 1-to-7 scale before the programme and again after. Subtract. Report the gain.
This is the most common design in corporate L&D and the least defensible, for a reason that has been documented since 1979. Howard and Dailey named it response-shift bias: the act of being trained changes the yardstick. A participant who rated themselves 5 on "gives developmental feedback" before the course did so using a naive definition of what developmental feedback is. After two days of practice, they understand the skill better — and rate themselves 5 again, or lower. Their capability went up. Their score did not.
Rohs (1999) caught this cleanly in a leadership development study. The conventional pre/post group (n = 30) moved from a total of 70.00 to 70.36 — no significant change on any item or on the total. A parallel group that rated their pre-programme selves after the course showed significant gains on every item, and their retrospective rating sat about 18 points below the conventional pretest (t = -4.46, p < .001). Same programme. Opposite conclusions.
There is a second problem underneath the first. Sitzmann, Ely, Brown and Bauer's meta-analysis of 166 studies and 41,237 learners found that self-assessed knowledge correlated only .34 with actual cognitive learning, while correlating .51 with reactions and .59 with motivation. Their conclusion is blunt: self-assessment is a better indicator of how learners feel about a course than of what they learned. About a third of the evaluation studies they reviewed were treating it as a learning measure anyway.
When self-report pre/post is still the right call
- You are measuring something that is genuinely subjective and has no external criterion — confidence, psychological safety, intent to apply.
- You want a reaction measure and you are willing to label it as one.
- The decision it feeds is low-stakes: iterating on content, not justifying budget.
Design 2: The retrospective pretest (then-test)
Administer the "before" rating at the same moment as the "after" rating: rate yourself as you are now, and rate yourself as you now realise you were then. Both ratings use the post-training yardstick, which is exactly what response-shift bias breaks.
It works, and it introduces a new failure mode: the design choices inside it move the answer as much as the training does.
Nimon, Zigarmi and Allen evaluated a single Situational Leadership II programme (15 classes, 139 usable responses) under four retrospective-pretest variants. The post-to-then effect sizes were:
- Pre-Post-Then (separate surveys, conventional pretest retained): d = .69
- Post-Then (separate post and then surveys): d = .74
- Post&then (one combined survey, adjacent items): d = 1.05
- Pre-Post&then: d = 1.22
Same programme, same measure, a 77% difference in the reported effect depending on the survey layout. The authors also carried a control scale that the training should not have touched. It moved too — d = .23 to .42 — and only the most conservative design produced a non-significant result on it. Where the combined-survey formats put then and post items side by side, participants appear to have constructed the contrast they thought was expected.
The same study found something that should settle the argument against conventional pretests on self-report scales: pre ratings correlated .014 and .004 with an objective skills test, against benchmarks of about .42 and .41 for the same instrument. The conventional pretest was not merely biased. It carried almost no information.
When the retrospective pretest is the right call
- The construct is one participants cannot accurately judge before instruction — most leadership and communication skills qualify.
- You use separate post and then surveys, not one form with adjacent items.
- You include a control scale that training should not affect, and you report what it did.
Design 3: Objective measure plus a comparison group
Score capability with an instrument that does not depend on the participant's self-concept — a knowledge test, a scored work sample, a rated scenario response — and compare the trained group against an untrained or wait-listed one.
This is the design the evidence base is actually built on. Arthur, Bennett, Edens and Bell's meta-analysis of organisational training reported a sample-weighted mean d of .62 across 397 effect sizes and 33,325 participants: .63 for learning, .62 for behaviour, .62 for results. Its method section notes that "studies based on single group pretest–posttest designs were excluded from the data." The best available estimate of what training does was built by throwing out the design most organisations use to prove it.
The same preference is codified outside L&D. The US Department of Education's What Works Clearinghouse Procedures and Standards Handbook (Version 5.0, 2022) sets standards for group designs with a comparison condition, regression discontinuity designs, and single-case designs with repeated baselines. A one-group before-and-after study is not among the things it rates.
When a comparison group is not worth it
- Compliance or certification training, where the question is whether each person reached a standard, not whether the cohort improved on average. A criterion-referenced cut score answers that on its own.
- Cohorts too small for the comparison to carry any precision.
- Situations where withholding training is genuinely unacceptable — in which case a staggered rollout gives you a wait-list comparison for free.
The arithmetic nobody runs
Even with an objective test, a raw gain score is less reliable than either test that produced it. Where pre and post have the same reliability, the difference score's reliability is approximately (r_xx − r_pre,post) ÷ (1 − r_pre,post).
Put real numbers in. With tests at .85 reliability and a pre-post correlation of .70, the gain score reliability is (.85 − .70) ÷ (1 − .70) = .50. Half the variance in your "improvement" column is noise. That is why individual-level gain scores should almost never drive a decision about a named person, and why residualised change — regressing post on pre and using the residual — is the better statistic when you need one.
Two practical consequences. First, use enough items: short post-tests make this worse, and the error band widens exactly where you want precision. Second, use linked forms rather than the identical test twice, or you measure recall of the test alongside the capability. Platforms that draw pre- and post-assessments from one calibrated item bank — AssessAll's AI-graded scenarios work this way, on pay-as-you-go credits — make that practical at cohort scale without maintaining two hand-written papers.
A four-line checklist before you report a training number
- Name the design. If it is self-report pre/post, say so in the same sentence as the number.
- Carry a control scale or an untrained comparison, and publish what it did.
- Report the standard error or confidence interval on the gain, not just the gain.
- State what the measure is anchored to — a criterion, a norm group, or nothing.
The takeaway
A training effect size is a property of the evaluation design at least as much as of the training. Before you argue about whether a programme worked, make someone state which of these three designs produced the number — because on the published evidence, that choice alone can nearly double it.
Sources: Howard & Dailey (1979), Journal of Applied Psychology 64(2); Rohs (1999), *Journal of Agricultural Education* 40(4), 28–37; Sitzmann, Ely, Brown & Bauer (2010), *Academy of Management Learning & Education* 9(2), 169–191; Nimon, Zigarmi & Allen (2011), American Journal of Evaluation 32(1), 8–28; Arthur, Bennett, Edens & Bell (2003), *Journal of Applied Psychology* 88(2), 234–245; Standards for Educational and Psychological Testing.