Build vs. Buy an Assessment: Three Conditions Decide Whether You Can Use Someone Else's Validity Evidence

Buying an assessment means relying on validity evidence someone else gathered; building one means producing that evidence for your own job. The choice is not about price. It is about whether published evidence transports — and in the United States that takes sound validity evidence, substantially the same major work behaviours, and fairness evidence for your own labour market.

Build-versus-buy is usually run as a procurement exercise: cost per candidate, catalogue breadth, integration effort. Those matter, but the decision is which option leaves you holding defensible evidence that the scores mean what you say they mean for the job you are hiring for.

The two options, stated precisely

Buy — an off-the-shelf instrument

• Items, norms and validity studies were developed on samples that are not yours.
• Your evidence file is the publisher's technical manual plus your argument that it applies here.
• Cost is per candidate and starts immediately; no development lead time.

Build — a custom instrument

• You run a job analysis, write and pretest items, set a cut score, and accumulate local evidence.
• The evidence file is yours, and it answers questions about your job directly.
• Cost is front-loaded in expert time and pretest candidates, not licence fees.

The asymmetry is this: buying does not remove the evidence obligation. It changes who produced the evidence, and raises the question of whether you are entitled to use it.

Borrowing validity evidence has three conditions, not one

Under the US Uniform Guidelines on Employee Selection Procedures, 29 CFR 1607.7(b), criterion-related validity evidence from another source is acceptable only when three conditions are met:

1. Validity evidence. The available studies must meet the technical standards of section 14B and clearly demonstrate that the procedure is valid.
2. Job similarity. Job analyses of both the studied job and your job must show that incumbents "perform substantially the same major work behaviors".
3. Fairness evidence. The studies must include a test-fairness investigation for each race, sex and ethnic group that is a significant factor in the borrowing employer's relevant labour market — with a narrow allowance to proceed where an internal fairness study is not yet technically feasible.

Two of those three conditions are about your setting, not the publisher's, and the EEOC's guidance on tests and selection procedures makes the same point from the enforcement side: the employer using the procedure owns the job-relatedness argument. The Guidelines are US law and do not bind a hiring team in Bengaluru, Manila or Riyadh, but the reasoning travels, because the *Standards for Educational and Psychological Testing* (AERA, APA and NCME, 2014) treat validity as a property of an interpretation for a specified use, not of an instrument.

What the professional principles add

SIOP's *Principles for the Validation and Use of Personnel Selection Procedures* set out what transportability requires: the original study reviewed for technical soundness and for conceptual and empirical relevance, a case resting on "a preponderance of key observable and/or underlying similarities", and "comparability in terms of job content or job requirements, job context, and job applicant group (if feasible)".

On meta-analysis they are stricter than sales material usually implies: generalisation is warranted only when the conditions and settings of the meta-analysis match the new situation, and generalising to procedures or settings that were never specified is not justified. The synthetic or job-component route requires a detailed work analysis too.

So the job analysis many teams buy off-the-shelf tests to avoid is the same job analysis needed to justify buying one. It is the most useful thing to know before the procurement call.

Category validity is not product validity

The revised meta-analytic estimates from Sackett, Zhang, Berry and Lievens (2022) put operational validity at .42 for structured interviews, .38 for empirically keyed biodata, .31 for general mental ability and integrity tests, .26 for situational judgment tests and .19 for conscientiousness measures, and the authors work through the consequences for whole selection systems in a 2023 follow-up.

Those are category estimates. They describe what well-constructed instruments of a type tend to achieve; they are not a warranty on one product, and they say nothing about how much a purchase adds to the screening you already run. A test that predicts well but measures what your interview already measures contributes far less than its headline coefficient suggests. Ask a vendor for incremental validity over your current process — and if nobody has computed it, say so rather than inferring it.

What building costs, counted in people rather than currency

Build costs are underestimated because they are quoted in money. Count them in participants instead.

Expert panel for content evidence

• Lawshe's content validity ratio sets a minimum agreement level by panel size. With 10 panellists the critical CVR is .80 — 9 of 10 experts must rate an item "essential" — falling to .50 with 20 panellists; Ayre and Scally (2014) recalculated the exact binomial values and confirmed both figures.
• So a small panel is not the cheap option. It demands near-unanimity on every item you keep.

Test length for reliability

• From the standard internal-consistency relationship, alpha of .80 takes about 23 items when the average inter-item correlation is .15, and about 36 items at .10. Pushing that same .15 pool to alpha of .90 takes roughly 51 items.
• Precision is bought in items, and items are bought in SME hours.

Pretest sample for calibration

• Şahin and Anıl (2017) calibrated 10-, 20- and 30-item subtests drawn from a 50-item English test taken by 6,288 students, across samples from 150 to 5,000. Under a two-parameter model, accurate item parameters needed about 750 examinees for the 10-item test, 500 for 20 items and 250 for 30 items; a one-parameter model managed at 150 throughout.
• That is specific to language tests of this kind, but the shape generalises: longer tests need smaller calibration samples, more complex models larger ones.

A criterion-related study is a separate and much larger ask, which is why most credible custom builds lead with content and construct evidence and add criterion evidence as hires accumulate.

If the instrument uses AI, the build/buy line moves legally

Under Article 25(1) of the EU AI Act, a deployer becomes the provider of a high-risk AI system if it puts its name or trademark on a system already on the market, substantially modifies it, or changes the intended purpose of a non-high-risk system so that it becomes high-risk. Recruitment and candidate-evaluation systems sit in the high-risk category.

So "lightly customised" is not always buying. White-labelling a vendor's scoring engine, or repurposing a learning diagnostic into a hiring screen, can move the full provider obligations onto the employer. If you hire into the EU, settle this in the contract before the pilot, not after.

When buying is the right call

• The role is common and well-studied, with published evidence for jobs sharing substantially the same major work behaviours.
• Volumes are low or uneven, so a development project would never amortise.
• You need to start screening in days: a defensible generic measure now beats a perfect bespoke one in six months.
• The construct is generic by nature — numerical reasoning, workplace English, integrity — where local item writing adds little.

This is the case pay-as-you-go procurement is built for: on AssessAll an assessment is one credit at ₹30 or US$0.50, with 100 free credits on individual signup and 250 on corporate signup, so a pilot can be run on real candidates before anything is committed.

When building is the right call

• The job is unusual, proprietary, or defined by work behaviours no published study covers.
• Volume or stakes are high enough that a few points of incremental validity change the economics.
• You need content evidence tied to your own job analysis, because pay-equity, regulatory or litigation exposure makes "it worked elsewhere" an uncomfortable position.
• The construct is context-bound: judgment in your escalation paths, your product, your compliance rules. Scenario-based items written on your own cases and AI-graded against your rubric do something no catalogue test can — the work AssessAll's custom assessment service exists for.

The decision rule

1. Write the job analysis first. It is required either way and it decides everything else.
2. Ask whether published evidence covers jobs with substantially the same major work behaviours. If it does not, the question is already answered.
3. Ask the vendor for the technical manual, the study samples, any fairness evidence, and incremental validity over your current process. Treat an unanswered question as a missing answer.
4. Price the build in expert hours, items and pretest candidates rather than licence fees, then weigh that against the cost of weaker decisions at your real hiring volume.
5. Either way, monitor your own outcomes. Borrowed evidence justifies starting; it never substitutes for watching what your scores do in your own funnel.

Buying is cheaper and faster but leaves you arguing comparability; building is slower and costlier but leaves you holding the evidence. Run the job analysis before the budget conversation and the choice usually makes itself.