Where does recruitment signal come from when candidates can use AI?
2 minutes
Ben Schwencke on why allowing AI into your selection process does not automatically mean you are measuring AI skill — and what it means for signal quality.
A question has started appearing more regularly in conversations about assessment design: should AI be allowed during the process, not just tolerated or policed? Some assessment providers and hiring teams are making the case for AI-integrated assessments — tests designed not to block AI use but to incorporate it. The argument is straightforward: if AI is part of how the role is done, an assessment that bans it may not reflect the real job.
It is a credible starting point, and the debate deserves a serious answer. Whether allowing AI in your assessments is productive or counterproductive depends almost entirely on what you are trying to measure — and whether the assessment design actually measures that once AI is in the room.
The case for embracing AI in assessments
The argument for AI-integrated assessment starts from a reasonable premise: if candidates will be using AI tools in the role, an assessment that forbids those tools may not reflect real working conditions. A copywriter who can produce strong briefs using AI assistance is valuable, regardless of whether they could do the same without it. A data analyst who can use AI to check their reasoning and spot errors in their work is a better hire than one who cannot, or will not.
There is also a practical argument. Preventing AI use is increasingly difficult. Candidates who are motivated to cheat will find ways, and detection is imperfect. Rather than running an arms race against every new AI capability, the argument goes, you might be better off designing assessments where AI use is expected — and where the skill being measured is how candidates use it.
These are real considerations. Some hiring teams and assessment providers have started building their processes around this logic, and it is worth examining what that looks like in practice.
What AI-integrated assessments actually look like
AI-integrated assessments take a few different forms. Some simply remove restrictions: candidates can use whatever tools they want during a traditional task, and the output is evaluated on quality rather than process. Others are designed specifically around AI interaction — candidates are asked to prompt an AI model to complete a task, evaluate the output it produces, identify errors or gaps in an AI-generated summary, or explain and defend the reasoning behind AI-assisted decisions.
The more sophisticated versions try to contain something the AI cannot do alone: a judgment call, a contextual decision, or a domain-specific error that requires real knowledge to catch. The candidate's score is meant to reflect not just the quality of the AI's output but the quality of the candidate's oversight of it.
In principle, this sounds like a sensible adaptation. In practice, there are serious questions about what these assessments are actually measuring — and whether it holds up as a useful predictor of performance.
The validity problem
The core issue with allowing AI in an assessment is that it changes the construct being measured. If you set a reasoning task and allow candidates to use AI, you are no longer measuring their reasoning ability. You are measuring some combination of the model they chose, the time they had, their familiarity with the interface, and — at best — some baseline judgment about the quality of the output. That is a much weaker and harder-to-interpret signal than the one you started with.
Using AI does not mean they are skilled in AI. The whole point of artificial intelligence is so you don't have to use your own.
This is the part that tends to get glossed over. When two candidates complete an AI-assisted task, how much of the difference in their outputs reflects a difference in their ability — and how much reflects which model they used, or how many attempts they made, or whether they had previous experience with that particular tool? These are not easy questions to answer, and most AI-integrated assessments do not answer them. The result is a score that is difficult to interpret and harder to defend if a hiring decision is challenged.
There is also a durability problem. Prompting skill is a rapidly moving target. What counts as effective prompting today may be unnecessary tomorrow as models become better at interpreting vague instructions. An assessment designed around current AI capabilities may have a short shelf life as a valid predictor of anything.
The key question to ask of any AI-integrated assessment: if two candidates with very different underlying ability both use the same AI tool for the same amount of time, how different are their scores likely to be? If the honest answer is "not very," the assessment is measuring access and familiarity, not ability.
When AI-native assessment can legitimately work
There are roles where assessing how someone works with AI is a genuine priority — and for those roles, a well-designed AI-native task can provide useful information. The key word is "well-designed." The task needs to contain elements that the AI genuinely cannot supply: domain judgment, contextual reasoning, or the ability to identify errors that require real knowledge to catch.
An example might be a task where a candidate is given an AI-generated report and asked to identify factual errors, questionable assumptions, or gaps in the analysis. If the errors require domain knowledge to spot, then a candidate who catches them has demonstrated something real. The AI helped produce the document; the candidate's ability is visible in how they evaluated it.
Similarly, tasks that ask candidates to defend or explain AI-assisted decisions — rather than just produce them — can surface genuine reasoning, because a candidate who does not understand the logic behind an output will struggle to explain it under questioning.
But these designs are narrow in scope. They measure AI-specific proficiency, not the broader cognitive traits that predict performance across most roles. They work for specialist positions where AI collaboration is central to the job and where the candidate pool is likely to have existing familiarity with the relevant tools. They are not a general-purpose replacement for ability testing.
| Assessment approach | What it actually measures | Valid for |
|---|---|---|
| Allow AI on a traditional ability test | Model quality + candidate familiarity with the tool | Almost nothing — the construct you needed collapses |
| AI-native prompting task (open-ended) | Familiarity with prompting; model access; effort | Weak predictor; hard to standardise or interpret |
| AI-native evaluation task (critique AI output) | Domain knowledge + judgment about AI output quality | AI-specialist roles where output evaluation is central to the job |
| AI-resistant ability test | Cognitive ability (unassisted) | Strong predictor of performance across most roles |
| AI-resistant ability test + AI evaluation task | Both constructs measured independently | Roles where both general ability and AI proficiency genuinely matter |
The construct you cannot afford to lose
The most consequential question in AI-integrated assessment design is whether the construct you actually needed has been preserved. Most organisations are not primarily trying to measure AI proficiency — they are trying to measure the cognitive capacity that predicts whether someone will perform well in the role. That construct and AI-assisted performance on a task are not the same thing, and conflating them does not produce a useful composite. It produces a score that is difficult to interpret and hard to act on.
The design principle that follows is not complicated: if you need to measure cognitive ability, the assessment must capture it unassisted. If you additionally need to measure how someone works with AI, design a separate task for that purpose. Two interpretable signals will always be more useful than one muddled one — especially when a hiring decision is later scrutinised or challenged.
That separation also clarifies who this debate is genuinely relevant to. For most roles, the question of whether to allow AI in assessments is primarily a question about construct integrity — and the answer is to keep the constructs clean. For a narrower set of AI-specialist positions, it is a real design challenge worth working through carefully, and the frameworks in the previous section are the right place to start.
See how we keep cognitive ability measurement clean
Our assessments are designed so that AI assistance is practically irrelevant to the score — measuring what you actually need without the validity risk.
Primary author
Ben Hopgood
Marketing Content Manager at Test Partnership, responsible for developing and executing content and marketing strategies across the business.
Contributed to by:
