In short: AI resistance is a continuum, not a label, and three design variables decide where an assessment sits on it: how dynamic the format is, how tight the time limits are, and whether the information needed to answer is fragmented across screens or shown all at once. An assessment that is purely text-based, generously timed and presents everything on one page is the most exposed. None of this replaces validity, reliability or fairness, but it is a quality criterion in its own right, and one most organisations never had to consider when they chose their provider.
AI-based cheating represents the greatest threat to the predictive validity of assessments since they first went online. Never before have candidates had such a convenient and effective way to cheat, raising important questions about test integrity.
But one thing is clear, and often overlooked in this conversation: not every assessment is equally vulnerable.
Perhaps the most vulnerable are CVs, application forms, and written content. When people talk about AI, they typically mean large language models (LLMs), which naturally excel at language-based tasks.
At the other end of the spectrum are face-to-face interviews, although even these aren't entirely immune, as candidates could use smart glasses or hidden devices.
Clearly, AI resistance exists on a continuum. But what characteristics make particular assessments more or less resistant to AI-based cheating?
In this article, I outline the three main variables from the ACT framework that underpin AI resistance from an assessment design perspective.
Dynamic format
Perhaps most intuitively, given that LLMs excel at working with static text, assessments with more dynamic formats are inherently harder to cheat on.
Working with images is generally harder for AI and can take considerably longer. This is especially true when information moves around the page, making screenshots less effective.
These formats can also allow for faster assessments, as reading lengthy blocks of text tends to require longer time limits. Time limits are another important factor in AI resistance, which I'll address shortly.
Gamified assessments are a good example of this approach, forgoing lengthy blocks of text or static images in favour of dynamic game mechanics.
Naturally, this approach works well for cognitive assessments that can benefit from additional complexity, but may be harder to adopt for specific skills and knowledge domains.
As with all assessments, dynamic formats can still be cheated, but doing so may require considerably more AI expertise, preparation, and careful model selection than traditional assessments, increasing their AI resistance.
Short time limits
The next important variable is the length of the time limit.
Assessments with unlimited time give candidates unlimited flexibility to experiment with AI, whereas tight time limits keep candidates focused on the task at hand.
Additionally, because AI takes time to analyse inputs and generate responses, shorter time limits penalise candidates for introducing this extra step, making it less likely they'll be able to submit an answer in time.
Consequently, the more time candidates have, the greater the opportunity for AI-based cheating and the more vulnerable the assessment becomes.
As with dynamic formats, even quick-fire assessments can be cheated, but doing so requires careful preparation and model selection, potentially deterring would-be cheaters and increasing AI resistance.
Information fragmentation
The final, and perhaps least well-known, variable influencing AI resistance is information fragmentation: whether key information is spread across separate pages or presented together.
Many traditional ability tests present a shared information source (graph, table, or passage of text) alongside multiple related questions on the same page. Consequently, a single screenshot allows AI to answer multiple questions, making cheating considerably more convenient.
Alternatively, assessments that administer one question at a time, or display information sources separately from questions, are inherently more AI-resistant, as multiple screenshots may be required to cheat.
This works alongside strict time limits to make AI-based cheating harder, less reliable, and less open to experimentation.
Together, these three variables can substantially reduce the effectiveness of AI-based cheating, helping preserve the predictive signal of an assessment.
Practical implications
For employing organisations, the implications are clear. AI resistance operates on a continuum, and every assessment sits somewhere along it.
Consequently, when choosing or reviewing assessments, AI vulnerability is an important variable to consider.
Crucially, we can evaluate an assessment's AI vulnerability using three straightforward questions:
- Is the assessment purely text-based?
- Are the time limits too generous?
- Is all the required information shown at the same time?
If the answer to all three is yes, the assessment is particularly vulnerable to AI-based cheating, and this should count against it.
Naturally, AI resistance isn't the only consideration. Validity, reliability, fairness, and candidate experience also matter, but test integrity cannot be ignored.
Final thoughts
Employing organisations must move away from evaluating assessments in binary terms.
We don't categorise assessments simply as "reliable" or "unreliable", or "fair" or "unfair", because these characteristics exist on a continuum rather than as binaries.
Test integrity, and AI resistance more specifically, operates in exactly the same way.
An assessment might have a dynamic format but excessive time limits. Alternatively, information might be fragmented, but the assessment uses only static text.
Instead, all relevant variables should be considered together, particularly glaring weaknesses in important indicators of assessment quality.
Although AI resistance is just one consideration among many, it's a uniquely pertinent issue that simply wasn't considered, or even necessary to consider, when many organisations originally chose their assessment providers.
Consequently, now is the time to re-evaluate. What constitutes an ideal assessment has changed, and organisations must adapt their assessment strategies accordingly.
Ben Schwencke
Chief psychologist at Test Partnership. MSc in Organisational Psychology with over 10 years' experience in psychometric testing.
