Isn’t model accuracy enough? Why test safety separately?

Accuracy tells you whether a model does its job; it says nothing about how it behaves when misused, whether it treats groups equally, or whether it will still behave this way next quarter. Performance benchmarks exist industry-wide. Safety benchmarks do not, which is exactly why safety, fairness, and robustness have to be measured deliberately rather than assumed to come along with a good accuracy score.

Can we automate all of this, or do we still need humans?

Automation scales the known problems: classifiers and LLM-as-judge handle high-volume, well-understood harms. But novel jailbreaks and context-dependent bias have too little labelled data to automate, so they need skilled human testers. The right model is a partnership: automation for coverage, humans for the edge cases and for judging whether a flagged issue is actually a problem in context.

Why do you treat a model refusing requests as a failure?

Because a model that refuses safe, legitimate requests is broken in a different direction. Over-refusal quietly destroys the usefulness you built the model for, frustrates users, and pushes them toward less-governed alternatives. Sophisticated assurance measures both failure modes, unsafe compliance and over-refusal, and looks for the balance point, not the most cautious model possible.

We use a third-party model. Doesn’t the vendor handle testing?

Vendors test for their context, publish selectively, and often strip evaluation metrics from public cards for competitive reasons. When you deploy a model under your own name, you inherit its behaviour and its accountability. Independent assessment, reconstructing the missing metrics and red-teaming against your specific use cases, is what makes a third-party model defensible to your own regulator and customers.

How many times should a test prompt be run?

Enough for the result to stabilise. Because decoding is stochastic, a single run is not evidence: the answer depends on temperature and sampling settings. A practical method is to run ~100 generations on a sample of prompts, track token-entropy and n-gram diversity, and use the count at which those metrics flatten. Lower temperature and constrained sampling reduce the repetitions needed; higher creativity settings demand more.

How does this connect to the rest of the governance framework?

Assurance sits in the middle of the stack. It depends on a trustworthy data foundation (Layer 2) and secure inputs (Layer 3); its results feed human oversight (Layer 5), where thresholds trigger review, and compliance (Layer 6), where the evidence is retained for audit. Every system in scope comes from the AI inventory (Layer 1).

Layer 04 · AI governance

AI Model Assurance & Testing

Prove your AI model is fit, fair and safe with evidence — not assumptions.

A model can perform brilliantly in a demo and still fail in production. AI model assurance provides the evidence needed to understand how a model performs, where it fails, how it behaves under attack and whether it remains reliable over time.
From LLM evaluation and AI benchmarking to fairness testing, AI red teaming, validation and drift monitoring, this layer creates a repeatable approach to AI model testing across the entire model lifecycle.

Illustration for AI model assurance and testing

Continuous assurance

Know What Your Model Can Do Before It Matters

A production model needs to do more than generate impressive outputs. It needs to perform consistently, resist manipulation, operate fairly and remain within defined safety boundaries.

0dimensions scored

0assurance score

0red-team rounds

0models monitored

Illustrative figures for a representative estate.

Running assurance…

01 · Where It Begins

Is Your AI Model Really Ready for Production?

Before approving an AI system, five questions need clear, measurable answers.

Do you know exactly what the model should — and should not — do?

Model cards→

Is the model actually good enough for its intended use?

Performance & third-party→

Does the model perform fairly for everyone it affects?

Fairness testing→

What happens when someone deliberately tries to break it?

Red-teaming→

Will the model still behave correctly six months from now?

Drift detection→

02 · The Controls

Five Controls That Make AI Models Assurable

Effective AI assurance is not a single test or score. It is a set of controls that create repeatable evidence throughout the model lifecycle.

Model cards

Document what your model can do — and where it stops.

Definition

A model card provides a standardised record of a model's intended purpose, capabilities, limitations and known risks.

A comprehensive model card covers

Intended and prohibited use
Model capabilities
Known limitations
Performance metrics
Safety considerations
Security considerations
Fairness risks
Evaluation methodology
Monitoring requirements
Version history

Why it matters

A model that performs well in one environment may be unsuitable in another. Clear documentation prevents narrow performance results from being mistaken for universal reliability.
Model cards should also evolve as new evaluation results, incidents and risks emerge.

Performance & third-party benchmarking

Measure performance against more than your own expectations.

Definition

AI benchmarking provides context around model performance.

Testing can compare a model against

Internal performance baselines
Previous model versions
Industry benchmarks
Alternative foundation models
Relevant competitor models
Customer-defined thresholds

Evaluation can cover

Accuracy
Hallucination
Task completion
Safety
Refusal behaviour
Reliability
Latency
Cost
Grounding
Tool-use accuracy

Third-Party AI Review

When an organisation deploys a third-party model, vendor claims are only one source of evidence.
A Third-party AI review evaluates the model against the organisation's actual use cases, data, users and risk requirements.

Fairness testing

A strong average score can hide serious problems.

Definition

A model can achieve excellent overall performance while producing significantly worse outcomes for certain groups.

How it works

AI safety testing should therefore examine group-level performance rather than relying only on aggregate metrics.

Testing can assess

False-positive rates
False-negative rates
Error-rate differences
Selection rates
Disparate impact
Calibration
Group-level accuracy
Intersectional outcomes

Why it matters

A model that performs well on average can still create unacceptable outcomes for the people most affected by its decisions.

AI Red Teaming & Adversarial Testing

Attack your model before someone else does.

Definition

Traditional testing asks: Does the model work? AI red teaming asks: How can we make the model fail?

Testing can cover

Jailbreaks
Prompt injection
Indirect prompt injection
Multi-turn manipulation
System-prompt leakage
Sensitive-data extraction
Unsafe requests
Tool abuse
Agentic attacks
Data exfiltration
Obfuscated prompts
Cross-modal attacks

AI Agent Red Teaming

As AI systems gain access to tools and autonomous actions, AI agent red teaming becomes increasingly important.

Testing should examine whether an agent can be manipulated into

Calling unauthorised tools
Accessing restricted information
Taking unintended actions
Following malicious tool instructions
Escalating privileges
Repeating actions uncontrollably

AI Drift Detection

A model can pass every test today and fail tomorrow.

Definition

AI systems operate in changing environments.

How it works

Users change. Data changes. Prompts change. Retrieval sources change. Models are updated. Integrations evolve.

How we help

These changes can introduce AI drift without an obvious deployment event.

Monitor changes in

Input distributions
Output distributions
Accuracy
Safety performance
Fairness
Hallucination
Refusal behaviour
Latency Tool usage
Incident frequency

LLM Observability

LLM observability provides visibility into how models behave after deployment.
It helps teams identify unexpected changes across requests, outputs, errors, performance and usage patterns.

Assurance scorecard

Measure Risk Across Six Critical Dimensions

A single model score cannot tell the whole story.

Baseline assessmentAfter T3 remediation

03 · A Practical Reference

What Should a Defensible Model Evaluation Measure?

Not every evaluation produces meaningful evidence.
A credible LLM evaluation defines the test methodology, sample size, scoring approach and expected outcome before testing begins.

Model evaluation reference: test types and what they catch
Test type	What it catches	Method	Reported as
Baseline safety	Harmful requests the model may comply with	Controlled prompt library	Violation rate
Adversarial / jailbreak	Novel misuse and multi-turn attacks	Human red-teamers + automated probes	Attack success and severity
Over-refusal	Safe requests incorrectly rejected	Benign prompt suite	Refusal rate
Fairness	Unequal outcomes across groups	Group-level metrics + significance testing	Disparate impact
Cross-modal	Text that conflicts with image or audio	Grounding and modality checks	Grounding accuracy
Consistency	Different answers to repeated prompts	Controlled repeated runs	Pass@k / agreement

□

Why Repetition Matters Language models are probabilistic systems. The same prompt can produce different answers, while small changes in wording can alter the result.
A single test run is therefore not sufficient evidence.
Defensible AI testing uses controlled generation parameters, repeated runs and appropriate statistical analysis to determine whether results are consistent.

04b · Standards Mapping

Where AI Model Assurance Meets Recognised Standards

Evaluation evidence should connect directly to recognised regulatory and governance frameworks.

Model assurance: control-to-standard mapping
Control	EU AI Act	NIST AI RMF	ISO / other	Evidence
Model cards	Art. 11 & Annex IV	Map 1–4	ISO/IEC 42001 §7.5	Intended use and limitations
Performance & benchmark	Art. 15	Measure 2	NIST AI 600-1	Accuracy and robustness
Fairness testing	Art. 10(2)(f–g)	Measure 2.11	ISO/IEC TR 24027	Discrimination assessment
Red-teaming	Art. 15(5)	Measure 2.7	OWASP LLM Top 10 · MITRE ATLAS	Adversarial robustness
Drift detection	Art. 72	Manage 4	ISO/IEC 42001 §9	Continuous monitoring

04 · What a credible assessment includes

Six Rules for Evidence You Can Stand Behind

Whether you run it in-house or independently, a model assessment worth the paper it’s printed on covers the following.

Independent scoring. The team that develops the model should not be the only team deciding whether it passes.
Risk-tiered depth. Testing depth should increase according to model risk, user impact and potential harm.
Explicit thresholds. Pass and fail criteria should be established before testing begins.
Both failure modes. Measure unsafe compliance and over-refusal.
Statistical rigour. Document sample sizes, repeated runs, generation parameters and statistical methods.
A living library. Every significant incident or red-team discovery should become part of future regression testing.

From Our Engagements — Fluent Doesn't Mean Safe

“It sounded right” is the most expensive sentence in production AI. .

AI systems can produce confident, fluent and completely incorrect answers.
That is why AI quality assurance needs to evaluate more than language quality.

Model Assurance Maturity

How Mature Is Your AI Testing Programme?

Most organisations are further behind than they assume.
The biggest shift comes when organisations move from one-off testing to continuous assurance.

STAGE 01

Ad Hoc — “We Tried a Few Prompts”

Manual testing before launch. No consistent baseline. No structured evidence. No reproducible methodology.

STAGE 02

Checklist — “We Have a Test Set”

A fixed prompt library exists, but testing is often single-shot and the development team evaluates its own work.

STAGE 03

Benchmarked — “We Know Where We Stand”

Models are measured against explicit thresholds and relevant alternatives using structured methodology and independent review.

STAGE 04

Adversarial — “We've Tried to Break It”

Testing includes jailbreaks, multi-turn attacks, multimodal evaluation, agent testing and a living adversarial library.

STAGE 05

Continuous — “We're Watching It All the Time”

Automated regression testing, LLM observability and drift monitoring operate throughout the model lifecycle. Material changes trigger reassessment against established baselines.

Failure modes

Six Ways AI Model Assurance Breaks Down

The shortcuts that pass internal review and fail in the wild.

Refusal Rate = Safety

A model refuses many requests, so the team assumes it is safe.

The problem It may still be vulnerable to jailbreaks or unnecessarily refuse legitimate users.Fix Measure unsafe compliance and over-refusal together.

A Handful of Manual Prompts

A small set of obvious prompts passes.

The problem Indirect injection, tool abuse and multi-turn attacks remain untested.Fix Build repeatable adversarial and regression suites.

Tested Once at Launch

The model passes pre-deployment testing and is never tested again.

The problem Changes in data, prompts, users or model versions can introduce AI drift. Fix Establish baselines and continuously monitor meaningful changes.

The Development Team Tests Its Own Model

The same team builds, tests and approves the system.

The problem Independent challenge is missing.Fix Separate development, testing, remediation and approval where risk requires it.

One Run Per Prompt

One response is treated as definitive evidence.

The problem LLMs are probabilistic, so behaviour can vary between runs.Fix Use repeated runs and statistically meaningful evaluation.

Text-Only Evaluation of a Multimodal Model

A multimodal model is assessed only through its text output.

The problem Cross-modal hallucinations and grounding failures remain invisible.Fix Test visual, audio and multimodal grounding independently.

Multimodal Assurance

Three Failures Text-Only Testing Cannot See

When AI works with images, audio or video, additional failure modes appear.

Cross-Modal Hallucination

FFluent output contradicts the actual image, audio or video.

Detect it · Measure grounding as a separate accuracy dimension.

Modality-Dependence Failure

The model ignores the visual or audio input and answers using language priors.

Detect it · Change the input modality while keeping the prompt constant and measure whether the response changes appropriately.

Cross-Modal Inconsistency

The model gives materially different answers when the same task is presented through different modalities.

Detect it · Compare equivalent text-only and multimodal evaluations.

□

Tier Multimodal Risk Across Two DimensionsModality coupling depth — how dependent is the correct answer on the image, audio or video?
Consequence severity — what happens if the model gets the answer wrong?
A model used for general image descriptions presents a different risk from one supporting clinical, financial or safety-critical decisions.

By the numbers

The Evidence Behind Independent AI Assurance

LLM01Prompt injection’s OWASP rankremains a leading risk in the OWASP Top 10 for LLM Applications, reinforcing the need for adversarial testing.

16 / 84MITRE ATLASprovides an adversarial knowledge base that can inform AI red teaming and security testing.

362Documented AI IncidentsAI incident reporting continues to demonstrate why measurable safeguards and continuous assurance matter.

40 / 100Frontier Transparencyindependent testing becomes increasingly important when model providers disclose limited information about training and evaluation.

62%Security and Riskremain major barriers to wider adoption of agentic AI.

36% / 33%Framework AdoptionISO 42001 and NIST AI RMF are increasingly used to structure responsible AI practices.

Sources: OWASP Top 10 for LLM Applications; MITRE ATLAS; AI Incident Database; Foundation Model Transparency Index; Stanford HAI AI Index Report.

Go deeper

The AI Assurance Playbook

Seven distinctions that separate rigorous testing from reassuring theatre.

DefinitionRed Teaming vs Adversarial Testing+

AI red teaming approaches an AI system from an attacker's perspective and searches for unexpected ways to bypass controls.

Adversarial testing is the broader discipline of deliberately exposing models to difficult, manipulated and unexpected inputs.

FrameworkThree Levels of Testing Automation+

Level 01: Automated Regression — run large libraries of known tests consistently.
Level 02 Automated Adversarial Testing — generate variations of jailbreaks, prompt injection and other attacks.
Level 03 Human Red Teaming — use experts to identify novel, contextual and high-impact vulnerabilities.

Know a test’s tier before you promise to automate it.

FrameworkHow Should Harm Be Prioritised?+

(Consider × Likelihood × Severity × Exposure) A low-frequency failure can still require immediate attention when the potential consequence is severe.

Field noteWhy There Is No Single AI Safety Score+

AI safety covers multiple dimensions:

SecurityPrivacyFairnessReliabilityRobustnessHarmful behaviourTransparency

MethodHow Many Times Should a Test Prompt Run?+

There is no universal number.

The appropriate sample depends on model variability, testing objectives and risk. Higher-risk evaluations require repeated testing and statistical analysis rather than relying on a single generation.

FrameworkAgentic AI Benchmarks+

Traditional benchmarks often measure whether a model can answer a question.
Agentic AI benchmarks must also evaluate

Planning
Tool selection
Tool permissions
Action accuracy
Goal adherence
Recovery behaviour
Multi-step reliability
Resistance to manipulation
Human escalation

An agent can produce the correct answer while still taking an unsafe action.

MethodBuild a Governed Prompt Library+
A mature evaluation library should be: Versioned · Risk-mapped · Reproducible · Continuously updated
It should incorporate incidents, adversarial examples, regulatory requirements, domain-specific risks, multilingual scenarios, multimodal tests and agentic workflows.
Every meaningful failure should strengthen the next evaluation cycle.

05 · In practice

AI Model Assurance in the Real World

Model assurance is not abstract. Each scenario below shows a genuine challenge, the controls that addressed it, and the outcome, anonymised across regulated industries.

Financial services
Retail Bank · Customer-Facing GenAI Assistant

Challenge
A retail bank was preparing to launch an AI assistant but could not demonstrate how the model would behave under adversarial pressure or compare its risk profile with competing systems.

Controls applied
PerformanceRed-teamingModel cards

Outcome
A structured benchmark identified a multi-turn jailbreak that exposed internal system instructions. The vulnerability was fixed before launch, while the resulting assurance scorecard provided evidence for the risk committee.

Key learning
The most useful metric was not the headline pass rate. It was the model's worst-case behaviour compared with relevant alternatives.

Healthcare
Diagnostic Imaging · Multimodal Model

Challenge
A digital-health provider had a vision-language model producing fluent clinical reports without evidence that its responses were correctly grounded in the underlying images.

Controls applied
PerformanceFairness testingDrift detection

Outcome
Cross-modal testing identified confident but incorrect extractions above the acceptable threshold. Additional human review was introduced for high-risk outputs.

Key learning
Multimodal models require dedicated grounding tests. Text-only evaluation cannot identify every image-related failure.

Technology / SaaS
AI-Native Scale-Up · Enterprise Sales Blocker

Challenge
An AI-native SaaS business repeatedly encountered security concerns during enterprise procurement. It could claim its product was safe but lacked independently generated evidence.

Controls applied
PerformanceModel cardsDrift detection

Outcome
A benchmarked evidence pack became part of the enterprise sales process, giving security teams measurable information about model behaviour and ongoing monitoring.

Key learning
AI assurance can become a commercial asset when enterprise customers require evidence before adopting an AI product.

Public sector
Government Agency · Third-Party Foundation Model

Challenge
A government organisation wanted to deploy a third-party foundation model but had limited independent evidence about its behaviour in the agency's specific use cases.

Controls applied
Model cardsRed-teamingFairness testing

Outcome
An independent evaluation established additional metrics, tested public-sector misuse scenarios and created documentation to support governance and oversight.

Key learning
Using a third-party model does not remove the deployer's responsibility to understand and validate its behaviour.

Disclaimer: illustrative use cases based on anonymised real-world scenarios.

06 · Questions Leaders Ask

AI Model Assurance Q&A

No. Accuracy measures performance against a defined task but does not necessarily reveal security vulnerabilities, unfair outcomes, jailbreak susceptibility, privacy leakage or robustness failures.
AI model testing needs to evaluate the risks relevant to the specific deployment.

Automation provides scale and repeatability, but human expertise remains important for novel attacks, contextual harms, ambiguous outputs and high-consequence scenarios.
A mature programme combines automated LLM evaluation with human assessment.

A model that refuses legitimate requests can become unusable.
Safety therefore requires measuring both unsafe compliance and inappropriate refusal.

Vendor testing provides useful evidence but does not replace an independent assessment.
Your prompts, users, data, retrieval systems, tools and business context can create risks that vendor benchmarks do not cover.

It depends on the model, test objective and risk.
Higher-risk evaluations should use repeated runs and appropriate statistical methods rather than relying on a single generation.

LLM observability provides visibility into model behaviour in production. It can help teams monitor requests, outputs, latency, errors, usage patterns and other operational signals.

It depends on the model, test objective and risk.
Higher-risk evaluations should use repeated runs and appropriate statistical methods rather than relying on a single generation.

AI model assurance sits within the wider AI governance lifecycle.
The AI inventory establishes which systems exist and their risk levels. The data foundation establishes the quality and provenance of supporting data. Security protects models and data. Assurance tests behaviour. Human oversight determines when intervention is required, while compliance processes retain and map the resulting evidence.

Continue through the stack

Related layers

L01 · scope for this layer← AI inventory
Every model tested through this layer should originate from the AI inventory, where AI systems are identified, documented and risk-classified.
L02 · below feeds this layer← Data foundation
Model evaluation depends on the quality, provenance and characteristics of the data supporting the system.
L05 · this layer feeds oversightAI risk management →
Testing creates evidence. Human oversight determines when results require review, escalation, intervention or override.
L06 · this layer feeds complianceCompliance & audit →
Evaluation evidence must be retained, mapped to relevant obligations and maintained in an audit-ready format.

Next step

How Well Do You Really Know Your AI Models?

Book a model-assurance review to assess how your organisation currently evaluates AI systems across model documentation, benchmarking, fairness, adversarial testing and continuous monitoring.
Identify where missing evidence could expose your organisation to operational, regulatory, security or commercial risk.
Book a review →

EMAILcontact@t-3.ai

WEBt-3.ai

UK+44 20 8087 0917

US+1 213 659 0224

Why T3

Why T3 for AI Model Assurance & Testing?

T3 is an award-winning AI implementation partner for high-risk industries.

We support trustworthy AI across the entire lifecycle, from pre-deployment AI testing and AI model validation to adversarial testing, AI red teaming, benchmarking and continuous assurance.
Our specialists provide AI Adversarial Testing Services, AI agent red teaming, third-party model assessments and ongoing LLM observability programmes.

We design and engineer bespoke AI controls, conduct adversarial red teaming on models and AI systems, and implement end-to-end AI governance operating models aligned with the EU AI Act, ISO/IEC 42001, and NIST AI RMF.

Where off-the-shelf GRC platforms stop, we build the custom controls, integrations and assurance that fit your stack, models, workflows and regulatory environment.

Trusted by two-thirds of BigTech and Financial Services, this is where policy meets engineering.