Turn specs into evals with ASSERT for agent testing

2026-06-13 GIGATAP Team #security

#AI evaluation #agent systems #Microsoft Research

Turn specs into evals: ASSERT reframes how AI agents are tested by converting written behavioral intent into executable evaluation pipelines.

It addresses a persistent gap in agent development: teams define desired behavior in documents, prompts, or policies, but rarely translate those definitions into structured, repeatable tests that evolve with the system.

The result is a drift between what systems are supposed to do and what evaluation benchmarks actually measure.

ASSERT, an open-source framework described in Microsoft’s Responsible AI work, formalizes this translation layer so that behavior specs become runnable evaluation assets rather than static documentation.

This article distills how ASSERT changes evaluation practice, why spec-driven evals matter for agent systems, and what developers should be careful not to overinterpret.

What is ASSERT and why it matters?#

ASSERT turns written behavior requirements into structured, executable evaluations for AI systems and agents. Instead of treating specs as background context, it converts them into testable artifacts that can be run, scored, and iterated.

The core idea is operational alignment. If a system is expected to refuse unsafe financial advice, escalate fraud signals, or follow tool-use constraints, those expectations should exist as tests—not just as prompt-level instructions.

This matters because generic metrics like helpfulness or relevance often fail to detect application-specific failures. An agent can perform well on broad benchmarks while violating critical product rules in real workflows.

Definition capsule#

ASSERT (Adaptive Spec-driven Scoring for Evaluation and Regression Testing): a framework that converts natural-language behavior specifications into structured evaluation pipelines, including test generation, execution, trace collection, and policy-aligned scoring.

What changed in evaluation design#

Traditional evaluation pipelines rely on fixed benchmarks or loosely defined heuristics. ASSERT shifts this toward specification-driven generation.

Instead of manually designing test suites, teams provide a behavior specification. The system then derives:

structured concept definitions
a behavior taxonomy of allowed and disallowed actions
stratified test cases across scenarios
execution traces from model or agent runs
scoring aligned to the original policy intent

This changes evaluation from a static artifact into a continuously regenerable system tied directly to product requirements.

The important operational shift is trace-level inspection. ASSERT does not only score outputs; it records tool use, intermediate decisions, and execution paths for agent systems. That makes failure analysis closer to debugging a system than grading a response.

Why this matters for agent systems#

Agent systems fail differently from chat models. The failure often appears in tool orchestration, hidden state transitions, or policy violations that are not visible in final text output.

ASSERT explicitly targets this gap. It evaluates behavior across multiple dimensions: task type, environment configuration, tool access, and adversarial conditions.

The implication is direct. If evaluation does not mirror real operating conditions, regression detection becomes unreliable. Model updates can improve benchmark scores while degrading real-world safety or compliance.

Spec-driven evals reduce that blind spot by binding tests to the same language used in requirements engineering.

How the pipeline works in practice#

ASSERT operates in four stages:

Systematization: converts a broad behavior statement into a structured concept definition with edge cases and distinctions.
Taxonomization: builds a reviewable behavior taxonomy separating permissible and impermissible actions.
Test generation: produces stratified scenarios covering task types, personas, tool access, and adversarial inputs.
Execution and scoring: runs tests on the target system and evaluates full traces against policy-aligned criteria.

Each stage is editable, meaning evaluation evolves with product requirements instead of drifting behind them.

Comparison: generic metrics vs spec-driven evals#

Dimension	Generic evaluation	Spec-driven evaluation (ASSERT)
Input basis	Static benchmarks	Natural-language behavior specs
Coverage	Narrow, often synthetic	Stratified across real conditions
Failure visibility	Output-only	Full execution traces
Drift resistance	Low	High, updates with specs
Agent suitability	Limited	Designed for tool-using systems

The structural difference is not incremental. It changes what “evaluation” means: from scoring outputs to validating system behavior against declared intent.

What to check before adopting spec-driven evals#

ASSERT depends on the quality of the initial behavior specification. Weak or ambiguous specs produce weak taxonomies and unreliable tests.

The main operational risks are not technical but semantic:

unclear policy boundaries produce inconsistent taxonomies
incomplete specs miss edge-case behaviors
overfitting to generated test distributions can still occur if stratification is poorly defined

In practice, evaluation quality becomes a reflection of specification quality.

What not to overclaim#

ASSERT does not remove the need for human review. It does not automatically guarantee correctness or safety in agent systems.

It also does not replace benchmark suites entirely. General metrics still provide cross-system comparability, even if they fail to capture product-specific constraints.

The correct framing is complementary: spec-driven evals handle behavioral fidelity, while generic benchmarks handle broad performance signals.

Why now#

Agent systems are increasingly embedded in workflows with tool access, external context retrieval, and multi-step execution paths. That increases the cost of hidden failures.

Evaluation frameworks that rely only on final outputs are structurally mismatched to these systems. ASSERT reflects a shift toward instrumentation-heavy evaluation, where traces matter as much as responses.

Internal references#

FAQ#

Is ASSERT replacing traditional eval benchmarks?#

No. It extends them. Benchmarks remain useful for baseline comparison, but they do not capture system-specific behavior constraints.

What is the main advantage of spec-driven evaluation?#

It aligns evaluation directly with written behavior intent and makes those intentions executable and testable.

Crypto clipper adds Tor and worm-like spread to theft model - Microsoft analysis shows crypto clipper evolving into a persistent threat combining clipboard hijacking, Tor-based control, and worm-like propagation.
BCI moves from trial to real use as AI national strategy expands - A brain implant user with ALS signals early real-world BCI use while South Korea accelerates national AI focus and infrastructure alignment.
Postinstall payloads in npm supply chains and Mastra breach - Mastra npm packages were compromised through a postinstall hook that executed during install, exposing CI/CD pipelines and developer environments to remote
Security beyond benchmarks: Microsoft MDASH moves to production - Microsoft’s MDASH moves from benchmark success into live engineering pipelines, embedding AI-driven vulnerability discovery directly into DevSecOps workflo
VHDX Delivery Chain Hides Remcos RAT via JavaScript Stage - A ZIP file uses a VHDX disk image to expose JavaScript after auto-mounting on Windows, leading to Remcos RAT deployment through layered execution.
AI branding as phishing leverage in modern attacks - Threat actors are using AI platform branding as a trust layer on top of standard phishing infrastructure, increasing engagement without changing core attac
AI coding agents need access, not custody - 1Password’s Codex integration points to a cleaner model for agentic development: scoped, just-in-time credentials that stay out of prompts, code, terminals
AI found the bugs. Now comes the patch problem - Anthropic’s Project Glasswing reportedly surfaced thousands of serious vulnerability candidates. The real issue is the widening gap between discovery, vali

Latest articles

Crypto clipper adds Tor and worm-like spread to theft model - Microsoft analysis shows crypto clipper evolving into a persistent threat combining clipboard hijacking, Tor-based control, and worm-like propagation.
US Government AI Inventory Raises Privacy Questions - US agencies list 3,611 AI use cases, exposing gaps in transparency and raising questions about automated decision-making and privacy.
BCI moves from trial to real use as AI national strategy expands - A brain implant user with ALS signals early real-world BCI use while South Korea accelerates national AI focus and infrastructure alignment.
Canada’s Bill C-22 Pushes Encryption Toward Built-In Access Risk - Bill C-22 expands surveillance powers in Canada and introduces mechanisms that could force access paths into encrypted systems, raising structural privacy
Postinstall payloads in npm supply chains and Mastra breach - Mastra npm packages were compromised through a postinstall hook that executed during install, exposing CI/CD pipelines and developer environments to remote
Security beyond benchmarks: Microsoft MDASH moves to production - Microsoft’s MDASH moves from benchmark success into live engineering pipelines, embedding AI-driven vulnerability discovery directly into DevSecOps workflo
VHDX Delivery Chain Hides Remcos RAT via JavaScript Stage - A ZIP file uses a VHDX disk image to expose JavaScript after auto-mounting on Windows, leading to Remcos RAT deployment through layered execution.
Wolf Gallery and the F-Droid packaging bottleneck - Wolf Gallery’s F-Droid request shows how strong local encryption and offline design still depend on packaging work before reaching users through trusted di