Elastic Built a Shared Eval Framework for Its AI Agents to Stop Regressions Across RAG and Security Workloads

Elastic has consolidated the way it tests its AI agents, moving from one-off evaluations scattered across teams to a single framework that runs against production workloads. Susan Chang, who works on the company's agentic AI systems, laid out the shift in a presentation, describing a pipeline that now covers retrieval-augmented generation and cybersecurity use cases under the same measurement regime. The change matters because agent behavior is hard to pin down: a prompt tweak that helps one workflow can quietly break another, and without a shared yardstick those failures surface in production rather than in review.

The framework rests on two scoring methods rather than one. Chang described combining LLM-as-a-judge, where a model grades an agent's output against criteria, with deterministic rules that check hard requirements such as format, tool calls and forbidden outputs. Neither approach carries the load alone: model judges handle the fuzzy, open-ended judgments that rules cannot express, while rule-based checks catch the mechanical failures that a judge model might wave through or score inconsistently run to run.

A second seam the team had to close runs between languages. Data scientists building evaluations tend to work in Python, while the agents themselves often ship as TypeScript services, so results and test definitions lived in different places. Chang described bridging the two so that an evaluation authored on the data side executes against the same production code path users hit, which keeps the measurement honest instead of testing a reimplementation that drifts from what actually runs.

Tracing sits underneath all of it. Chang said Elastic added deep tracing to follow an agent's steps across multi-stage RAG pipelines and security workflows, so when a score drops, engineers can see which retrieval call, tool invocation or reasoning step caused it. That trace-level detail is what turns an evaluation number into an actionable bug report, and it preserves the domain context that generic benchmarks strip out — a security agent's correct answer depends on rules and threat models that a general-purpose eval would not encode.

The presentation reflects a broader pattern among companies shipping agents: evaluation is becoming infrastructure rather than a notebook exercise. Teams that started with a handful of hand-checked examples are finding that approach does not scale once agents chain tools, retrieve from large corpora and run unattended. Elastic's answer is to treat evals as production code, versioned and run continuously alongside the systems they measure.

What to watch is whether this kind of framework travels beyond the team that built it. Evaluation setups are notoriously tied to the workloads they were written for, and the open question is how much of Elastic's Python-to-TypeScript bridge and tracing stack can be reused by other organizations, or whether each company ends up rebuilding the same plumbing for its own agents.