Find out where your agents and models fail in production, then test the fix before it ships. Signals watches traffic on Managed Agents and Serverless Inference and flags the sessions that went wrong, with nothing extra to instrument. Evaluations and Simulations test your change on realistic multi-turn scenarios built from those sessions or from your agent's configuration.
Standard observability tells you the system was up and latency was fine. It can’t tell you the model gave a confidently wrong answer, the agent picked the wrong tool, or a frustrated user gave up. The tool call returned 200 — it just returned the wrong region. The conversation completed — the user left annoyed.
Raw trace volume can exceed 100 GB/s. Grading every session with an LLM judge is prohibitively expensive, and manual review doesn’t scale at all. Most teams learn about failures when a support ticket arrives.
Hand-built tests cover a narrow slice of real behavior. Synthetic test sets aren’t grounded in your agent or your users. Live traffic is realistic, but users pay for the mistakes. And single-response evals miss the failures that only show up across turns: a prompt tweak fixes step two and quietly breaks step five, or a fine-tuned model scores well on in training and regresses in real conversations.
Signals finds the agent and model sessions that went wrong in production and turns them into evaluations you can run. Nothing extra to instrument.
Agent and inference observability
Signals efficiently finds the agent and inference sessions that went wrong in production and turns them into evaluations you can run. Nothing extra to instrument.
For agents on Managed Agents and models served through Serverless Inference.
Evaluations scores agents and models against a consistent bar. Simulations, new in Evaluations, tests any change across full multi-turn conversations before users see it.
Scoring and validation
Evaluate any candidate on your inference stack: Serverless Inference models, Dedicated Inference endpoints, imported BYOM models, and Inference Router policies
New in Evaluations
Generate scenarios from Signals, plain-language descriptions, or your agent’s configuration
Your models serve through Inference Engine; your agents run on Managed Agents. Nothing to instrument.
Signals stitches turns into sessions, triages them, and identifies trends that you can drill into.
Flagged sessions become evaluation scenarios in one click.
Simulations runs your changed model or agent through those scenarios as full conversations in an isolated sandbox, then grades each conversation on multi-turn metrics.
Deploy the validated change. Signals keep watching. Scenarios worth keeping become regression tests.
Scenarios come from real sessions, and candidates run on the actual production harness, tools, and config. Real behavior, not a synthetic guess.
The same session that reveals a problem becomes the evaluation that validates the fix. No manual test-data prep.
Runs on Managed Agents and Serverless Inference with nothing extra to instrument, whether you’re operating agents, serving models, or both. One console, one bill, one team boundary for your data.
Turn on Signals for an agent or model you're already running, and rehearse your next change in Simulations before your users see it.
