Inside DigitalOcean Simulations: how user simulation, execution, and validation work

Ask a coding agent which Python version to upgrade to from 3.11, and it picks one from memory. It cites a few changes from that version’s release notes and calls the upgrade straightforward. A single-turn evaluation asking “Is this a reasonable recommendation?” would likely pass it.

An experienced human engineer works through it over several exchanges:

• They check which versions are currently supported.
• They lay out the options and let you choose.
• Then they work out what breaks between where you are and where you are going, including the releases in between, and check your code for it.

Each step builds on the previous one’s findings, and none of that is visible in a single prompt and response.

That is the gap Simulations - a capability of DigitalOcean Evaluations now in Public Preview - is built to close. Instead of scoring a single response, Simulations runs a full conversation between a simulated user and your agent, records every message and tool call in a trajectory, and has an LLM judge the whole thing. The launch announcement covers what Simulations does. Here’s how it works under the hood.

We’ll follow one question through every stage: which Python version should a service on 3.11 upgrade to? You’ll see how to write the scenario, simulate the user, run the agent, score the conversation, and compare a baseline against a fix. Along the way, we’ll flag the details that decide whether a simulation’s results are worth trusting.

Simulations, at a glance:

• Whole conversations, not single responses. A simulated user plays each scenario turn by turn, reacting to what the agent and its tools actually return.
• Real harness, real tools. The agent runs in an isolated Harness Runtime sandbox with its production configuration and Action Gateway tools.
• Stopping criteria as the goal. stopping_criteria lists what the agent should do, and the conversation ends once it has. expected_outcome is only needed for Agent Policy Adherence.
• Trajectory-level scoring. The judge sees every message and tool call at every turn, so it can distinguish a reasoning failure from a tool-use failure and catch an agent that forgets earlier turns.
• Baseline vs candidate. Run the same scenarios against versions of your agent, or two models, and see exactly what improved and what regressed.
• CLI and API first. Every step is available through doctl evaluation and the DigitalOcean API.

DigitalOcean Simulations: the four components

Think of a simulation as a staged conversation with a grader in the back of the room. These four components make it work:

• The candidate is what you are testing: a managed agent on Harness Runtime, a model, Dedicated Inference deployment, or Inference Router that the simulation wraps in an agent. For this example, we’ll refer to it as the agent we’re testing.
• The scenario describes a situation for the user simulator to act out: what the user wants, who they are, what the agent should get done before the conversation ends, and how many turns to allow.
• The user simulator is a model that plays the user, one turn at a time, based on the scenario.
• The judge is a model that scores the finished conversation against the metrics you pick.

You’ll also want to understand these Simulations concepts:

• A turn is one exchange: a user message and the agent’s reply, including any tool calls.
• An episode is one scenario played once against the candidate, until the stopping criteria are met or the turn limit is reached.
• A trajectory is the record of an episode: every message and tool call, in order. This is what the judge scores.
• A simulation run is a group of episodes: every scenario in a scenario set, played against the candidate as many times as you ask.

Here’s what the flow looks like:

Example: Testing a Python upgrade recommendation

Our agent is a coding agent on Harness Runtime, working on a small Python service called orders-api that runs on Python 3.11. The developer wants to upgrade, but does not know which version to pick or what will break. The agent can fetch web pages and search the repository.

We’re using an agent because the example needs tools, but you can also test the inference calls your application makes. Point the simulation at a model, Dedicated Inference deployment, or Inference Router, and it creates an agent configuration that wraps the model and your model access key. The user simulator talks to it the same way, the judge scores it with the same metrics, and the configuration stays on your team so you can reuse it in later runs.

A good conversation about this takes three turns, and each requires findings from the turn before it:

1. The user asks which version to upgrade to: The agent fetches the Python release status page, returns the supported versions and how long each is supported, and asks the user what matters to them.
2. The user picks one of those versions and asks what will break: The agent fetches the release notes for every version between 3.11 and the one the user picked, not just the target, and lists what was removed.
3. The user asks whether their code is affected: The agent searches the repository for the modules removed from that list and suggests a replacement for each one it finds.

The user can only pick from the versions the agent found. The agent can only list the right changes if it remembers both where the user started and what they chose. And it can only check the code against the list it built.

In other words:

• Skip the fetch in turn 1, and the options may be out of date.
• Forget 3.11 in turn 2, and the list misses everything removed in the releases in between.

That is the shape of failure Simulations is designed to catch.

Scenarios: where realism comes from

A scenario is one JSON object in a scenario set, stored as JSONL (one scenario per line).

There are three ways to create scenarios:

• Use Scenario Builder: Describe what you want to test in a sentence or two, and a generator model writes a set of scenarios for you. Step 3 of the forthcoming doctl walkthrough later in this article shows the same thing from the command line.
• Create them from production traces with Signals: When Signals flags a problem in a Harness Runtime agent’s conversations, export those conversations and DigitalOcean turns them into a scenario set you can replay.
• Write them by hand and upload the JSONL file.

Here’s our scenario, formatted for readability. In the actual file, it’s a single line:

{
  "name": "Pick a Python upgrade target for orders-api",
  "description": "A developer maintains orders-api, a Python service on Python 3.11, and wants to upgrade before 3.11 stops getting security fixes. They are not sure which version to move to or what will break.",
  "user_persona": "A backend developer who prefers a release that has been out for a while over the newest one. Knows the codebase well but has not followed recent Python releases. Asks short, direct questions.",
  "stopping_criteria": [
    "The agent fetches current Python release information and lets the user choose from the supported versions",
    "The agent lists breaking changes from every release between 3.11 and the chosen version, not only the chosen one",
    "The agent searches the repository for the removed modules and suggests a replacement for each affected use"
  ],
  "max_turns": 6
}

Note that only description is required - everything else is optional.

These optional fields do the most to shape how a simulation plays out and how it’s scored:

stopping_criteria: says what the agent should get done Each criterion is something the agent does: “The agent fetches…”, “The agent lists…”. The user simulator keeps the conversation going until the agent has met every criteria, then ends it. If you leave stopping_criteria empty, the simulator decides when to end the conversation based on the description. If it never decides the conversation is done, the conversation stops at max_turns. An agent that never meets the criteria runs into that limit, which is a signal in itself.

expected_outcome: only needed for Agent Policy Adherence This metric checks whether the agent followed the policy you write in expected_outcome, such as a rule it must follow or an end state it must reach. Our scenario doesn’t use the metric, so we leave the field out. The other conversational metrics, including User goal completion, don’t need it. They score the conversation as it happened.

The user simulator determines what to respond to in every turn: the description, the persona, and what the agent actually returns. That matters here because the options returned by the agent are derived from a live web fetch. The scenario has no way to know in advance which versions the agent will find, so it cannot script the user’s choice. For the same reason, the stopping criteria are written relative to the chosen version instead of naming one.

Note that once a simulation run uses a set, the set locks. This keeps every past run readable. When you review a run’s episodes, the scenario you see is exactly the one the user simulator played, and results stay comparable over time.

To change a set that a run has used, duplicate it and edit the copy. Versioning for scenario sets is planned, so you will be able to edit a set in place while each run stays tied to the version it played.

User simulation: the nuances

The user simulator is a model that plays the user, one turn at a time. During each turn, it reads the scenario and the conversation so far, writes the next message, and decides whether to stop.

Here’s what that looks like in practice:

• It reacts to what the agent returns. During turn 1, the agent’s web fetch decides which Python versions the user can choose from, and the simulator picks one based on the persona. If the agent lists the wrong versions, the user picks from the wrong list, just as a real user would.
• It only knows what the persona knows. Our user knows their codebase, not the Python changelog, so they will not say “3.12 removed distutils” and do the agent’s job for it.
• It stops when the work is done. The conversation ends when the agent meets the stopping criteria, or at the turn limit. An agent that answers from memory and has to be corrected uses up turns, and may never get to the code check.

Playing a scenario more than once shows whether the agent really follows the user. One good conversation can be luck: a fixed answer that happens to match what the user picked.

Episodes per scenario (exploration_budget in the API, up to 5) sets how many times a scenario is played, and each episode can take a different path. Play our scenario three times, and the user might pick version 3.14 twice (once after the first ask about the newest release), and version 3.13 once.

The right list of changes is different each time, so a simulation where the agent that passes every episode provides strong evidence that it is following the user’s choice instead of giving a fixed answer.

Execution: running the agent for real

Each episode runs your agent in a dedicated Harness Runtime sandbox, with the same configuration and Action Gateway tools it uses in production.

Here’s what that means in practice:

• It is the real harness. Access Claude Code, Codex, or OpenCode in this preview - the behavior you see closely tracks the behavior you would ship.
• It works with a managed agent or a model. Point the simulation at a managed agent, or give it a model, Dedicated Inference deployment, or Inference Router, and it wraps that in an agent for you.
• Tool calls are live, not mocked. Our scenario example needs a tool: the web fetch reads the real Python docs, and the repository search runs inside the sandbox.
• You can watch it live. An episode goes from Preparing to Running, and ends Finished or Failed. The trajectory is written after every exchange, so you can follow along with each turn.
• The trajectory records everything in each episode: Every message, the turn index, timestamps, token usage, and each tool call with its input, output, and whether it succeeded.

Validation: scoring the whole conversation

When every episode has finished, the run moves to Evaluating and a judge model scores each trajectory - every message and tool call - against the metrics you picked. Only conversational (multi-turn) metrics can be used in a simulation.

These built-in metrics carry most of the signal:

• User goal completion: Did the user get the changes their code needs? An answer that only covers Python version 3.14 does not pass, because the user started on 3.11.
• Knowledge Retention: Does the agent remember that the user started on 3.11 and chose 3.14, and search the code for the modules on its own list?
• Tool Use Accuracy: Did the agent fetch instead of answering from memory, and call each tool with the right arguments?
• Conversation Completeness: Did the agent answer all three questions: which version, what breaks, and whether the code is affected?

Beyond those, Turn Relevancy flags agent responses that drift off topic, and Agent Policy Adherence checks the agent against the scenario’s expected_outcome. Agent Policy Adherence is the only metric that needs that field.

Custom metrics fill the gaps. An upgrade plan that skips a release looks complete, but it misses breaking changes the built-in metrics may not catch. For example, a custom metric could pass the agent only if it covers every release between the current and chosen version, then checks the code for what changed.

The star metric is the number you care about most. Pick one metric and a success threshold from 0 to 100. We use User goal completion.

Low scores point to different fixes, depending on where the agent went wrong:

• No fetch at all is an instruction problem.
• Fetching the wrong pages or searching for the wrong names is a tool-use problem.
• Reading the right notes but leaving modules off the list is a reasoning problem.
• Listing only the 3.14 changes is a retention problem.

Any of these problems can end with a confident answer, and only the trajectory shows the difference.

A note on models: Besides your agent, a simulation uses a user simulator, a judge, and, when you generate scenarios, a generator model. Each has a default, and you can change any of them: generator-model-uuid on scenario-set generate, --judge-model-uuid on simulation-run create, and user_simulator_model_uuid in the API. Keep the simulator and judge the same across runs you plan to compare.

Running Evaluations with doctl

Everything described earlier works from the command line with doctl 1.182.0 or later and a token with access to Evaluations. Use the same flow in the Control Panel, under Data & Learning > Evaluations.

Here’s how to run Evaluations:

1. Find the agents. We compare two configurations of the same agent.

The baseline’s instructions say “answer Python questions concisely.”

The candidates add: “Look up current release information with web fetch instead of answering from memory. When there are options, present them and let the user choose. For upgrades, cover every release between the current and target version, and check the user’s code before giving a verdict.”

doctl harness-runtime config list

2. Pick the metrics. Use the UUIDs of the conversational metrics you want to measure:

doctl evaluation metric list --format UUID,Name,Category

3. Create the scenario set. Upload the scenario:

doctl evaluation scenario-set create \
  --name python-upgrade \
  --file ./scenarios.jsonl

Or give a goal description and let the generator model write the scenarios:

doctl evaluation scenario-set generate \
  --name python-upgrade-gen \
  --goal-description "A developer asks a coding agent which Python version to upgrade a service to and what will break. The agent should look up current release information, let the developer choose from the supported versions, cover breaking changes in every release between the current and chosen version, and check the developer's code for anything affected." \
  --num-scenarios 5

Generation runs in the background. Wait for the status to go from GENERATING to READY:

doctl evaluation scenario-set get <scenario-set-uuid> \
  --format UUID,Name,Status,ScenarioCount

Then check what you are about to test. Each description should explain the situation, and each stopping criterion should describe what the agent must accomplish:

doctl evaluation scenario-set list-scenarios <scenario-set-uuid> \
  --format Name,Description,UserPersona,MaxTurns

# Full scenarios, including stopping criteria
doctl evaluation scenario-set list-scenarios <scenario-set-uuid> -o json

4. Run the baseline and the candidate. The only difference between the two commands is the agent UUID:

doctl evaluation simulation-run create \
  --name python-upgrade-baseline \
  --scenario-set-uuid <scenario-set-uuid> \
  --agent-uuid <baseline-agent-uuid> \
  --metric-uuids <goal-completion-uuid>,<knowledge-retention-uuid>,<tool-use-accuracy-uuid>,<conversation-completeness-uuid> \
  --star-metric-uuid <goal-completion-uuid> \
  --star-metric-success-threshold 80 \
  --max-turns 6 \
  --exploration-budget 1

doctl evaluation simulation-run create \
  --name python-upgrade-candidate \
  --scenario-set-uuid <scenario-set-uuid> \
  --agent-uuid <candidate-agent-uuid> \
  --metric-uuids <goal-completion-uuid>,<knowledge-retention-uuid>,<tool-use-accuracy-uuid>,<conversation-completeness-uuid> \
  --star-metric-uuid <goal-completion-uuid> \
  --star-metric-success-threshold 80 \
  --max-turns 6 \
  --exploration-budget 1

--max-turns and --exploration-budget override the values on each scenario.

To test a model or Inference Router instead of a managed agent, use the API for now:

curl -X POST "https://api.digitalocean.com/v2/gen-ai/simulation_runs" \
  -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "python-upgrade-model",
    "scenario_set_uuid": "<scenario-set-uuid>",
    "candidate_model_uuid": "<model-uuid>",
    "candidate_model_source": "CANDIDATE_MODEL_SOURCE_SERVERLESS",
    "model_access_key": "<model-access-key>",
    "evaluation_config": { "metric_uuids": ["<goal-completion-uuid>"] }
  }'

5. Monitor the run. It moves through Pending, Running, and Evaluating, and ends Succeeded, Partially successful, Failed, or Cancelled:

doctl evaluation simulation-run get <run-uuid> \
  --format Name,Status,TotalEpisodes,EpisodesFinished

# The JSON output also carries the linked evaluation run UUID
doctl evaluation simulation-run get <run-uuid> -o json

6. Read the trajectories. List the episodes, then open any trajectory to see what the agent did:

doctl evaluation simulation-run list-episodes <run-uuid> \
  --format UUID,ScenarioUUID,EpisodeIndex,Status,DurationSec

doctl evaluation simulation-run get-trajectory <run-uuid> <episode-uuid> -o json

# Download link for the full trajectory file
doctl evaluation simulation-run get-trajectory-url <run-uuid> <episode-uuid>

Each trajectory has the messages, tool calls, turn count, and per-metric scores with the judge’s reasoning. This is where you can see whether the baseline fetched anything in turn 1, and which release notes it read in turn 2.

7. Read the evaluation results. Each simulation run has a linked evaluation run, with scores grouped by scenario and then by episode:

doctl evaluation model-run get <evaluation-run-uuid> \
  --format Name,Status,OverallScore,Progress

doctl evaluation model-run get <evaluation-run-uuid> -o json

# Full results file
doctl evaluation model-run results-download-url <evaluation-run-uuid>

The table below shows an illustrative comparison of scores between the baseline and candidate runs

The differing trajectories between the baseline and candidate lead to the gap in scores between the two. The baseline fetched the release page in turn 1, but in turn 2 it read only the 3.14 notes, so it missed distutils and cgi, and the code search came back clean. It still ended with a confident answer that a final-response check would have passed. In turn 2, the candidate fetched three release notes pages and the baseline fetched one.

Simulations Pricing and limits

Simulations have no separate per-run charge. You instead pay serverless inference token rates for the user simulator and the judge, and for the generator model when you generate scenarios. The agent under test is charged as Harness Runtime usage, including its compute, storage, tools, and model inference. Prices are in US dollars, exclude taxes, are subject to change, and preview pricing may change at general availability. Agent Droplets plans apply their discount (Pro: 15%, Team: 20%) to all of these components, for DigitalOcean-hosted models.

Simulations: what’s next

• Standalone episodes: Today, episodes come from a simulation run. We are working toward running a single episode on its own, any one scenario against any one agent or model, scored the same way. You’ll be able to use it to change one thing and re-test without starting a new run.
• Scenario versioning: Edit a scenario set in place and keep each run tied to the version number, instead of duplicating locked sets.
• The next article in this series will cover where scenarios come from: This includes Scenario Builder and scenario generation from production conversations flagged by Signals.

The above reflects our current plans and product direction, and is subject to change without notice. It is provided for informational purposes only and is not a commitment to deliver any material, feature, or functionality.

DigitalOcean Simulations is available today in Public Preview to customers in supported regions, subject to export control and sanctions laws. Preview features have no service level agreement, are not recommended for production workloads, and may change or be withdrawn; see the Preview Terms. Start with the Simulations documentation and the doctl reference for evaluation scenario-set and evaluation simulation-run.

Claude Code, Codex and OpenCode are trademarks of their respective owners, used here only to identify the products supported in the preview; no affiliation or endorsement is implied.

This post contains forward-looking statements under the Private Securities Litigation Reform Act of 1995. Actual results may differ materially; see DigitalOcean’s most recent Form 10-K and subsequent Forms 10-Q. DigitalOcean undertakes no obligation to update these statements.