Skip to main content
An eval set replays conversations with simulated users and scores each one. Use it as the gate between a change and a release. For the screens and settings, see Evaluations.

Use eval sets as a release gate

Eval sets live at Project → Evaluate → Evals.
  • Studio has no Compare screen. The platform_eval_runs compare operation reports evaluator-level aggregates; read the individual run cases for scenario-level results.
  • Runs need compatible, complete snapshots to compare. Keep scenarios, evaluators, personas, and variation counts stable for regression runs.
  • A Human Review evaluator leaves a run provisional. Keep that set out of automated baselines.
  • Put scenarios that write real rows in a separate set with one variation, and reset the rows before each run.

Add exact checks to scenarios

Judges score what can’t be checked exactly. Set exact checks for everything that must be exact. A Trajectory evaluator on a scenario with none of these checks has nothing to score.
  • Variations default to 3. Keep 3 on read-only sets, and read the spread per scenario. Use one variation for sets that write.
  • Set Judge Model explicitly, and prefer a model other than the agent’s.
  • After you change an evaluator, re-judge a finished run with runMode: "evaluator_only" and sourceRunId.
  • platform_eval_runs compare doesn’t check the execution target or the model. Record both with every run.

Compare runs with a baseline

A baseline records the starting point for a stable eval set. Use it to ask whether a change regressed measured behavior. It doesn’t replace reading the conversations or checking a policy-sensitive path.
1

Keep a core set

Keep scenarios, personas, evaluators, and the variation count stable. Record the execution target and model.
2

Add the failure scenario

Add the new scenario that captures the failure you want to prevent.
3

Run the unchanged target

Before you change the agent, run the full set against the unchanged target. Confirm the new scenario shows the known failure and the existing cases behave as expected.
4

Set the baseline

Set the set’s baselineRunId to this complete run, through platform_eval_sets or the API. Studio has no button that marks a run as the baseline.
5

Change and rerun

Make one bounded change, then rerun the same full set against the same target and model.
6

Review the results

Review regression flags and evaluator-level compare results. Open the changed cases, and read their conversations, tool results, and traces.
Comparison can be refused when results are incomplete or the scenario, evaluator, or variation snapshots differ. Reasons include run_results_incomplete, scenario_snapshot_changed, evaluator_snapshot_changed, and variant_count_changed. Resolve the mismatch, or create a new baseline after you change the set on purpose.

Use personas that exercise different paths

A persona describes how a simulated user communicates and what they want. A scenario supplies the opening request, fixed booking or account data, expected outcome, and exact checks. Keep personal or case-specific data in the scenario fixtures, not in a reusable persona. Start with one persona per communication pattern that changes a path. Evals → Library → Personas includes adversarial templates, and Arch or platform_eval_personas can generate drafts. Review generated goals and constraints before you use them in a release set.

Check the judges against human review

An LLM Judge is useful only when its rule is clear and its examples agree with how your team applies the rule. Calibration is a repeatable review process; the platform doesn’t calculate human-judge agreement for you.
  1. Write the evaluator as observable pass and fail criteria. Include one example at the boundary.
  2. Hand-score 10 to 20 conversations from a completed run. Record the case ID and a short reason for each decision.
  3. Use Re-run evaluators only, or evaluator-only mode, on the same source run.
  4. Count disagreements and inspect the evidence. Separate an unclear rubric from a missing conversation trace or a judge mistake.
  5. Revise the prompt, examples, or exact checks, then repeat the sample.
  6. Keep a judge out of the release gate until the remaining disagreements are uncommon and explainable.
Choose the judge model explicitly, preferably a different model from the agent’s. Set temperature to zero where available, use Pass/Fail for binary rules, and ask the judge to cite the conversation evidence. Read cases marked Needs human review. A judge’s rationale is supporting evidence; it doesn’t replace a reviewer’s decision on a high-impact outcome.

Run evaluations in CI

Use a CI-only project as the integration target. Store API credentials in the CI secret store. The pull-request gate uses read-only tools and fixtures; keep real writes in a separate set with controlled test rows and a reset plan. The sequence below shows the CLI calls. Adapt the JSON checks to your runner. Require the validation diagnostics to be valid: lifecycle validate can exit successfully while its JSON still reports package errors. Preview the sync first, and apply it to the CI project only after you review the returned digest and issues.
CI validation and evaluation
Set the pass-rate and score thresholds on purpose, and keep them aligned with the suite’s Passing score and regression settings. A failed eval gate returns exit code 2. The run count is scenarios × personas × variations: 11 scenarios × 3 personas × 1 variation is 33 conversations. See Use the Artemis CLI. The CLI has no compare command. The CI/CD API can start a release eval with an optional baseline, return per-scenario results, compare release scores, and report gate status. The release-compare endpoint returns the overall score delta. Inspect the run’s scenario breakdown to see which cases passed. Run a broader set nightly or before release.