Use eval sets as a release gate
Eval sets live at Project → Evaluate → Evals.- Studio has no Compare screen. The
platform_eval_runs compareoperation reports evaluator-level aggregates; read the individual run cases for scenario-level results. - Runs need compatible, complete snapshots to compare. Keep scenarios, evaluators, personas, and variation counts stable for regression runs.
- A Human Review evaluator leaves a run provisional. Keep that set out of automated baselines.
- Put scenarios that write real rows in a separate set with one variation, and reset the rows before each run.
Add exact checks to scenarios
Judges score what can’t be checked exactly. Set exact checks for everything that must be exact.
A Trajectory evaluator on a scenario with none of these checks has nothing to score.
- Variations default to 3. Keep 3 on read-only sets, and read the spread per scenario. Use one variation for sets that write.
- Set Judge Model explicitly, and prefer a model other than the agent’s.
- After you change an evaluator, re-judge a finished run with
runMode: "evaluator_only"andsourceRunId. platform_eval_runs comparedoesn’t check the execution target or the model. Record both with every run.
Compare runs with a baseline
A baseline records the starting point for a stable eval set. Use it to ask whether a change regressed measured behavior. It doesn’t replace reading the conversations or checking a policy-sensitive path.1
Keep a core set
Keep scenarios, personas, evaluators, and the variation count stable. Record the execution target and model.
2
Add the failure scenario
Add the new scenario that captures the failure you want to prevent.
3
Run the unchanged target
Before you change the agent, run the full set against the unchanged target. Confirm the new scenario shows the known failure and the existing cases behave as expected.
4
Set the baseline
Set the set’s
baselineRunId to this complete run, through platform_eval_sets or the API. Studio has no button that marks a run as the baseline.5
Change and rerun
Make one bounded change, then rerun the same full set against the same target and model.
6
Review the results
Review regression flags and evaluator-level compare results. Open the changed cases, and read their conversations, tool results, and traces.
Comparison can be refused when results are incomplete or the scenario, evaluator, or variation snapshots differ. Reasons include
run_results_incomplete, scenario_snapshot_changed, evaluator_snapshot_changed, and variant_count_changed. Resolve the mismatch, or create a new baseline after you change the set on purpose.
Use personas that exercise different paths
A persona describes how a simulated user communicates and what they want. A scenario supplies the opening request, fixed booking or account data, expected outcome, and exact checks. Keep personal or case-specific data in the scenario fixtures, not in a reusable persona.
Start with one persona per communication pattern that changes a path. Evals → Library → Personas includes adversarial templates, and Arch or
platform_eval_personas can generate drafts. Review generated goals and constraints before you use them in a release set.
Check the judges against human review
An LLM Judge is useful only when its rule is clear and its examples agree with how your team applies the rule. Calibration is a repeatable review process; the platform doesn’t calculate human-judge agreement for you.- Write the evaluator as observable pass and fail criteria. Include one example at the boundary.
- Hand-score 10 to 20 conversations from a completed run. Record the case ID and a short reason for each decision.
- Use Re-run evaluators only, or evaluator-only mode, on the same source run.
- Count disagreements and inspect the evidence. Separate an unclear rubric from a missing conversation trace or a judge mistake.
- Revise the prompt, examples, or exact checks, then repeat the sample.
- Keep a judge out of the release gate until the remaining disagreements are uncommon and explainable.
Run evaluations in CI
Use a CI-only project as the integration target. Store API credentials in the CI secret store. The pull-request gate uses read-only tools and fixtures; keep real writes in a separate set with controlled test rows and a reset plan. The sequence below shows the CLI calls. Adapt the JSON checks to your runner. Require the validation diagnostics to be valid:lifecycle validate can exit successfully while its JSON still reports package errors. Preview the sync first, and apply it to the CI project only after you review the returned digest and issues.
CI validation and evaluation
The release-compare endpoint returns the overall score delta. Inspect the run’s scenario breakdown to see which cases passed. Run a broader set nightly or before release.