Gather journey evidence
Write scenarios
Each scenario describes a starting situation and a result you can check. Write at least one ordinary case, one boundary case, and one failure or recovery case for every important journey.
Write a short sample dialogue before you build. It shows missing decisions: when the identity check happens, what a yes confirms, which facts come from the system, and what the agent says after a failed tool call.
Plan journeys that use files or media
For a journey that involves a document, image, or other uploaded file, include the file in the scenario. Decide which file types the channel accepts, what the agent extracts, whether the content is sensitive, and what to do when a file is unsupported or still processing. Agents can use the built-inlist_attachments and get_attachment tools to find a session file and read its extracted text. Project attachment settings control extraction. content and imageDescription can be null while the file processes or when extraction is off. A missing extraction isn’t evidence that the file holds nothing relevant.
Ask for confirmation before the agent sends a file to an external service or hands it to a person. Use an approved path for file retention and access. Test a successful extraction and an unavailable or unsupported file.
Set measurable targets
A targets sheet is a short agreement between the people who own the journey and the people who build it. Keep one row per top request, with an observable result and a scenario that proves it.
Set a numeric limit only when the owner can explain the measurement and the trade-off. Measure from a defined start and end point, with the same method in every comparison. Give a rate a numerator and a denominator, and define what counts as an eligible case. For each target, record the owner, the measurement method, and the response when the target is missed.
Budget each turn
Write a target for model calls and input tokens per turn for each top request. Check each design choice against it.
See Tuning for latency for the levers that reduce these costs.
Sort scenarios by risk tier
Sort scenarios by what the agent can change. Each tier needs different controls and different test data.A confirmation question isn’t authorization by itself. The system that performs the action still checks identity, permission, current record state, eligibility, and duplicate requests.
Plan the test set with the journey
For each rule, create at least one case that passes and one case that would expose a violation. Make every evaluator check something visible in the transcript, trace, tool result, or resulting record.- Cover neighboring intents that could route incorrectly.
- Include vague wording, interruptions, missing fields, and corrections.
- Test unavailable tools and empty or conflicting data.
- Keep read-only scenarios in one eval set and write scenarios in another. Test chats and evals change real records, so reset the rows after each write run.
- Use a small, stable baseline for routine regression, and a broader set before release.
Build a scenario portfolio
A scenario portfolio covers the ways a person can enter, continue, correct, and leave a journey. The same journey can sound different on web, chat, and voice, so keep channel-specific input where it affects behavior.Next steps
- Agent design guidelines: choose the structure that meets these targets.
- Evaluate agents: turn the scenarios into eval sets.