A customer asks an agent to cancel an order before it has been dispatched. The same request may need different handling once the order is on its way.
AI agent testing examines how an agent handles requests in the situations where its work takes place. A test scenario describes the relevant circumstances and request, while the test itself checks what the agent does and whether the expected result occurs. Preparing a useful set of scenarios is a separate task from running the agent.
Adapted from AI Agent Testing: Where Do Test Scenarios Come From? by Minchan Chung, Co-Founder and Co-CEO of CUBIG, on the CUBIG Tech Blog (November 2026).
Key takeaways
- A request becomes a more useful test when it includes the situation in which the agent must act.
- Work procedure documents can provide a starting point for candidate scenarios when failure history is limited.
- CUBIG’s internal comparison evaluated candidate sets for unexpectedness, diversity and relevance; it did not run an agent.
- Human review, expected outcomes and execution checks remain necessary before drawing conclusions about agent performance.
What does AI agent testing cover?
AI agent testing covers the situation around a request, the agent’s actions and the result of those actions. A request such as “Please cancel my order” is a starting point, but the order’s dispatch status can change the appropriate handling. A scenario gives the tester enough context to decide what the agent should do in that instance.
The distinction matters when an agent can use tools or change records. An agent saying “I’ve cancelled it” is different from the order actually being cancelled in the system. A test needs an expected outcome and a way to check the relevant result, as well as the conversation itself.
The τ-bench research offers an example of testing in a tool-using setting: it simulates user conversations under domain-specific policies and compares the final database state with a goal state. Its approach also considers reliability across repeated trials. For a broader introduction to assessing an agent’s work, see agent evaluation.
CUBIG Shorts
It Worked in the Demo. Not in the Pilot.
A short take on the same point: an agent that handles the demo can still miss the situations real work brings.
Why do new agents lack test scenarios?
New agents may have little failure history, leaving teams to construct situations before production experience supplies examples. Procedure documents can describe the work the agent is supposed to perform, yet a tester still needs to turn those instructions into concrete circumstances and requests. Each candidate needs enough context for a reviewer to understand what is being tested.
A team can start by asking which facts change the handling of a request. In the cancellation example, dispatch status is such a fact. The team can then consider what the user asks, what the agent is expected to do and what evidence would show the outcome.
The starting material does not have to anticipate every possible interaction to be useful. Its immediate purpose is to give reviewers a set of cases they can discuss and select for their work. Business exception handling provides a related way to think about circumstances that depart from a routine procedure.
How can work procedure documents become test scenarios?
Work procedure documents can be used to construct candidate situations and requests for review. As Minchan Chung described in November 2026, CUBIG’s internal experiment began with five documents describing an agent’s work procedures. The process used the documents to construct situations and requests, then selected from the candidates it generated.
A reviewer can read a procedure for the action it describes and the conditions that affect that action. Those conditions can become scenario context, while a user request gives the agent something to respond to. The resulting candidate should remain connected to work the organization actually performs and tools the agent can use.
For example, an order’s status can be stated alongside a cancellation request. The reviewer can then specify the expected handling for that status and identify where to check the result. The procedure supplies the work context; the scenario makes a particular decision visible.
Behavioral testing offers a useful perspective on organizing cases. CheckList describes a task-agnostic methodology for testing NLP models, with test types including Minimum Functionality, Invariance and Directional Expectation tests. A team can use such categories to discuss what behavior a set is meant to examine, while checking that each case fits its own agent and procedure.
What did CUBIG’s internal comparison show?
CUBIG’s internal comparison found differences between candidate sets produced by the preparation process and a comparison method, as assessed by an internal evaluation model. The first comparison used a single generation step for the comparison method, while CUBIG’s process generated and reviewed candidates repeatedly. As Chung notes, the extra opportunities might have explained the better candidates in that first result.

Every figure in the following table comes from CUBIG’s internal experiment, as described by Minchan Chung in November 2026. In the second experiment, the method in the “Single generation” column was also allowed to generate and select repeatedly, using the same model and generation budget.
| Comparison | Single generation | CUBIG’s preparation process |
|---|---|---|
| First experiment: selected candidates | 80 | 80 |
| First experiment: candidates classified as unexpected | 23 | 67 |
| First experiment: relevance | 79/80 | 80/80 |
| First experiment: diversity score | 0.97 | 0.79 |
| Second experiment (comparison method also repeated, same model and budget): candidates | 886 | 926 |
| Second experiment (same conditions): diversity score | 0.4953 | 0.7879 |
In CUBIG’s internal experiment, as described by Minchan Chung in November 2026, the first comparison used the same generative model for both methods and selected 80 candidates from each. The internal evaluation model classified more candidates from CUBIG’s process as unexpected, while assigning that set a lower diversity score.
For the second comparison, both methods could generate and select repeatedly, using the same model and generation budget. In CUBIG’s internal experiment, as described by Minchan Chung in November 2026, CUBIG’s preparation process scored higher on unexpectedness, diversity and relevance under those conditions. The comparison concerns the candidate material that remained, so it should be read alongside the evaluation limits.
How should you judge an AI agent test set?
Judge a candidate test set by considering unexpectedness, diversity and relevance together. An unexpected situation can prompt a useful question, but it has little value for the intended test if it assumes work the organization does not do or tools the agent cannot use. Relevance connects a case to the agent’s actual task.
Diversity is a property of the whole candidate set. A collection can be relevant to the work yet repeat similar requests and leave other situations uncovered. Reviewers can look across the candidates for distinct circumstances, then decide which differences affect the expected handling.
CUBIG’s internal experiment, as described by Minchan Chung in November 2026, illustrates the tradeoff: CUBIG’s first set had more candidates classified as unexpected, yet its diversity score was lower than the single-generation set’s score. That result is a reason to examine the qualities together. An unusual case count alone cannot describe the set’s range.
The intended use also shapes selection. A candidate may be plausible and distinct but less useful for the immediate work than another case that covers a decision the team needs to examine. An evaluation set gives those choices a place to be recorded and reviewed.
What does the comparison not tell us about agent performance?
The comparison does not tell us how well an agent completed its work because no agent was run. The reported scores are internal model-based assessments of candidate test cases, not task success rates. They describe a preparation result under the comparison conditions.
Human review and comparisons in other work settings are still needed, as Chung states in the essay. The cost of carrying out the eventual tests and whether the observed differences could be explained by chance were not fully established. Those open questions limit how far a reader can extend the findings.
Model-based judgment also deserves scrutiny on its own terms. Research on LLM judges describes possible position, verbosity and self-enhancement biases in such evaluations. That research does not establish which biases affected CUBIG’s internal assessment, but it supports examining model judgments alongside human review.
A reviewer should therefore keep the candidate-generation question separate from the execution question. Preparing cases creates material for a test; observing an agent act and checking the outcome produce a different kind of evidence. The LLM-as-a-judge glossary explains the evaluation approach in more detail.
What should a test scenario set include before you run an agent?
A test scenario set should give reviewers enough context to select cases, define expected handling and check outcomes. The following checklist turns the essay’s preparation and verification questions into a practical review:
- A work-grounded situation: Describe the circumstances that affect the task, such as the order’s dispatch status.
- A request: State what the user asks the agent to do in that situation.
- Relevant tools and procedures: Check that the case assumes work the organization performs and tools available to the agent.
- An expected outcome: Decide what should happen before running the case, including any relevant system state.
- A verification approach: Identify how the team will check whether the action occurred, alongside the agent’s response.
- A set-level review: Examine unexpectedness, diversity and relevance together, and select cases that matter for the work.
The checklist is a starting point for discussion before any agent is assessed. A team still needs to run the selected cases, inspect results and decide what failures mean for the task.
Where does CUBIG fit in AI agent testing?
CUBIG prepares and checks enterprise data for specific AI tasks, so teams can see whether that data fits the work an agent is meant to do. AI-ready data is prepared for a particular task, its results are checked, and relevant parts can be checked again when data or conditions change. Readiness is not a one-time score.
Syntitan is CUBIG’s AI-ready data platform. Its Ready for Agent area is where teams create Tasks and Cases, look for missing situations and failure causes in execution results, and connect those findings to supplementing and re-checking.
Chung says CUBIG is developing the scenario preparation process described in the essay into software. Readers should treat that work as ongoing development, separate from the Syntitan capabilities described above.
The candidate comparison is a starting point, with human review and actual agent execution still needed before assessing performance. Try it on your data in Syntitan: prepare and check the data behind a specific AI task.
