A support agent retrieves the return policy, cites the seven-day window and approves a refund. The same policy required a safety review for defective appliances, and the agent skipped it.
Agentic RAG is a retrieval-augmented generation setup in which an AI agent decides what to retrieve and plans its next steps from the results. In many enterprise deployments, the same agent also calls tools that change business systems. Retrieval makes the right policy available to the agent; it does not, by itself, make the agent follow that policy’s conditions, order and approvals.
The example above is hypothetical, but a closely related gap has been measured. When agents in the SOPBench study were given ground-truth constraint verdicts, 12 to 38% of cases still failed. In the CUBIG Tech Blog essay this article adapts, Jwkweon argues that this distinction matters whenever an agent is built around SOP documents: an agent can verify a rule and still fail to act under it.
Key takeaways
- Retrieval success is not procedural success. A cited source does not prove that every condition was checked.
- SOPs written for people hold conditions, exceptions, dependencies and approvals that a vector search does not turn into enforced steps.
- Separate retrieval, decision and execution, and enforce high-impact rules outside the model.
- Test what the agent did, not only what it said: outcome, procedure and consistency across repeated runs.
- Readiness depends on the task and the authority the agent has. Record which policy version and test cases a result was checked against.
What does agentic RAG change compared with standard RAG?
Once an agentic RAG system can call tools that change business systems, the job changes from describing a policy to acting under it. Retrieval-augmented generation, named in a 2020 paper by Lewis and colleagues, combines a model’s trained knowledge with retrieved documents. In a typical enterprise setup, an application pulls relevant material from a knowledge base and gives it to the model as context, so answers can draw on internal documents and cite sources.
Two requests show the difference:
- “What is our refund policy?” needs a faithful description of the policy.
- “Process this customer’s refund.” needs a decision that satisfies the policy for this customer, followed by an authorized sequence of tool calls.
The same retrieved passage can support both requests. An answer can be grounded in that passage and still be procedurally wrong, because a policy may permit an action only when conditions hold that depend on customer history, product type, region, earlier approvals or the state of another system.
For a question-answering assistant, a miss produces a misleading reply. For an agent connected to business systems, the same miss can produce an unauthorized transaction, a skipped approval or an irreversible change. RAG vs. fine-tuning covers the earlier choice of how a model gets company knowledge at all.
Why can an agent retrieve the right policy and still take the wrong action?
An agent can retrieve the right policy and still act wrongly because finding one relevant passage is not the same as checking every condition that applies. Research on tool-using agents has started to measure that gap.
The τ-bench paper by Yao and colleagues (2024) tested agents that talk with simulated users and call domain APIs under policy guidelines. The policy sits in the agent’s system prompt, so the benchmark tests what happens after the right policy is already in hand. Its authors report that even state-of-the-art function-calling agents, such as GPT-4o, succeeded on fewer than half of the tasks and were inconsistent: pass^8, the rate of succeeding on all of eight repeated trials, was below 25% in the retail domain.
The τ-bench figures describe the models tested in 2024, not today’s production agents; the lasting point is that completing a task once differs from completing it reliably under constraints.
SOPBench, published at EMNLP 2026, tests procedural compliance directly. The benchmark covers seven customer-service domains with 70 service-specific SOPs, 167 executable tools and 830 validated test cases, and its code-based verifiers check compliance at the outcome, step and trajectory levels without human or LLM judges. Reasoning models performed best but remained far from reliable. In a layered ablation, agents given ground-truth constraint verdicts still failed 12 to 38% of cases, which the authors read as a sign that the challenge lies not only in retrieving or verifying constraints but also in consistently turning verified constraints into compliant actions.
SOPBench does not set a failure rate for enterprise agents. Its results do suggest that verifying a rule and acting under it are separable capabilities, and an agentic RAG design should test them separately.
What does an SOP contain that retrieval alone does not enforce?
Many enterprise SOPs are written for people, and the knowledge a person brings to them does not travel with the text into a vector database. An employee knows that an instruction applies only to some customers, that an exception overrides the general rule, or that another department must approve first. Some of that is written down, some is spread across documents, and some exists only in the organization.
A typical SOP mixes several kinds of information:
| Element | Question it answers | Refund example |
|---|---|---|
| Conditions | When does a rule apply? | Within seven days of delivery |
| Exceptions | What overrides the general rule? | Defective electrical appliances |
| Dependencies | Which steps must come first? | Safety review before any refund |
| Authority | Who may decide or act? | Only an authorized reviewer approves |
| Evidence | What must be checked before acting? | Order ownership, delivery date, product category |
| State transitions | How does one result change what is allowed next? | A pending review blocks the refund |
A retrieval system can return every relevant instruction and still give no guarantee that the agent runs them in order, separates mandatory conditions from guidance, or sees that a review step can stop the refund step.
Overlapping documents make the problem harder: a general returns policy, a product-specific exclusion, a newer regional addendum and an internal approval matrix can each be correct while only some of them apply together. A larger context window holds more of this text, but it does not settle which document takes precedence. Why AI misses business rules and exceptions looks at where those rules live before any agent is built.
How do you separate retrieval, decision and execution?
A practical agentic RAG design splits one model call into three stages: retrieval finds the evidence, decision checks the applicable rules against the current situation, and execution acts only after the required checks pass. The split is conceptual and does not require a fully deterministic rule engine, but it does require the system to keep evidence, interpretation and authorization apart.
For the appliance refund, an illustrative decision record might read:
- Request: refund for an opened, overheating appliance
- Known facts: delivered four days ago, used once, defect reported
- Applicable rule: defect-related requests require a safety review
- Required next step: create a safety-review case
- Restricted action: issuing the refund before the review is complete
- Policy reference: return policy, version 3, sections 2.1 and 4.3
A record like this makes the decision inspectable: a reviewer can ask whether the right policy was chosen, the customer facts were collected, the exception was applied and the next action was permitted. A trace of retrieved chunks and a final message shows what the model saw, not which conditions were checked before a system changed.
For high-impact actions, enforcement belongs outside the model’s reasoning. A model may propose a refund, while the application requires a completed eligibility check and a valid approval before it exposes or runs the refund operation. An LLM’s statement that a condition is met is not independent verification of it.
The OWASP entry on excessive agency makes a related recommendation: require a human to approve high-impact actions, a control that can sit in a downstream system outside the LLM application or in the LLM extension itself. Converting policies into structured rules carries its own risk, since an extraction can drop a qualifying condition or turn guidance into a hard rule. The value of structure is that assumptions become visible, reviewable, versioned and testable, so policy owners can approve the interpretation.
Where do agentic RAG failures happen?
Agentic RAG failures can be traced to different points in the pipeline, and locating the point narrows what to fix. Suppose an agent retrieves the correct policy, identifies the exception and states that a safety review is required, then calls the refund API anyway. The failure did not happen in retrieval; it happened when the understood constraint failed to control the next action.
The CUBIG Tech Blog essay this article adapts proposes the following engineering taxonomy. It is a working classification, not one reported by a benchmark.
| Failure point | What went wrong | Possible fix |
|---|---|---|
| Evidence | The relevant policy or customer fact was unavailable | Indexing, document parsing, policy coverage |
| Interpretation | The evidence was there, but the rule was misread | Clearer representations, examples, task-specific model evaluation |
| Planning | The right rule was found, but the planned steps broke it | Explicit dependencies and intermediate checks |
| Execution | The restriction was not enforced at the tool boundary | Permissions, transaction guards, validation outside the model |
| Verification | The system could not establish whether the action complied | Action logs and policy-aware test oracles |
Calling all five “the model hallucinated” hides what needs to change. Treating every failure as a retrieval problem can lead teams to add documents or expand context when the problem sits between decision and execution.
How should you test an agentic RAG system before it acts?
Testing an agentic RAG system means checking what the agent did, what it was allowed to do and what the record can prove, not only the final reply. Two agents can send the customer the same reassuring message while only one created the required review case; an answer-quality score could rate both well.
Three questions keep the dimensions apart:
- Outcome correctness: did the system reach the expected final state?
- Procedural compliance: did it meet the mandatory conditions in the authorized order?
- Execution reliability: does it behave the same way when the same case runs again?
Test cases should be built to separate those possibilities. For the refund policy, a set might include an unused eligible item, a used item with a reported defect, a missing purchase verification, a refund that needs approval, conflicting policy versions, and a customer who asks the agent to skip a check. The last case is a trust-boundary problem: “the manager already approved it” is a claim, and the agent needs to tell it apart from an approval recorded in the business system.
Some failures can be caught with executable assertions, such as a refund API that must never be called without a valid approval. Others need domain judgment, such as whether a customer’s description counts as a safety complaint. LLM judges can help classify or summarize traces, but their verdicts should not be the only authority on safety-critical or objective policy constraints. The agent evaluation entry in CUBIG’s glossary covers the wider practice.
Where does Syntitan’s Ready for Agent fit?
Syntitan’s Ready for Agent turns work materials into test cases for an agent a team has already built, so the team can test the agent against situations the manual does not spell out before putting it to work. The team names the agent and its main task and uploads materials such as a work manual or the agent’s MD file. Syntitan then lays out the steps, decision conditions, exceptions and output format for each task.
Where the analysis is wrong or incomplete, the team adds rules one per line in an if-then form and runs the analysis again; each re-analysis creates a new analysis version, and cases made earlier should be regenerated against it.
Ready for Agent then generates test cases that extend beyond the situations the materials describe, and it shows side by side how many cases come from the manual as written, from a general LLM and from Syntitan. The count measures how much test coverage there is to review, not how good the agent is. The cases download as a zip file with the inputs and the answer criteria kept apart.
Ready for Agent does not design, build, run or fix the agent. The team runs the cases in its own environment, whatever framework the agent uses. Running the cases against the agent and comparing its results, and connecting through an SDK, are in preparation. Teams that already trace and score agents with evaluation tools can keep them; Ready for Agent supplies the cases, including exceptions, derived from the business materials the agent is supposed to follow.
What should a team check before an agent takes consequential actions?
Before an agentic RAG system is allowed to change records, issue payments or approve requests, a team should be able to answer these questions from evidence.
- Which policy version governed the decision?
- Which facts were verified, and which were only asserted by the user?
- Which required checks were completed?
- Which actions were permitted, blocked or escalated?
- Which test cases show compliance, including the exceptions?
- Who owns the rules, and who updates the tests when the rules change?
The same SOP can be enough for an agent that answers refund questions and not enough for one that issues refunds; readiness depends on the task and the authority the agent holds. A policy change, a new exception or an extra approval step can make an earlier test result a weaker statement about the system now running, so the policy version and test conditions belong with the evidence. A citation shows where a statement came from; it does not show that an action was authorized.
Try it on your data in Syntitan, or start Ready for Agent with its sample manual before using your own.
