A marketing team asks an agent to compare a discount offer with the same push notification sent without a coupon, split by members and non-members. The warehouse is clean, and the contract table shows which membership contracts are still open, so the agent uses it. But a member whose card is declined loses member benefits after a short grace period, while the contract stays open until it is canceled weeks later. The comparison therefore classifies customers who no longer receive member pricing as members. This task comes from Argo-Bench, and many of the runs that failed it made exactly this choice.
Two recent agent benchmarks gave agents consistent simulated data with the needed evidence available, under different controls. Agents still struggled to find the evidence a task required or to interpret the available records correctly, and some picked the wrong record, unit, or objective. Consistent data still matters. These studies show it is not the same as data prepared for a specific task.
Key takeaways
- Two recent preprints tested AI agents in simulated companies under different controls: Argo-Bench kept its warehouse free of drift and inconsistencies on purpose, and WorkWorlds compared agents given all of a role’s files with agents given curated files.
- Agents struggled to find the evidence a task required or to interpret the available records correctly, for example by using the wrong record, unit, or objective.
- Passing checks for completeness, valid values, and structural consistency does not establish that an agent has selected the right records or interpreted them correctly.
- Name the record, unit, and objective each task depends on, evaluate the agent on a labeled sample from your own data under fixed conditions, and treat public benchmark results as background.
What the two AI agent benchmarks tested
WorkWorlds (September 2026) builds one simulated pharmaceutical company and gives agents tasks from six job roles. It compares two conditions. In one, the agent sees everything a role could open, including unrelated files. In the other, a practitioner who read the task picks the relevant files first. The source data, tasks, agents, and grading stay the same. Agents scored higher with the curated files, but once they reached enough evidence, the two conditions performed almost alike. Most of the gap opened before the agent found the right material. The authors, affiliated with Harvard, Stanford, an agent evaluation company, and a startup, argue that task-specific curation does part of the work a benchmark claims to measure.
Argo-Bench (October 2026) simulates a year of a food delivery platform in one city and exports it as an ERP-style warehouse. The authors deliberately excluded drift and inconsistencies to keep the ground truth unambiguous. Agents submit actions, such as suspending fraudulent accounts or paying back wages, instead of answering a query, and the simulator’s true state stays hidden. The code repository belongs to a company that builds data agents. The authors state that the benchmark compares agents with one another and does not estimate performance on a real company’s warehouse.
Why AI agents still fail: wrong record, unit, objective, or evidence
Argo-Bench names its failure types, and several share a pattern: the method was reasonable, but the agent selected the wrong evidence or optimized for the wrong objective. WorkWorlds adds one step earlier, where the agent never reaches the right material. The table interprets both studies through a data-preparation lens.
| Failure type | What happened in the benchmark | What basic data and access checks can establish | What a task check asks |
|---|---|---|---|
| Wrong record | Membership taken from contract dates instead of billing history and orders where benefits applied | The contract table is complete, with valid keys and no missing values | Which record defines a member for this comparison? |
| Wrong unit | The agent counted retained location reports without accounting for idle check-ins sent every half hour but stored only hourly | Timestamps are valid and no hourly rows are missing | What does one row represent, and does the task need the app’s own count? |
| Wrong objective | A bonus budget cut planned to protect courier hours, while the bonuses substituted for surge pay | Cost columns are present and reconcile | What is this budget for, and what should the action optimize? |
| Missed evidence | The agent did not reach the file that held the needed evidence among everything the role could open | Files are accessible and permissions are correct | Which sources does this task depend on, and did the agent use them? |
- 01Task
Compare a coupon offer with the same notification without one, by members and non-members.
- 02Pick
The agent defines members from the contract table, which is complete and clean.
- 03Gap
Members lose benefits after a grace period following a declined payment, while their contracts remain open.
- 04Result
The comparison counts customers without member pricing as members.
In the budget task, some runs optimized for courier hours rather than the surge-pay savings the bonuses were intended to deliver. The authors report higher scores when the prompt stated what the bonuses were for and that a holdout group existed. Because the prompt changes bundled several hints together, the experiment does not isolate the effect of stating the objective alone. Model choice and agent settings also changed outcomes: in Argo-Bench, some runs caught these problems and others did not. So the question is not data instead of models. It is which record, unit, and objective a task depends on, checked alongside the agent that will use them.
What these studies do not show
- How agents will do on your data. Both are preprints that have not been peer reviewed, and each uses one simulated company or city. Argo-Bench’s authors say directly that it does not estimate performance on a real warehouse.
- That real data behaves the same way. Real warehouses can contain drift and inconsistencies. Argo-Bench deliberately excludes these, and neither study establishes how they would affect performance in your environment.
- That failures are caused by data alone. In Argo-Bench, prompt wording and model settings changed results too.
How to check your data before an agent acts on it
These checks come from the failure types above and do not need a new tool.
- Name the record that defines each business term for this task. For a member comparison, that may be billing history and orders where a benefit applied, not contract status.
- Write down what one row means in every table the task counts from. An hourly summary row and an event row produce different totals for the same activity.
- State the objective the action must serve inside the task. If a budget exists to replace another cost, say so in the task.
- Evaluate the agent on a labeled sample from your own data. Fix the model, prompt, and evaluation criteria, and record the data version you used. Use public benchmark results as background, not as a directly comparable score.
Our articles on task-specific data preparation and semantic layers explore the same distinction: passing general data quality checks does not establish readiness for a particular AI task.
Related product: Syntitan
For structured data, CUBIG’s Syntitan supports use-case-specific preparation and validation. Teams define the use case, prepare field meanings, units, and relationships for it, validate the data with a selected model or agent under fixed evaluation conditions, and release it as a data version with a Qualified, Not Qualified, or Inconclusive result for the task. Explore Syntitan.
