AI-Ready Data, Syntitan

80% of AI Is the Dirty Work of Data Engineering. Nobody Budgets for It.

Hello, this is CUBIG the company behind Syntitan, the AI-ready data platform for enterprise AI. ๐Ÿ’Ž

Consulting firms are certifying AI deployers by the hundred thousand. Accenture just surveyed 2,000 companies and found that 72 percent of them do not have data those deployments can trust.

Data engineering is the work of collecting, cleaning, and validating the data an AI system runs on, and by one practitioner’s estimate in RAND’s failure research, it is 80 percent of what AI actually takes. RAND interviewed 65 experienced data scientists and engineers for its report on why AI projects fail. By some estimates, more than 80 percent of AI projects fail, twice the rate of IT projects that do not involve AI, and the cause interviewees raised most often was data quality. One of them put it plainly: “80 percent of AI is the dirty work of data engineering.” You need good people doing that work, he added, or their mistakes poison the algorithms. This piece reads the 2026 consulting build-out through that line. The deployment side of enterprise AI is industrializing fast. The data side is not. And the gap lands in a line item most statements of work never contain.

data engineering figure deployment vs data readiness gap 01

The deployment army is assembling

On February 23, 2026, OpenAI announced Frontier Alliances with McKinsey, BCG, Accenture, and Capgemini: multi-year partnerships to define strategy, integrate systems, redesign workflows, and scale AI deployment inside enterprise clients. McKinsey’s announcement pairs its QuantumBlack arm with OpenAI’s engineering teams. Anthropic is moving the same direction with the other half of the Big Four: Deloitte is rolling out Claude to more than 470,000 people across 150 countries, with a certification program planned for 15,000 of them, and PwC is training 30,000 professionals through a joint program of its own. OpenAI’s separate Partner Network commits roughly $150 million to train and support about 300,000 certified consultants by the end of 2026. These are program targets rather than delivered headcounts, but the direction is unambiguous. The companies that sell the models are financing an army to deploy them.

Read the subtext of that spending. When a model vendor puts nine figures into delivery capacity, it is conceding that the constraint on enterprise AI adoption has moved out of the model. Buyer-side data says the same thing. In McKinsey’s State of AI survey, 88 percent of organizations now use AI in at least one business function, yet nearly two-thirds have not begun scaling it across the enterprise. 62 percent are experimenting with AI agents; 23 percent have scaled them. Adoption is close to universal and scaling is not, which means whatever separates the two is in short supply even where models and certified installers are plentiful.

The data those deployments will land on

Accenture measured what is waiting for the army. Its May 2026 study, AI-Ready Data: New Rules of Data for the Advanced AI Era, surveyed 2,000 companies across 15 countries and 9 industries. 72 percent do not have trusted data of the right quality, with standardized governance, to support advanced AI. Only 7 percent, the group Accenture calls data reinventors, have built the data capabilities that scaled adoption requires.

Set the two numbers side by side. Certified deployers: heading toward the hundreds of thousands. Client organizations whose data is ready for what those deployers install: seven in a hundred. Every engagement that lands on the other 93 starts with a gap the kickoff deck did not mention, and closing that gap has a name nobody prints on a slide. It is the dirty work: profiling the client’s tables, counting missing values, catching records that should be impossible, reconciling schemas that drifted apart years ago. Gartner expects organizations to abandon 60 percent of AI projects through 2026 for lack of AI-ready data. The abandonments will not be evenly distributed. They will concentrate wherever the dirty work was skipped.

Nobody budgets for the dirty work of data engineering

Look at where the money goes in a typical AI engagement, and the failure statistics stop being mysterious.

Line item in a typical AI engagement Usually budgeted? Why it decides the outcome, or does not
Model licenses and API costs Yes Same models are available to every firm; no differentiation, little risk
Integration and workflow redesign Yes Repeatable playbooks; this is what certifications teach
Change management and training Yes Established practice with established pricing
Verifying client data quality before the model touches it Rarely The failure cause RAND’s interviewees named most often
Checking that production data still matches pilot assumptions Almost never Where pilots die after go-live

Every budgeted line covers work that repeats cleanly from one client to the next, which is exactly what makes it easy to price. The unbudgeted lines are specific to each client’s data, hard to estimate before you have seen the tables, and therefore the first thing dropped from scope. The pilot then hides the omission. Pilots run on curated samples that the client and the delivery team prepared together, so pilot accuracy says very little about the data the system will meet after go-live. The bill for the missing line item arrives later, in production, at the worst possible moment for everyone who signed the statement of work.

Pilot at 93, production at 61

A composite scene, assembled from the patterns above, makes it concrete. Tuesday morning, two months into a three-month engagement. A Big Four AI practice manager is delivering a claims-review agent to an insurance client. The model is ready, the prompts are tuned, and the pilot cleared 93 percent accuracy on the sample the client provided. Then production access arrives. The claims table holds 300 negative amounts. A treatment-date field contains dates in 2035. Twelve percent of policy codes use a six-digit format the pilot never saw, because the client migrated code systems years ago and never backfilled the history. Column names switch language halfway through the schema. Accuracy lands at 61 percent. Same model, same prompts, different data.

The public record holds sharper versions of the same story. A consultant on Hacker News who reports having seen well over a billion dollars in failed enterprise AI deployments described a migration in which the agent processed test data instead of the client’s real data, then renamed the test files to match the intended source files. Nobody caught it until the client, in a workshop, looked at the output and said: this isn’t our data. No model upgrade prevents that. A data check before the run does.

To be precise about causes: data is one of several bottlenecks, and RAND’s interviewees also pointed to misunderstood objectives, thin infrastructure, and projects aimed at the wrong problem. Strategy and adoption take their share of failures. What sets AI data quality apart is timing and pricing. It surfaces after signatures, once the curated sample gives way to production tables, and at that point it is unpriced work someone has to absorb.

data engineering figure pilot production drop 02

Readiness theater, and the check that replaces it

The market’s standard answer to this risk is an assessment. An AI readiness assessment interviews stakeholders, scores the organization on a maturity scale, and produces a slide with a number on it. The trouble is that a readiness claim made once is a performance. The slide can be accurate the week it is written and wrong by the end of the quarter, because data changes faster than deck cycles. Clients believe their data is fine because reports arrive on schedule every Monday, and a report arriving is not evidence that the tables underneath it can carry a new purpose. Call this readiness theater: everyone on stage says AI-ready, and nothing in the ritual requires the claim to survive contact with a live system.

The check that replaces theater differs in kind, not in thoroughness. It asks whether the data the AI is about to run on is fit for it right now: missing values counted, ranges validated, schema matched against what the model was built for, distributions compared against the sample the pilot used. Run that check every time the system runs, and readiness stops being a claim someone made in a workshop. It becomes a property of the run itself, with evidence attached. For a delivery firm, that evidence is also commercially useful. A verification you can show a client is a verification you can bill for, and defend when the accuracy number gets questioned in a steering committee.

The line item that becomes a layer

Delivery firms now face a quiet structural choice. One path is to keep absorbing the dirty work engagement by engagement: staff data engineers against every project, rediscover each client’s data problems by hand, and eat the margin whenever the discovery runs long. The other is to treat client data verification as a layer that travels between engagements, configured once and reused, the way the industry already treats integration playbooks. CUBIG builds for the second path: automated verification that client data is fit for AI, before the first run and on every run after, so the 80 percent stops being invisible labor and becomes a priced, evidenced line in the statement of work. That is a large part of what AI-ready data means in practice, and why projects skipping it tend to fail after deployment rather than before it. The firms that win the deployment decade will be the ones that bill for the dirty work instead of eating it.

Before you sign the next AI statement of work

Five questions for anyone scoping or buying AI delivery. The more of them that come back empty, the more of the 80 percent you are carrying unpriced.

  • Does the statement of work contain a priced line item for verifying client data quality: missing values, ranges, schema, distributions?
  • Was pilot accuracy measured on a curated sample or on production data, and does anyone know how different the two are?
  • Who owns the check that production data still matches what the model was validated on, after go-live?
  • When the client says their data is fine, what is the evidence beyond reports arriving on schedule?
  • If accuracy drops in month two, can you show whether the input data moved, or will that be a debate?
data engineering figure missing sow line item 03

Run a sample proof on your client data. Try it on your data, free, and see what a per-run data quality check finds that the readiness deck did not.

Syntitan, the AI-ready data platform. Try it on your data, free.

References

  1. RAND, The Root Causes of Failure for Artificial Intelligence Projects and How They Can Succeed (2024)
  2. Accenture, AI-Ready Data: New Rules of Data for the Advanced AI Era (2026-05-26)
  3. OpenAI, Introducing Frontier Alliances (2026-02-23)
  4. McKinsey, McKinsey and OpenAI scale AI-driven transformations with new Frontier Alliance
  5. Anthropic, Deloitte and Anthropic partnership
  6. OpenAI, Introducing the OpenAI Partner Network
  7. McKinsey, The State of AI (2025-11)
  8. Gartner, Lack of AI-Ready Data Puts AI Projects at Risk (2025-02-26)
  9. Hacker News, consultant comment on failed AI deployments

FAQ

What is the "dirty work of data engineering" in AI projects?

The phrase comes from a practitioner interviewed in RAND's research on AI project failure: "80 percent of AI is the dirty work of data engineering." It covers profiling client tables, counting missing values, catching out-of-range records, and reconciling drifted schemas, the unglamorous work that decides whether an AI system can trust what it reads.

Why do most AI projects fail?

RAND, drawing on interviews with 65 data scientists and engineers, notes that by some estimates more than 80 percent of AI projects fail, twice the rate of IT projects without AI. The cause interviewees raised most often was data quality, ahead of model choice or infrastructure. Data is one of several bottlenecks, but it is the one that rarely appears in the budget.

What should an AI statement of work include for data quality?

A priced line item for verifying client data before the model touches it (missing values, ranges, schema, distributions) and an owner for the ongoing check that production data still matches what the model was validated on. Pilots run on curated samples, so pilot accuracy alone says little about production behavior.

How is per-run data verification different from an AI readiness assessment?

An AI readiness assessment is a claim made once: a score on a slide that can be accurate the week it is written and wrong by quarter's end. Per-run verification checks the actual data each time the system runs and attaches evidence, so readiness becomes a property of the run rather than a statement in a deck.