The Diagnose step is the first stage of DTS, where the engine measures why a dataset cannot be used to train or run AI, scoring what is restricted, what is scarce, and what is structurally broken before any data readiness diagnosis leads to a rebuild.
When RAND interviewed engineers and data scientists about why AI efforts collapse, it put the failure rate above 80%, double what conventional IT projects suffer, and problems with data quality and availability ranked among the five root causes named most often. Most teams read that as a modeling problem. It is closer to a diagnosis problem: they never measured what was wrong with the data before they tried to train on it. The Diagnose step exists to catch that at the start, so the rebuild that follows is aimed at a real defect rather than a guess.
What data readiness diagnosis actually measures
DTS is CUBIG’s AI-ready data transformation engine, and it works in three moves: Diagnose, Transform, Rebuild. The Diagnose step comes first because you cannot rebuild what you have not measured. Its job is narrow and concrete: read the dataset as it stands and report, field by field, why a model cannot learn from it in its current form.
Three failure modes dominate. Data can be restricted, meaning regulation or contract forbids the raw values from reaching a training pipeline. Data can be scarce, meaning the events you most need to model, fraud, rare defects, unusual cohorts, appear too few times for a model to generalize. And data can be structurally broken, meaning schema drift, inconsistent encodings, or mismatched joins have quietly corrupted the relationships a model depends on. Diagnose produces a profile that separates these, because each one calls for a different rebuild.
The output is not a pass or fail stamp. It is a readiness profile with a defect map attached: which columns are blocked, which patterns are underrepresented, and where the structure has broken down. That map is what the Transform step reads next.
Why teams skip diagnosis, and what it costs
The usual sequence in a stalled AI project is to blame the model, then the features, then the labels, and only much later the data itself. By then the team has spent weeks tuning around a defect that a proper data readiness diagnosis would have surfaced on day one. Redman’s often-cited estimate, published through Harvard Business Review, put the cost of bad data in the US at roughly $3.1 trillion a year, and most of that waste is downstream of a data quality assessment nobody ran.
Skipping diagnosis also hides which problem you have. A team that assumes its data is merely scarce will reach for augmentation and get nowhere, because the real blocker was a restricted field they were never allowed to use. A team that assumes a compliance wall will over-anonymize data that was structurally fine, destroying the signal in the process. Naming the defect correctly is half the fix.
The three checks inside the Diagnose step
Diagnose runs against the raw data through DTS’s non-access architecture, so the data profiling reads statistical shape and structure without exporting the sensitive values themselves. Three checks run in parallel, and each one feeds a different part of the rebuild plan.
| Check | Question it answers | What it feeds into the rebuild |
|---|---|---|
| Restriction check | Which fields are blocked by regulation or contract from reaching a model? | Structure-Preserving Synthesis of the blocked fields |
| Scarcity check | Which important patterns are too rare for a model to generalize? | Rare-pattern augmentation of the underrepresented cases |
| Structure check | Where has schema drift or bad encoding broken the data’s relationships? | Repair of joins, types, and consistency before synthesis |
The restriction check maps every field against what is allowed to leave the boundary. The scarcity check measures class balance and the density of the rare events you actually care about, not overall row count, because a million rows can still hold only forty fraud cases. The structure check compares the data’s current shape against what the schema promises and flags where they diverge.
How diagnosis hands off to Transform and Rebuild
Diagnosis is only useful if it changes what happens next. The readiness profile is a set of instructions for the rest of DTS. Restricted fields route to Structure-Preserving Synthesis, which rebuilds them into values a model can learn on while keeping the distributions and relationships intact. Scarce patterns route to rare-pattern augmentation, which increases the density of the underrepresented cases without inventing signal that was never there. Broken structure routes to repair first, because synthesizing on top of corrupted joins would only launder the corruption.
This is why the Diagnose step is a measurement stage and not a cleaning stage. It does not fix anything on its own. It tells Transform and Rebuild exactly where to act, and it records the before-state so the improvement can be checked afterward. DTS runs on the Syntitan platform, which keeps that before-and-after state so a rebuilt dataset can be compared against what it replaced rather than trusted on faith.
Run this quick diagnostic on your own dataset
Before you commit to a rebuild, you can run a rough version of the Diagnose logic by hand. If you answer no to any of these, your data is not yet ready for the model you have in mind:
- Can you name every field that regulation or contract forbids from reaching a training pipeline?
- Do you know the raw count, not the percentage, of the rarest event your model must predict?
- Does the data’s current schema match what your pipeline expects, with no silent type or encoding drift?
- Have you measured class balance on the specific outcome you care about, rather than assuming it from row volume?
- Can you tell whether your blocker is restriction, scarcity, or broken structure, rather than lumping them together as “bad data”?
Most teams stall on the second and fifth questions. They know the data is a problem but cannot say which problem, and that is exactly the gap a data readiness diagnosis closes. Regulation now assumes this measurement exists: Article 10 of the EU AI Act holds the training, validation, and testing data behind high-risk systems to documented data-governance practices and to explicit quality criteria, meaning data that is relevant, sufficiently representative, and as close to complete and error-free as possible.
Where the Diagnose step fits in CUBIG’s operating layer
DTS is CUBIG’s AI-ready data transformation engine, and Diagnose is its opening move: measure the defect, then Transform and Rebuild against it. The rebuild uses Structure-Preserving Synthesis and rare-pattern augmentation, and the whole engine runs through a non-access architecture so restricted values are never exported to be profiled. DTS runs on the Syntitan platform, so a dataset that Diagnose flagged and Rebuild repaired can be diffed against its earlier state instead of being taken on trust.
The practical takeaway is that diagnosis is not overhead you add to a data project; it is the step that decides whether the rest of the project is aimed at the right target. Run it first, and the transformation that follows has something specific to fix.
Try it on your data for free. Run a sample proof and see it on your own workflow.
