A subscription business prepares a churn model from three systems. Billing stores timestamps in local time, the CRM uses its own product codes and the support tool logs one row per ticket. Before training, the team converts every time to one zone, maps product codes to a shared list and rolls tickets up into weekly counts per customer.
Data transformation is the process of converting data from one format, structure or set of values into another so it can be stored, combined, analyzed or used by a machine learning model.
Typical operations include type conversion, normalization, aggregation, joining tables, encoding categorical fields with methods such as one-hot encoding and deriving new columns. Transformation is the T in ETL and ELT, which differ in whether data is transformed before or after it is loaded into the target system.
In machine learning work, many transformations are fitted to data, such as a scaler that learns a column’s mean. The scikit-learn guide to pipelines and composite estimators explains that bringing statistics from test data into preprocessors such as scalers or imputers makes cross-validation scores unreliable. Its ColumnTransformer applies different transformations to different columns inside a pipeline that is safe from this kind of data leakage.
How is data transformation different from data cleaning?
- Data cleaning corrects errors: duplicates, invalid values and inconsistent spellings.
- Data transformation changes structure or representation: units, formats, encodings and the level of aggregation.
- Most pipelines do both as part of data preparation. A cleaning step fixes a wrong value. A transformation step changes how a correct value is represented.
Where does data transformation break in AI work?
- Stale mappings: a survey vendor reverses its rating scale, so 5 now means the opposite. The column name, type and schema stay the same, and the transformation that treats 5 as very satisfied keeps running.
- Fitted on the wrong data: scaling or encoding fitted on the full dataset before the split leaks evaluation information into training.
- One transformation for every task: weekly aggregation that suits a churn model can erase the event timing a fraud model needs.
- Unrecorded steps: when a transformation lives in an unsaved notebook cell, nobody can later say which version produced a given training set.
CUBIG Shorts
The Survey Flipped. Nobody Told the Model.
Episode 5 of It Worked in the Demo, CUBIG’s webinar series on enterprise AI.
How do teams keep data transformations reliable?
Teams write transformations as versioned, repeatable code, fit them on training data only and apply the same fitted steps to evaluation and production inputs. When a source system changes, they confirm field meanings with the data owner and update the field mapping before the next run, since a valid schema does not guarantee that the values still mean the same thing.
Recording each step supports reproducibility and data lineage. A team can then trace a result back to the exact transformations applied and compare results before and after a change under the same evaluation conditions.
Related entries: feature engineering, schema drift and data versioning.