"Garbage In, Garbage Out" is a LIE

The long-held mantra in data science, “garbage in, garbage out,” is being challenged by the surprising success of predictive models trained on large, messy datasets. The paper, From Garbage to Gold: A Data Architectural Theory of Predictive Robustness, provides a theoretical explanation for this phenomenon, shifting the focus from algorithmic behavior to the inherent characteristics of the data architecture itself.

This work is particularly relevant in high-stakes, regulated domains like healthcare and finance, where decision-makers are often skeptical of predictions derived from notoriously messy real-world data, such as electronic health records. Even with empirical evidence of a model’s success, the belief that data must be meticulously cleaned before use often hinders adoption.

The Core Theory: Latent Drivers and Two Types of Noise

The theory’s central insight is that in most complex systems, the data we observe—both predictors and outcomes—are manifestations of an underlying, unobservable latent structure. The path to robust prediction lies in recovering this latent structure from the observed data.

This framework partitions noise into two distinct categories:

1. Observational Error: This is the familiar type of noise, including incorrect measurements, data entry mistakes, and missing values. This is the “garbage” that traditional data cleaning aims to fix.
2. Structural Uncertainty: This noise arises because observed variables are merely imperfect proxies for the true latent drivers. For example, Body Mass Index (BMI) is a proxy for the latent concept of “metabolic syndrome,” but it doesn’t capture it completely. Even a perfectly measured variable still has this structural ambiguity.

The key takeaway is that while traditional data cleaning can reduce observational error, it does little to address structural uncertainty. In fact, the common practice of reducing the number of features to make cleaning manageable can be detrimental. By shrinking the predictor space, you may be left with a small set of clean variables that still fail to fully capture the underlying latent drivers, thus limiting predictive power.

Embracing Breadth and Redundancy

The theory proposes that the way to overcome structural uncertainty is to expand the predictor set. By incorporating a larger number of variables, even if they are individually noisy, you provide more angles from which to triangulate the true latent signals.

• The Power of More Data: Adding more error-prone predictors can paradoxically improve the model. The math shows that as the number of predictors approaches infinity, you can perfectly recover the latent drivers. More data, even with errors, can be better than a small amount of clean data.
• Redundancy is a Signal: Contrary to the traditional preference for parsimonious models (Occam’s razor), this theory shows that redundancy (collinearity) is beneficial. Correlated variables are often influenced by the same latent driver. Including them provides multiple, distinct views of that driver, which helps to more efficiently and accurately estimate its state, effectively averaging out the observational errors in each individual variable. The goal is prediction, not causal inference, so the traditional concerns about collinearity are less relevant.

Practical Application and Feature Selection

While the asymptotic theory suggests “use everything,” practical constraints require intelligent feature selection. The goal shifts from finding error-free data to building a portfolio of predictors that comprehensively and redundantly covers the latent space.

• Residual-Based Selection: If a clean outcome variable is available, a straightforward approach is to build an initial model and then correlate potential new predictors with the model’s residuals. The variable that has the highest correlation with the remaining uncertainty provides the most information gain and should be added next. This creates an iterative, boosting-like process for expanding the feature set.
• Explainability at the Latent Layer: Instead of explaining a model’s output by citing the influence of raw, noisy input features (e.g., “BMI was a factor”), explanations should be made at the more stable and reliable latent layer. One would explain the prediction based on the uncovered latent drivers (e.g., “the model detected a high risk of ‘metabolic syndrome’”), which can be semantically described by the group of observed variables that load onto them.

Relation to Tabular Foundation Models

This theory provides a principled, data-architectural explanation for why modern approaches like tabular foundation models—which ingest massive, raw, and often messy relational data—can be successful. They implicitly leverage this “dirty breadth” to triangulate latent signals.

However, a key difference lies in the deployment philosophy. The theory advocates for building models on local data that is representative of the specific context and time, which is critical for high-stakes, mission-critical applications. A universal foundation model may carry biases from its diverse pre-training data that cannot be fully overcome by fine-tuning, especially when the local dataset is small. The proposed approach is a “method transfer” paradigm, where a generalizable method is applied to local data to build a bespoke, stable, and governable predictive system.

• Tags:
• Feature
• Data-Centric AI
• Machine Learning
• Tabular Data
• Predictive Robustness
• Latent Variables
• Noisy Data
• Explainability