The Bitter Lesson for Biology: Scaling Data, Intuition, and the Limits of AlphaFold

The panel opens with a sharp correction to a common narrative: protein folding is not solved. AlphaFold learned to replicate structures deposited in the PDB, which has proven enormously useful, but it did not establish the true ground-state structure or the full distribution of conformations a protein adopts. As Pushmeet Kohli puts it, “that does not mean that we have understood all of protein dynamics.”

Reframing the Bitter Lesson for Data

Brandon Anderson frames the discussion around a data-centric version of Rich Sutton’s bitter lesson: that general methods that scale eventually win. Sal Candido pushes back on the idea that scaling laws are simply everywhere. In his view, much of the real work is discovering the regime where adding compute and data reliably improves results. Once that scaling law is found, “you can kind of just turn the crank,” and the problem becomes engineering. He stresses that architecture matters, but data matters just as much, since a model can only generalize from the statistical information its training set actually contains.

On the question of whether modelers chase available data rather than the most impactful problems, Sal admits a pragmatic bent (“I’m lazy so I’m going to work with what’s available”). He points to a useful paradox: the protein language models he trains on metagenomic sequences, which are often fragments and far from pristine, nonetheless improve at designing and understanding real proteins. The danger is concluding that one should simply scale up whatever data is easy to generate. The remedy he advocates is open, community-driven work in which modelers and data generators collaborate from the start, so the field can determine which data is actually needed and then locate the scaling law for it.

Problem First, Then Solution Space

Pushmeet recalls Rich Sutton’s lecture at DeepMind and reads the bitter lesson conceptually rather than as a statement about data. The real pitfall, he argues, is approaching problems religiously, as a modeler or as a data generator. Machine learning often falls into this trap by treating a fixed dataset with train and test splits as the problem itself. The more useful stance is to define the scientific problem first, remain flexible about whether it calls for modeling, compute, or new data collection, and recognize that expertise is part of the process alongside data and models.

He illustrates this with AlphaFold. The team worked within the existing PDB because expanding it by an order of magnitude was infeasible given the effort the global structural biology community had already invested. In cell genomics, by contrast, the team concluded that the data was not yet adequate for the grand ambition of a virtual cell. His advice to newcomers is to become multidisciplinary, understand the problem and the real constraints on data and model size, and fail fast toward approaches that are feasible in the long term.

Handcrafted Architectures Versus Scale

The conversation then turns to AlphaFold 2 as a work of craft: carefully engineered features that encode biophysical intuition, such as the idea that residues influence one another rather than acting independently. Pushmeet argues that baking this knowledge in gives the model an “unfair advantage,” making it far more data efficient. He also argues that curating good data is itself a hard problem, since simply replicating the same data yields no progress. As he puts it, “it’s not just about big data it’s about good data,” and understanding data coverage is arguably a more interesting challenge than scale itself.

Sal agrees that the right amount of inductive bias depends on the data regime. With limited data, strong priors help, but as data grows the model can discover things its designers did not anticipate, and an incorrect bias can become a constraint. He also disputes the notion that scaling is free of craft: it involves substantial algorithmic work in training, infrastructure, and inference speed. He emphasizes that “we’re not in a post-Transformer world in any way, shape or form,” yet architectures are being modified into more bespoke, fit-for-purpose forms that work at internet scale. The central question, in his framing, is less about data and more about information: what the next batch of information a model needs is, and what architecture extracts the most from it at each scale.

The Limits of Structure Prediction

Brandon raises the common claim that structure prediction is solved while function, dynamics, and design remain open. Pushmeet emphasizes that science advances by isolating problems, and that the “building blocks” metaphor misleads. Proteins are disordered, their shapes shift with context, and AlphaFold’s achievement was replicating a single deposited conformation. Communicating progress to the public is easier than conveying how much remains, and he urges continued investment in structure prediction and dynamics because, in his words, “we are just getting started.”

Asked for a magic wand, Pushmeet, whose background spans security research, computer vision, Bayesian theory, discriminative learning, AI for coding, and science, points to cryo-EM micrographs. Rather than working from curated structures that may omit information, he argues models should operate at the source and extract the dynamical and distributional information encoded in those images. He acknowledges this requires substantial work but sees it as a promising route, perhaps for future researchers to pursue.

Sal extends the point with an analogy: current models are like studying a single spoke to understand a bicycle. Useful as they are, the field must move from models of spokes to wheels to whole bicycles, placing proteins in their biological context so that interactions can be learned more broadly. He also notes that protein language models already contain more information than is commonly appreciated. Interpretability work has revealed representations of structure, functional properties, and motions, suggesting that a great deal remains to be unlocked, and that compressing evolutionary information into a model may amount to a kind of world model.

Design, Understanding, and Trust

Brandon brings up Feynman’s principle that “what I cannot create, I do not understand,” noting that generative tools now create things without understanding them. Sal argues that designing with black boxes is not new and remains useful, but that understanding becomes essential as intelligence grows cheaper and more available. The challenge is extracting knowledge from these machines. He also observes that scientists generally want understanding, though translational medicine ultimately has to deliver results.

Pushmeet offers a more nuanced position. Some level of understanding is necessary, but the right target is trust rather than full interpretability. AlphaFold 2 is not perfect, and even at high accuracy, an uncalibrated confidence measure such as pLDDT could mislead scientists for a year before the errors surface. Calibration of uncertainty was therefore critical. Interpretability, he argues, depends on who is doing the interpreting. A human scientist with cognitive limits cannot readily interpret AlphaFold 2, but a much larger model with access to activations might develop a theory of how it works. Behavioral characterization, meaning a clear account of what the model can and cannot do, is what users need in order to avoid being harmed by a tool that should help them. He also notes that AlphaFold 2 turned out to be a strong predictor of disordered regions, something not anticipated before release, which demonstrates generalization.

Timelines and the Case for 10x Breakthroughs

In the closing questions, Brandon asks when AI-driven results will reach the clinic. Pushmeet observes that AI already permeates drug discovery, so the real question is what is being accelerated: lead optimization, target discovery, preclinical work, or clinical trials. A 10x or 100x shift in timelines will require tackling hard biological challenges, and only better biological models, the goal of the effort announced that day, will unlock that transformation.

Sal believes rapid progress is coming, though he is unsure whether a drug made entirely by AI is near. He draws on a lesson from his time at Google: aiming for a 10x improvement rather than 10% can force a broader view and reveal approaches that incremental thinking misses. Both paths are valuable, but Biohub’s mission to cure all disease demands the 10x mindset. The panel closes on that note.

• Tags:
• Drug Discovery
• Computational Biology
• AlphaFold
• Interpretability
• Scaling Laws
• Google DeepMind
• Bitter Lesson
• Biohub
• Virtual Cell
• Cryo-EM
• Protein Structure Prediction
• Protein Language Models
• Uncertainty Calibration