Solving the Wrong Problem Works Better - Robert Lange
Robert Lange from Sakana AI discusses Shinka Evolve, a framework combining LLMs with evolutionary algorithms for open-ended program search. The conversation explores how Shinka Evolve addresses the limitations of systems like AlphaEvolve by co-evolving problems and solutions, its sample-efficient architecture using UCB bandits and quality-diversity search, and its applications in circle packing, competitive programming, and evolving MoE loss functions. The discussion also delves into the philosophical debate on whether these systems produce true novelty or are parasitic on their starting conditions, and the future role of the "AI Scientist" as a human co-pilot.
Current evolutionary systems for program synthesis, such as AlphaEvolve, are powerful but limited by a fundamental constraint: they optimize solutions for a fixed, human-defined problem. The core idea behind Shinka Evolve is that true scientific innovation often requires inventing a new, simpler, or intermediate problem first. This concept of co-evolving problems and solutions is crucial for escaping local optima and enabling a truly open-ended discovery process.
This is exemplified by a practical finding in the circle packing problem: initially using a problem formulation with a tiny amount of “slack” (allowing circles to overlap slightly) served as a valuable surrogate problem. It allowed the system to find a high-quality solution faster than when it was constrained to the exact, no-overlap setting from the start. This points towards the need for systems that can automatically discover and leverage such surrogate problems, moving beyond a fixed fitness function. This approach draws inspiration from foundational concepts like POET (Paired Open-Ended Trailblazer) and PowerPlay, which co-evolve agents and their environments to create an auto-curriculum.
The Architecture of Shinka Evolve
To achieve its goals with high sample efficiency, Shinka Evolve introduces several key architectural innovations to the evolutionary search process. The system maintains an archive, or a database of programs, which can be visualized as a tree of evolving solutions. Large Language Models (LLMs) act as the primary mutation operators.
Three key innovations set it apart:
- Diverse Mutation Operators: Instead of relying on a single method, Shinka Evolve uses a variety of operators to maintain population diversity. This includes proposing
diff-based patches, performing full program rewrites to escape local optima, and executing crossover mutations that ask an LLM to merge the complementary strengths of two different parent programs. - Quality-Diversity and MAP-Elites: The system’s archive functions similarly to the MAP-Elites algorithm. It maintains a diverse population of programs, preventing premature convergence on a single solution. This allows the system to explore a wide range of “stepping stones”—solutions that may not be optimal for the main objective but possess unique characteristics that could be useful later.
- Adaptive Model Ensembling with UCB Bandits: A significant challenge is assigning credit when a sequence of mutations from different models leads to a breakthrough. To address this, Shinka Evolve uses an Upper Confidence Bound (UCB) bandit algorithm to adaptively select which frontier model (e.g., GPT-5, Gemini, Sonnet 4.5) to use for a given mutation. Each model is an “arm” of the bandit, and the system learns over time which model is most effective for different types of programs or stages of the evolutionary process, optimizing both cost and performance without prematurely converging on a single “best” model.
- Semantic Guidance via a Meta-Scratchpad: To inject a higher level of understanding, the system summarizes each program and extracts key insights into a global “scratchpad.” These insights are then used to generate meta-recommendations that are fed back into the system prompt, helping to guide the evolutionary search semantically rather than relying purely on the fitness score.
Applications and Key Results
Shinka Evolve has demonstrated its effectiveness across several challenging domains:
- Circle Packing: It achieved state-of-the-art results while requiring dramatically fewer program evaluations (under 200) than previous methods, highlighting its sample efficiency.
- Competitive Programming: When used as a “co-scientist” to refine solutions from another agent, the combined system would have placed second in a real-world AtCoder competitive programming challenge.
- Mixture-of-Experts (MoE) Models: The framework successfully evolved novel load-balancing loss functions, illuminating the trade-offs between model performance and expert utilization.
- Agentic Systems: Using the Automated Design of Agentic Systems (ADAS) framework, Shinka Evolve was used to evolve the agent scaffold itself for solving AIME math benchmarks, improving the performance of cheaper models like GPT-4.1 Nano.
The AI Scientist: Co-Pilot or Autonomous Researcher?
A central theme of the discussion is the role of these systems in the future of science. The AI Scientist v2 paper demonstrates a system capable of automating the full research pipeline, from hypothesis generation to running experiments and writing a paper that successfully passed the acceptance threshold at a workshop.
However, this raises the “slop critique”: are these systems producing genuine scientific insight, or are they just generating workshop-level content that mimics the structure of science without deep understanding? Robert Lange is candid that the current version is more of a co-pilot than an autonomous researcher. The vision is a future where the human acts as a “shepherd,” steering the high-level direction of a massively parallel, multi-threaded search process that runs autonomously.
“I really hope that that’s the way forward, right? In the sense that humans are the ones steering the ship while just being massively amplified in their productivity.”
This human-in-the-loop model addresses the critique that LLMs, when run fully autonomously, often fail to produce interesting novelty and remain “parasitic on their starting conditions.” The counterargument is the stepping-stone philosophy: evolution doesn’t need to extrapolate from first principles, but rather recombine diverse, existing ideas in useful ways. The key is to maintain a rich pool of these stepping stones for the system to draw from.
In the next 5-20 years, the process of conducting science is expected to fundamentally change, moving from a “with AI” to a “by AI” paradigm, where humans provide high-level orchestration.