How A Team Of 7 Keeps Breaking AI Benchmark Records
Poetiq, a startup by former DeepMind researchers, has developed a recursive self-improvement meta-system that builds "reasoning harnesses" on top of existing LLMs. This approach avoids the costly "fine-tuning trap" and has achieved state-of-the-art results on benchmarks like ARC-AGI and Humanity's Last Exam by automatically optimizing prompts and discovering novel reasoning strategies.
Poetiq is building a recursively self-improving system that acts as a “reasoning harness” for large language models (LLMs). The core insight is a method for recursive self-improvement—where an AI makes itself smarter—that is significantly faster and cheaper than traditional approaches, which typically require retraining a new LLM from scratch at a cost of hundreds of millions of dollars.
The Challenge: The “Fine-Tuning Trap”
Many companies building on LLMs face a significant challenge: the “fine-tuning trap.” The conventional approach involves:
- Collecting a large dataset (tens of thousands of examples).
- Spending a great deal on compute to fine-tune a frontier or open-weights model.
- Achieving improved performance on a specific task.
However, this process is vulnerable to the “bitter lesson.” Soon after, a new, more powerful base model is released (e.g., GPT-4o) that outperforms the specialized, fine-tuned model out of the box. The company is then faced with the choice of repeating the expensive fine-tuning process or going out of business.
Poetiq’s Solution: “Stilts” for LLMs
Poetiq offers a different paradigm. Instead of fine-tuning, it provides an agentic system, or “harness,” that sits on top of one or more LLMs. This harness enhances the base model’s capabilities, acting like “stilts” to make it perform better on specific, hard problems.
Key advantages of this approach include:
- Model Agnostic: When a new frontier model is released, the same harness remains compatible and provides an immediate performance boost.
- Cost-Effective: The optimization process is vastly cheaper than fine-tuning.
- Continuous Improvement: The harness can be further optimized for the new model to achieve even greater performance, ensuring the system always outperforms the underlying base models.
How the Meta-System Works
Poetiq’s core technology is a recursively self-improving meta-system. The output of this meta-system is not just a solution, but systems that solve hard problems. This automated optimization process can generate a complete reasoning harness—comprising code, prompts, and data—from scratch.
Furthermore, if a startup has already built its own agent, Poetiq can ingest that system and optimize its components, such as prompts or reasoning strategies. The meta-system analyzes the problem data to discover failure modes and identify robust reasoning paths, effectively outsourcing the deep, manual data analysis to the AI itself.
Automating Prompt and Reasoning Engineering
The system moves beyond simple prompt optimization. While automated prompt tuning (like the popular DSPy framework) provides some gains, the most significant improvements come from discovering novel reasoning strategies that are implemented in code.
In one example from a previous project at DeepMind, manual prompt optimization on a very hard task took performance to 5%. However, by adding optimized reasoning strategies, performance jumped from 5% to 95%. The Poetiq meta-system automates this discovery process, often generating non-intuitive prompts and strategies that a human might not devise.
Demonstrated Success on Key Benchmarks
Poetiq has validated its approach by achieving top rankings on difficult AI benchmarks:
- ARC-AGI: Shortly after a new model set the state-of-the-art at 45%, Poetiq’s system achieved 54% accuracy. Notably, it did so by building on a cheaper base model, costing only $32 per problem compared to the previous SOTA’s $70+.
- Humanity’s Last Exam: On this set of 2,500 expert-level questions, Poetiq achieved a score of 55%, surpassing the previous record of 53.1% set by Anthropic’s Claude Opus. The entire optimization run for this achievement cost less than $100,000, a fraction of the cost of training a foundation model.
These results demonstrate the system’s ability to enhance both complex reasoning (ARC-AGI) and deep knowledge extraction (Humanity’s Last Exam).
Advice for Engineers in the AI Era
The world is changing rapidly, and the best way to keep up is to engage directly and consistently. The advice for engineers is to try things and do something with AI every day. Don’t limit your imagination; use AI to see how far you can push the boundaries of what’s possible. What took a weekend to build eight months ago is now even faster and easier, highlighting the incredible pace of progress.