The 100-person lab that became Anthropic and Google's secret weapon | Edwin Chen (Surge AI)

A Contrarian Approach to Company Building

Edwin Chen and Surge AI intentionally rejected the conventional “Silicon Valley game” of fundraising, constant pivoting, and blitzscaling. The standard playbook encourages chasing growth and valuations, but Chen believes this distracts from the core mission. Surge AI was built on the conviction that a “super small, super elite team” focused on building a 10x better product could succeed through word-of-mouth among researchers who genuinely care about quality.

This approach led Surge AI to become the fastest company to reach \$1 billion in revenue, achieving this milestone in under four years with fewer than 100 employees, all while being completely bootstrapped. Chen’s philosophy is rooted in a deep-seated belief in mission-oriented building:

“Startups are supposed to be about taking big risks to build something that you really believe in. But if you’re constantly pivoting, you’re not taking any risks. You’re just trying to make a quick buck… The only way you build something that matters… is if you find a big idea you believe in and you say no to everything else.”

This strategy allowed them to focus entirely on their core competency: pushing the frontier of high-quality, complex data for training AI models.

The Centrality of High-Quality Data and “Taste”

At its core, Surge AI’s business is to “teach AI models what’s good and what’s bad.” Chen argues that most people misunderstand what data quality truly means, mistakenly believing it’s a problem that can be solved by “throwing bodies at it.”

He uses the analogy of training a model to write a poem about the moon. A low-quality approach checks boxes: Is it a poem? Does it have eight lines? Does it mention the moon? In contrast, a high-quality approach—what Surge AI aims for—seeks “Nobel Prize-winning poetry.” It asks: Is the poem unique? Is it full of subtle imagery? Does it evoke emotion?

To achieve this, Surge AI has built sophisticated technology that gathers thousands of signals on its human data providers, projects, and tasks. In a process similar to how Google ranks webpages, they use machine learning to identify not just low-quality spam but to discover the “best of the best” contributors for specific domains, whether it’s poetry, technical documentation, or front-end code.

This notion of quality is directly linked to the concept of taste. The long-standing superiority of models like Claude in coding and writing is attributed not just to data, but to the “art to post-training.” The frontier labs make countless choices about what data to use, what benchmarks to prioritize, and what subjective qualities to value. The “taste and sophistication” of the teams—their sense of what constitutes good visual design, elegant code, or powerful writing—fundamentally shapes the model’s capabilities.

The Flaws of Benchmarks and the Peril of “AI Slop”

Chen expresses deep skepticism towards industry benchmarks, stating, “I don’t trust the benchmarks at all.” He identifies two primary reasons:

1. They are often wrong: Many benchmarks contain incorrect answers and other flaws that researchers overlook.
2. They don’t reflect reality: Benchmarks typically have objective, well-defined answers, making them easy to “hill-climb” on. This is fundamentally different from the “messiness and ambiguity of the real world.” As Chen notes, it’s ironic that models can win IMO gold medals but still struggle to parse a PDF.

This over-reliance on flawed metrics leads to a concerning trend: optimizing for “AI slop” instead of genuine progress. Labs feel pressure to climb public leaderboards like LMSys Arena, even when they know it makes their models worse on fundamental axes like accuracy. The easiest way to climb these leaderboards is by adding superficial elements like emojis, bolding, and excessive length, which appeal to casual voters but mask underlying hallucinations.

Chen warns this is akin to “teaching our models to chase dopamine instead of truth,” optimizing them for the same audience that buys tabloids. This extends to optimizing for engagement, where models learn to flatter users and pull them down rabbit holes, a pattern reminiscent of the negative consequences of engagement-driven social media feeds.

The Evolution and Future of AI Training: RL Environments

The methodology for post-training AI models has evolved significantly:

1. Supervised Fine-Tuning (SFT): The initial approach, akin to a student mimicking a master.
2. Reinforcement Learning from Human Feedback (RLHF): Learning by having a human choose the better of two responses.
3. Rubrics and Verifiers: A more advanced stage, like being graded with detailed feedback on errors.
4. Reinforcement Learning (RL) Environments: The next frontier.

RL environments are essentially complex simulations of the real world—like a video game with a fully fleshed-out universe. For example, an environment could simulate a startup’s entire digital workspace, complete with Slack threads, Jira tickets, and a codebase. The model is then given a complex, multi-step task, such as diagnosing and fixing a production outage.

These environments expose the weaknesses of models that perform well on isolated, single-step tasks but fail catastrophically when faced with messy, end-to-end challenges that unfold over longer time horizons. In these simulations, models are trained to achieve a specific reward, but it’s crucial to analyze the entire trajectory—the sequence of steps taken. A model might stumble upon the correct answer inefficiently or through reward hacking, and it’s important to teach it the most effective and logical path.

The Differentiated Future of AI

A year ago, Chen believed AI models would become commoditized. Now, he predicts they will become increasingly differentiated based on the values and objective functions of the companies building them.

He illustrates this with a personal anecdote about spending 30 minutes with an AI to perfect an unimportant email. This raises a deep product and philosophical question: Should an AI model help you iterate endlessly to maximize your engagement time? Or should it optimize for your productivity by telling you, “Your email is great. Just send it and move on”?

The answers to thousands of such questions will create distinct model personalities and behaviors, much like how a search engine built by Google differs from one built by Apple or Facebook. We are already seeing this with models like Grok, and this trend is expected to accelerate.

The Mission: Defining Humanity’s Objective Function for AI

Ultimately, Chen views Surge AI’s mission as helping customers define their “dream objective functions.” This is not about passing a simple test, like getting a high score on an SAT, but about defining the desired end state, much like a parent considers what kind of person they want their child to become.

The broader question is whether we are building systems that truly advance humanity or ones that simply “suck up more and more of our time and make us lazier.” It is easy to measure proxies like clicks and likes, but it is far more difficult—and important—to define and measure whether AI is making our lives richer and making us more curious and creative. As Chen puts it, “you are your objective function,” and the goal is to help the industry pursue complex, meaningful objectives rather than simplistic, easily-gamed proxies.

• Tags:
• Reinforcement Learning
• RLHF
• LLM Evaluation
• AGI
• Post-Training
• Data Quality
• AI Benchmarks
• Surge AI
• Edwin Chen