No Priors Ep. 124 | With SurgeAI Founder and CEO Edwin Chen
Edwin Chen, CEO of Surge AI, discusses the critical role of high-quality human data in training frontier models, the flaws in current evaluation benchmarks like LMSys and IF-Eval, the future of complex RL environments, and why he bootstrapped Surge to over $1 billion in revenue.
Edwin Chen, founder and CEO of Surge AI, shares the principles that allowed his company to bootstrap to over a billion in revenue by focusing on the power of high-quality human data to advance AI.
The Philosophy of Quality Data
The founding thesis of Surge AI was a belief in the power of high-quality human data. Historically, data for tasks like drawing a bounding box around a car was treated as a commodity with a low ceiling on quality—anyone from a second grader to Hemingway would draw the same box. However, in the generative AI era, tasks like writing poetry or code have an “unlimited ceiling” on quality. A Nobel Prize laureate will write a vastly more meaningful poem than an average person, even if both fulfill the basic instructions.
Most companies approach data by hiring people and creating checklists (e.g., “Is it an eight-line poem? Does it mention the moon?”). This leads to mediocre, commodity data. Surge’s approach is different; it focuses on capturing the richness, creativity, and subjectivity that constitutes true quality. This requires building technology to identify what makes a poem, a math proof, or a web app genuinely great, not just functional. Training models on this richer data helps them learn the deeper patterns that make language and the world compelling, rather than just learning to follow instructions.
A Critique of Modern AI Evaluation
A significant plague on AI progress is the reliance on flawed evaluation methods that incentivize the wrong behaviors.
- Superficial Human Evaluation: Platforms like the LMSys arena, where users spend 5-10 seconds “vibing” with a model’s response, are a major problem. This encourages models to produce “clickbait”—longer responses with more emojis and formatting—because it looks more impressive. Chen notes that researchers have admitted to him that they are willing to make their models worse on factuality and instruction following just to improve their rankings on these leaderboards.
- Benchmark Hacking: Many academic benchmarks are “completely divorced from the real world.” For example, the IF-Eval (Instruction Following Eval) benchmark tests models on contrived tasks like writing an essay about Abraham Lincoln while capitalizing specific letters in his name. Teams become preoccupied with optimizing for these “silly SAT-style benchmarks” instead of focusing on real-world progress.
The gold standard for evaluation is proper, time-intensive human evaluation that involves fact-checking, verifying instructions, and having people with good taste assess the quality of the output.
The Enduring Role of Humans and Scalable Oversight
Even as models become superhuman, human feedback will never run out. Models are not self-consistent and require an “external signal” to stay aligned with human objectives. For instance, a top frontier model today might randomly start outputting Hindi and Russian characters in the middle of a response, a flaw it is not aware of itself.
Furthermore, synthetic data has its limits. Customers often discover that a small amount of curated, high-quality human data is more valuable than millions of synthetic data points, 99% of which may be useless.
The key is “scalable oversight,” where humans and AI work together to produce data that is better than either could alone. This involves building the right tools and interfaces, such as having a human start with a model-generated draft and then layering on the creativity, ingenuity, and nuance that only a human can provide.
The Future of RL and the Competitive Landscape
There is a massive demand for rich, complex Reinforcement Learning (RL) environments. These are not simple simulations but aim to model an agent’s entire world. For example, an environment for a salesperson might include interactions with Salesforce, Gmail, and Slack, alongside dynamic external events like a simulated car accident that affects their schedule. Chen believes there is “no ceiling” on the realism and complexity needed for these environments, as more richness provides more opportunities for models to learn over long time horizons.
Looking at the broader market, Chen does not believe frontier models will become a commodity. Instead, he foresees a diversity of models, each with unique strengths, personalities, and principles.
- Anthropic: Strong focus on enterprise.
- OpenAI: Big consumer focus due to ChatGPT.
- Grok: A different personality and willingness to discuss certain topics.
Users will continue to switch between models depending on the task. When asked about an underdog who could catch up to the leaders, Chen points to xAI, noting they are “very hungry and mission-oriented.”