Tokenless

Production ml

Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

Building Closed-Loop Evals for a Multimodal Agent at Scale — Soumya Gupta & Jai Chopra, Uber

This talk details how Uber Eats designed and implemented a multimodal AI agent to enhance food photography for independent merchants, addressing challenges like maintaining authenticity, merchant brand, and marketplace diversity while operating at scale. It covers the intricate evaluation strategies for routing and image editing agents, including continuous learning loops, managing drift, countering reward hacking, and balancing creative freedom with rigid safety guardrails. The speakers explain how they built a closed feedback loop combining offline human labeling, internal dogfooding, and online production signals to ensure robust and adaptive performance.

Stop AI Agent Hallucinations: 5 Techniques + Production Patterns - Elizabeth Fuentes, AWS

Stop AI Agent Hallucinations: 5 Techniques + Production Patterns - Elizabeth Fuentes, AWS

Explore five research-backed techniques to overcome common AI agent architectural limitations like hallucination, inefficient tool use, and rule violations. Learn how to implement semantic tool selection, Graph-RAG, multi-agent validation, neurosymbolic guardrails, and agent steering through code changes, not prompt engineering, and deploy them using AWS Bedrock Agent Core for robust and cost-effective AI agents.

Research to Reality: Bringing Frontier ML Research to Production - Vaidas Razgaitis, Higharc

Research to Reality: Bringing Frontier ML Research to Production - Vaidas Razgaitis, Higharc

Vaidas Razgaitis, Senior Research Engineer at Higharc, shares three tactical tips to accelerate the transition of novel AI/ML research into production-ready features. He emphasizes addressing the critical handoff challenge between ML researchers and software engineers through structured documentation (Research Prototype Taxonomy Document), a well-organized monorepo utilizing decoupled microservices, and a systematic approach to code decomposition and PR review. These strategies aim to improve legibility, maintainability, and delivery speed for ML-driven products.

It's 2026, and We're Still Talking Evals

It's 2026, and We're Still Talking Evals

Maggie Konstanty, AI Product Manager at Prosus, provides a candid look into the realities of LLM evaluation in production. She argues that standard metrics like accuracy are misleading and advocates for a culture of continuous, goal-oriented evaluation focused on deep failure analysis and understanding real user behavior, asserting that mature teams inevitably build custom tooling to meet their specific needs.

Agents as Search Engineers // Santoshkalyan Rayadhurgam

Agents as Search Engineers // Santoshkalyan Rayadhurgam

Large language models are transforming search from a static, stateless process into a dynamic, agent-based reasoning system. This talk explores the practical patterns—like query rewriting, hybrid retrieval, and agent-based reranking—for building and deploying these 'agentic search' systems at scale, covering the architectural principles, production challenges, and the future trajectory where search itself may dissolve into understanding.

LLMOps for eval-driven development at scale

LLMOps for eval-driven development at scale

Mercari's engineering team shares their practical, evaluation-centric approach to LLMOps. Learn how they leverage tiered evaluations, strategic tooling for observability, and rapid iteration to productionize LLM features for over 23 million users, emphasizing that good 'evals' are often more critical than model fine-tuning or RAG.