Jeff Crume examines Jev, a new AI approach that makes structured decisions rather than generating text, through a cybersecurity lens. He argues that confidence scores are not evidence, that AI decisions expand the attack surface, and that observability, decomposed reasoning, independent controls, and human oversight are needed before trusting them.
A hands-on workshop from Langfuse on evaluating AI applications where mistakes are costly. Using a banking assistant as the running example, Annabell Schäfer and Doneyli De Jesus walk through the AI engineering loop: tracing with PII masking, building datasets, turning domain-expert comments into failure modes, mixing LLM-as-a-judge and code evaluators, and CI gates that block prompt changes when scores drop.
Remy Guercio of Tailscale argues that the era of unlimited AI budgets is over and that teams need to escape lock-in while proving real ROI. He breaks an internal AI deployment into four layers (models, data connectors, interfaces, and sandboxes) and presents an AI gateway as the way to keep each layer swappable. He demonstrates how Tailscale's Aperture uses tailnet identity instead of shared API keys, sets per-model budgets, serves as both an LLM and MCP gateway, centralizes authentication for coding agents, and provides logs for diagnosing agent behavior.
Doug Guthrie showcases how Braintrust connects observability with AI agent improvement. Using a support agent, he demonstrates setting up tracing, establishing evaluation baselines, and publishing custom scorers. The workshop delves into Braintrust's Topics feature for discovering unknown failure modes, configuring online automations, and importing real-world traces. A key focus is on automating the improvement cycle: using the Loop assistant for SQL-based trace analysis, integrating coding agents with Braintrust CLI skills to propose code changes, add regressions, run evals, and generate pull requests, thereby closing the feedback loop from production to development.
Modal CTO Akshat Bubna discusses the company's shift from developer to agent experience, highlighting why traditional cloud infrastructure fails for bursty AI workloads. He details Modal's primitives like elastic inference with GPU snapshotting and speculative decoding, agent sandboxes for RL rollouts, multi-node training with RDMA, and a "supercloud" strategy across 17 providers. The conversation also covers the importance of observability, hard guardrails for production agents, and AI's role in making infrastructure exciting again.
Sandipan Bhaumik presents a five-pillar framework for successfully moving AI systems from demos to production, inspired by a retail bank's failed chatbot PoC. The framework covers defining numerical success (Evaluation), tracing every AI decision (Observability), building robust data pipelines (Data Foundation), managing multiple AI interactions (Multi-agent Orchestration), and ensuring accountability and security (Governance). He illustrates these concepts with a banking chatbot case study, emphasizing continuous evaluation, data quality, and a proactive incident playbook.