In Case You Missed It in August 2026 (Ep. 1024 with Jon Krohn)

In ICYMI Episode #1024, the discussion spans the critical gap between AI investment and return, exploring both technological and human-centric challenges and solutions.

The AI ROI Gap: Why Investments Don’t Always Yield Returns

Pete Johnson, MongoDB’s Field CTO of AI, highlights a significant disconnect: 83% of organizations establish AI steering committees and success metrics, yet only 19% achieve a matching return on investment (ROI). This gap stems from organizations picking the wrong problems to apply AI to.

He identifies two common traits among companies successfully generating ROI:

• Employee-facing applications: These projects have a lower bar for data security and quality compared to customer-facing ones. Measuring ROI is also more straightforward as employee performance metrics are already established, allowing direct attribution of gains to the AI agent.
• Human-in-the-loop systems: Solutions are not fully autonomous, ensuring human oversight and intervention.

Johnson stresses the importance of measurable success metrics, advocating for business-centric outcomes like “shipping code faster” rather than vanity metrics such as “lines of code” or “number of tokens generated.” He cites examples from Uber and Meta where focusing on token consumption led to inefficiency and budget overruns. The rapid innovation cycle in AI makes it challenging for engineers to keep up, requiring constant learning and adaptation. Johnson personally manages this by engaging with over a hundred companies across six countries, gaining diverse insights into effective RAG pipelines, agentic memory, and retrieval quality.

Mathematical Optimization in the Agentic AI Era

Jerry Yurchisin of Gurobi Optimization distinguishes between the capabilities of Large Language Models (LLMs) and mathematical optimization solvers, particularly in high-stakes decision-making. While LLMs excel at generating “highly likely responses,” they inherently lack the rigor to guarantee adherence to “hard constraints.” Agents, despite their confident assertions, can easily overlook critical constraints (e.g., environmental regulations) leading to severe financial penalties.

Mathematical optimization, in contrast, guarantees constraint satisfaction, providing a trustworthy framework for decisions. Yurchisin proposes a division of labor:

• Agents’ Role: To aid in problem definition, identify relevant regulations, assist in code generation, and formulate mathematical models.
• Optimization Solvers’ Role: To execute the solving process, deliver optimal solutions, and offer explainability and defendability. This allows answering fundamental business questions like “why this, why not that?” by quantifying the trade-offs (e.g., cost differences between production facilities).

Structuring AI Skills and Budgeting for Success

Priyanka Vergadia, a best-selling author and former Microsoft/Google AI leader, shares practical advice for leveraging generative AI, specifically focusing on structuring skills in tools like Claude to produce high-quality output and a strategic budget framework.

Structuring Claude Skills for Quality Output

To avoid “slop” when using LLMs for tasks like blog writing, Vergadia advocates breaking down complex tasks into granular subtasks, much like a human would approach them. Her method involves:

1. Thorough Research: Defining specific prompts to ensure high-quality, authoritative sources (e.g., “high-quality content written by researchers and scientific research from schools and universities”).
2. Human-in-the-Loop Synthesis: After research, the AI provides a summary (e.g., via text or audio) for human review and refinement, ensuring crucial decisions remain human-controlled.
3. Iterative Content Creation: Instead of asking for a full blog post, she requests an outline first. Subsequent steps involve adding storytelling, analogies, and specific formatting requirements (e.g., three developer-focused images, a practical example).

This structured approach aims for an “80% there” output, significantly reducing post-generation editing.

The 10-20-70 AI Budget Framework

Vergadia introduces a framework for AI budget allocation to maximize ROI and foster effective AI adoption within enterprises:

• 10% on Tools: Investing in AI software and platforms.
• 20% on Execution: The direct use of these tools.
• 70% on Education and Skilling: This largest portion is dedicated to upskilling employees. She argues that AI adoption is a “habit” that requires sustained effort, not just tool acquisition. The prevalent issue of low AI production use (8-9% in a Stanford report) is often due to a lack of effective training.

She emphasizes the need to show employees effective ways to use AI for their specific job functions, rather than just telling them to use it. Building internal communities where employees share practical use cases and experiences is vital. While a significant upfront investment, this “J-curve” trajectory ultimately leads to widespread, effective AI integration, enabling the creation of specialized agents across business units and transforming company-wide efficiency.

The Semantic Layer: Crucial for Analytics Agents

Tristan Handy, CEO of dbt Labs, explains the increasing importance of the semantic layer, especially with the rise of analytics agents. For small organizations, a semantic layer might be overkill, but for large enterprises (e.g., Siemens with 2,000 data engineers serving 300,000 employees), it’s essential for scaling knowledge and ensuring consistent data interpretation.

Historically, answering complex organizational questions required lengthy meetings to establish consensus on measurement. The semantic layer provides tooling to standardize these decisions and store them, enabling successive queries to be answered consistently without constant re-negotiation. It solves an organizational problem by providing a common “lingua franca” for data meaning.

For analytics agents, a semantic layer is critical because without it, agents must “rederive” how to measure things. This process is both “slow and expensive” due to excessive token usage, and prone to errors, often leading to answers that, while plausible, don’t align with the organization’s established metrics. The semantic layer thus extends seamlessly into an agent-driven world, enhancing efficiency and accuracy.

Handy outlines a “knowledge or meaning hierarchy”:

1. Raw data
2. Modeled data
3. Semantic layers: Define how tables join and how metrics are measured.
4. Ontologies: An even deeper layer of meaning, modeling business processes and causation. Ontologies are highly specialized, typically used in high-value domains like genetic research or defense, where the context is stable and changes less rapidly.

• Tags:
• Agentic AI
• RAG
• Feature
• Enterprise AI
• Vector Search
• Upskilling
• Data Science
• Semantic Layer
• Mathematical Optimization
• AI ROI
• Data Governance
• AI Budget
• Claude Skills
• Dbt
• LLM Constraints