Tokenless

The Caveman Prompting Challenge

James Barney, Head of Forward Labs at MetLife, discusses the operational challenges and strategic considerations for AI enablement in a large enterprise. Key topics include FinOps approaches to model selection based on cost, speed, and accuracy, optimizing LLM token usage through 'caveman prompting' and the proposed 'CaveBench' benchmark, and navigating volatile model release pricing. The conversation delves into measuring AI's value beyond cost, differentiating R&D from production token spend, and establishing robust AI governance using policy as code. A significant shift from traditional UIs to chat-based interfaces is explored, alongside the critical 'latent shopper problem' – the challenge of serendipitous data discovery in an AI-driven, query-focused world.

6 min read
The Caveman Prompting Challenge
source: https://www.youtube.com/watch?v=2QHE75oWT2w

The conversation features James Barney, Global Head of AI Enablement at MetLife, discussing the practicalities and challenges of integrating AI into a large, regulated enterprise. His role involves providing business partners with AI tools, navigating a complex landscape of global laws and regulations, and understanding customer rights in the application of new technology. Despite leadership responsibilities, James remains hands-on, actively pushing and reviewing code, emphasizing a problem-solving approach to AI adoption.

FinOps Foundation AI Working Group & Model Selection: James is also involved with the FinOps Foundation’s AI working group, which addresses the fundamental question of “what model to use for which workload.” With the proliferation of LLMs, this decision has become increasingly complex. The working group’s research consistently highlights a core trade-off between cost, speed, and accuracy. They aim to provide insights and resources, such as whitepapers and code repositories for token counting and chargeback systems, to help organizations make informed decisions and initiate conversations around optimizing LLM value.

“Caveman Prompting” and Token Efficiency: A significant theme revolves around “caveman prompting”—the art of using minimal words for maximum effect to save tokens, thus reducing costs. The discussion explores the limits of this technique, posing the idea of a “CaveBench” to benchmark how far prompts can be compressed before output quality deteriorates. This highlights the ongoing need to maximize the value-to-cost ratio in LLM interactions, saving money on both input and output tokens.

Model Releases and Pricing Strategies: Anthropic’s Fable model launch pricing, which offered a temporary price before June, is analyzed as a strategic “test drive.” This approach is interpreted as a way for model providers to gather early usage data, address initial issues, and manage demand, allowing developers to experience the model without immediate long-term commitment. This volatile landscape of model releases necessitates flexibility in planning for businesses, as new capabilities constantly emerge and “leapfrog” existing ones.

Measuring AI Value and Business Outcomes: Measuring the value of AI extends beyond simple cost savings. MetLife focuses on two key areas:

  1. Cost Avoidance: Quantifying how AI helps avoid expenditures elsewhere.
  2. Driving Business Outcomes: Identifying tangible unit metrics that align with revenue or KPIs. James highlights the “North Star” problem: initially, stakeholders might disagree on the primary metric, but by proposing one, others are prompted to articulate their true concerns, thus revealing all relevant unit metrics for value assessment. This aligns with FinOps principles of understanding the unit of value the business cares about. Tying token spend to these outcomes (e.g., “$10 per outcome”) allows for classic optimization conversations rather than abstract AI discussions.

Optimization Strategy: Start Smart, Then Refine: The recommended approach for AI development is to “start with the smartest model” for initial exploration and rapid prototyping. Once functionality is proven, the process shifts to optimization: “step down one level of a model” (e.g., from 300 billion to 70 billion parameters) and “sprinkle in a little bit more of a harness agent to harness capability.” This iterative process chips away at costs while maintaining quality. A unique aspect of AI is its self-tuning capability; one can ask the model itself how to improve its performance or identify inefficiencies, accelerating the optimization process significantly.

R&D vs. Production Tokens and Governance: The distinction between R&D tokens and production tokens is crucial for budgeting and classification. R&D tokens are likened to “personal productivity” or “local builds”—exploratory, experimental work that shouldn’t be immediately tied to production costs. Production tokens, conversely, are for workloads running in pipelines. While the lines can blur, particularly with rapid iteration and continuous integration, separating these allows for a more accurate financial understanding.

AI Governance and Policy as Code: Governing AI agents mirrors existing cloud governance principles. Key elements include:

  • Proactive Prevention: Implementing guardrails (e.g., preventing public S3 buckets) so that if a human can’t do it, an agent can’t either.
  • Reactive Checks: Automatically addressing deviations from policy (e.g., out-of-spec Kubernetes configurations).
  • Policy as Code: Defining policies in a machine-readable format allows agents to understand and adhere to rules without human intervention. The principle is: “If you wouldn’t let a human just go in and do it, then you certainly shouldn’t let an AI agent go in and do it.” This emphasizes applying established Software Development Life Cycle (SDLC) practices to AI.

The Obsolete UI and Chat as the New Browser: The modern graphical user interface (GUI) is increasingly perceived as obsolete. Chat interfaces and Multi-Cloud Platform (MCP) apps are emerging as the new “browser” or central hub for interaction. Instead of navigating multiple dashboards, users can ask a central AI agent questions, which then aggregates and processes information from various systems on demand. This shift allows AI to gather data differently—on-the-fly and as needed—rather than the human tendency to frontload mass data collection.

The “Latent Shopper Problem”: Despite the efficiency of direct querying, a critical challenge arises: the “latent shopper problem.” This refers to the loss of serendipitous discovery that occurs when browsing data without a specific question in mind, akin to walking into a store without a clear item in mind. Dashboards, with their visual anomalies (e.g., “what the heck is that spike over there?”), often inspire new ideas or reveal unknown problems. If AI only provides answers to explicit questions, the opportunity for browsing-driven insights might diminish. This highlights a future tension between direct AI queries and the value of visual data exploration for fostering creativity and identifying unforeseen issues. The conversation concludes with the importance of staying “intimate with the data” to gain these insights, acknowledging that manual exploration sometimes remains the best approach for true understanding.

Share post

Stay In The Loop! Subscribe to Our Newsletter.

Get updates straight to your inbox. No spam, just useful content.

Recommended for You

Reading Group July 2026 - Loop Engineering

Reading Group July 2026 - Loop Engineering

This session provides an in-depth exploration of Loop Engineering, a paradigm shift from manual AI prompting to designing autonomous systems that orchestrate AI agents. Speakers share practical experiences, from building production-grade platforms with automated code generation and adversarial AI reviews to experimental loop structures and foundational infrastructure layers. Key discussions address challenges like managing token costs, preventing agent chaos, and implementing robust verification mechanisms for industrializing software development.

Frontier results, on device - RL Nabors, Arize

Frontier results, on device - RL Nabors, Arize

RL Nabors discusses the significant costs associated with using frontier AI models, covering security, latency, and financial implications. She introduces a framework for right-sizing AI solutions by leveraging smaller, task-specific models and Small Language Models (SLMs). The framework details how to prove task feasibility, establish success criteria with golden datasets, conduct capability evaluations (using tools like Phoenix), and select the most appropriate "Small And Good Enough" (SAGE) model. Nabors further demonstrates how prompt engineering, particularly few-shot prompting, and post-processing can close performance gaps with larger models, while advocating for continuous regression evaluations to maintain performance integrity. The overarching message is to "prototype big, deploy small" to optimize AI deployments.