Frontier results, on device - RL Nabors, Arize

RL Nabors discusses the significant costs associated with using frontier AI models, covering security, latency, and financial implications. She introduces a framework for right-sizing AI solutions by leveraging smaller, task-specific models and Small Language Models (SLMs). The framework details how to prove task feasibility, establish success criteria with golden datasets, conduct capability evaluations (using tools like Phoenix), and select the most appropriate "Small And Good Enough" (SAGE) model. Nabors further demonstrates how prompt engineering, particularly few-shot prompting, and post-processing can close performance gaps with larger models, while advocating for continuous regression evaluations to maintain performance integrity. The overarching message is to "prototype big, deploy small" to optimize AI deployments.

June 29, 2026 6 min read

Frontier results, on device - RL Nabors, Arize — source: https://www.youtube.com/watch?v=fWXJM-J0ZB8

The High Cost of Frontier Models and How to Right-Size Your AI Stack

RL Nabors begins by highlighting the substantial costs associated with relying on large, frontier models like GPT-5 or Claude. These costs are multifaceted, impacting security, user experience, business finances, and environmental sustainability.

The Hidden Costs of One-Size-Fits-All Inference

Security: Sending data to remote cloud-based LLMs carries inherent risks of exposure, interception, and retention by third parties. Cases exist where sensitive business data was breached due to remote AI chatbot usage.
Latency: High latency negatively impacts user experience. Research indicates that users perceive a 4-second delay as the limit of believability for AI interactions, a threshold many calls to large models often exceed.
Business Costs: Third-party inference costs are difficult to control and can escalate rapidly. While individual token prices may fall, agentic reasoning workloads consume tokens at such a high rate that total inference spend is rising.
Offline Limitations: Remote models are unusable without internet connectivity, limiting productivity in secure environments or areas with poor Wi-Fi.
Environmental Impact: Large models consume significant energy, contributing to a larger carbon footprint.

The Solution: Small Language Models (SLMs) and Task-Specific Models

Nabors advocates for replacing frontier models with smaller, task-specific models or Small Language Models (SLMs) where appropriate. These models offer significant advantages:

SLM Characteristics: SLMs contain millions to billions of parameters, a stark contrast to the trillions in large LLMs. This vast difference in parameter size translates to much lower computational requirements.
Energy Efficiency: SLMs consume significantly less energy—approximately 25% of an LLM for the same task, and task-specific models about half of that. This leads to substantial energy and cost savings.
Production Readiness: SLMs are increasingly production-ready, with research (e.g., Nvidia in 2025) indicating their sufficient power for agentic tasks.
Deployment: Many SLMs can be deployed with quantization (8-bit, 4-bit), drastically reducing disk and memory requirements (e.g., 1 billion parameters fitting in about 2GB FP16). This allows for on-device deployment, such as the Pixel Pro shipping with an embedded SLM.
Benefits: SLMs offer enhanced security (data stays on device), offline functionality, no per-call fees, improved efficiency, and lower latency due to the absence of network round trips.

A Framework for Model Selection: “Prototype Big, Deploy Small”

Nabors outlines a four-step framework, co-developed with Google, for right-sizing AI models:

Prove it’s Possible: Start by using the largest, most capable model (e.g., a foundation model like Claude or Gemini) to determine if the desired task is achievable at all. If the frontier model can do it, a smaller model likely can too.
Set Success Criteria and Collect a Golden Dataset: Define clear metrics for success. Create a golden dataset: a curated, high-quality collection of human-labeled input-output pairs that serve as the ground truth for evaluation. For a summarization task, this includes defining metrics like JSON validity, reference structural validity, factual consistency (often judged by an LLM or humans), length compliance, and latency (P50, P95).
Test from Small to Large: Utilize evaluation tools like Arize’s open-source Phoenix to perform capability evals. Compare the performance of various smaller models against a baseline (e.g., Claude Sonnet). The goal is to find the smallest model that meets acceptable performance criteria.
Select the SAGE Model: Choose the “Small And Good Enough” (SAGE) model—the smallest model that provides acceptable responses for the specific use case. Nabors’ own experiment for a social feed summarization feature (Mima) showed Llama 3.2 (3 billion parameters) outperformed Gemma 4 E2B (5 billion parameters) in terms of speed and accuracy, getting close to Claude Sonnet’s performance while being significantly cheaper (zero cost for on-device inference).

Closing the Performance Gap with Prompt Engineering

Even after selecting a SAGE model, further optimization is possible through prompt engineering. Nabors details an iterative process:

Isolate Variables: Test one prompt variant at a time to determine its impact.
Prompt Variants: Nabors experimented with several prompt types for Llama 3.2:
- Original: The baseline prompt.
- Numbered Input: Reformatting the input thread as numbered messages, hypothesizing smaller models track natural language indexing better than JSON array offsets.
- Few-Shot: Adding examples of input-output pairs, based on the hypothesis that small models learn format from examples faster than rules. This proved to be the most effective, significantly improving length compliance and factual consistency with minimal latency increase.
- Strict Rules: Explicit negative constraints (e.g., “No preamble”). This surprisingly made performance worse, as the model responded negatively to rigid negative commands.
- Chain of Thought: Forcing the model to identify key moments before summarizing. This offered slight improvements in length but increased latency.
Post-processing: Simple post-processing steps (e.g., truncating summaries if too long, validating reference counts within the harness) can further bridge the gap, achieving near-perfect results for structural validity and closing remaining discrepancies.
LLM-as-Judge Caveats: Nabors notes that when using an LLM to judge model outputs (e.g., Claude judging Llama’s summaries against Claude’s own), the judge LLM might exhibit bias, favoring outputs closer to its own generation. It’s crucial to inspect raw eval results to understand the nature of disagreements.

Through this process, Llama 3.2, combined with few-shot prompting and post-processing, was able to meet and even beat Claude Sonnet’s performance for the summarization task, while incurring zero inference costs on the user’s device.

Sustaining Performance with Regression Evals

To prevent future performance degradation when models or prompts are updated, Nabors emphasizes the importance of continuous regression evals. These evaluations, similar to CI/CD tests, ensure that changes do not negatively impact critical metrics (e.g., causing summaries to hallucinate or expand excessively).

Call to Action

Nabors challenges the audience to re-evaluate their current reliance on frontier models and identify opportunities for adopting SLMs. He suggests considering on-device models, such as Gemini Nano accessible via Chrome’s Prompt API, and utilizing tools like Arize Phoenix for building and running evaluations. The core message is to “prototype big and deploy small” to achieve cost-effective, secure, and efficient AI solutions.