How a Logistics Giant Keeps AI Data Locked Down

Jason Ward, FinOps for AI at C.H. Robinson and a member of the FinOps Foundation’s AI working group, addresses the critical challenge of managing AI costs, particularly for startups with burgeoning AI engineering teams. His approach emphasizes value over sheer spend and leverages AI to gain deeper insights into AI usage.

Foundational FinOps for AI Practices

Ward’s primary advice for managing AI spend is to tag every AI resource (whether it’s a specific model, application, or agentic workflow) to establish clear ownership. This foundational step enables subsequent analysis. He stresses that focusing solely on spend is insufficient, as efficiency and value are paramount. For instance, a more expensive but highly efficient model could be more cost-effective than a cheaper, slower alternative. He suggests measuring value through metrics like “cost per hour saved” for tools like GitHub Copilot, especially given the current “billing blind spot” for such services where direct cost attribution is difficult.

Cost breakdown is another key area, advocating for analysis by AI application and by individual model. This granular view empowers developers to select the most efficient models for their specific workloads.

AI Use Cases and Platform Strategy at C.H. Robinson

C.H. Robinson, a leading third-party logistics company, integrates AI extensively across its operations, including order entry, email classification, pricing, quoting, booking, and tracking loads. Their primary AI platform is Azure OpenAI. However, for specific needs, they utilize GCP Vertex AI to route Anthropic models.

This multi-cloud approach is driven by a strong emphasis on data security. Routing Anthropic models through Vertex AI ensures that C.H. Robinson maintains strict control over its data, preventing it from being shared with external entities. Ward likens this to the need for on-premise GPU solutions in highly sensitive or governmental contexts, underscoring the importance of keeping proprietary data isolated.

Leveraging AI for AI Observability (“AI Hacking AI”)

One of Ward’s most innovative strategies involves using AI to enhance AI observability. He employs an LLM observability platform equipped with an LLM agent to discover metrics that weren’t initially apparent. This “AI hacking AI” approach helps uncover signals critical for optimization.

Key metrics identified through this method include:

• Model Chattiness/Verbosity: This measures the token consumption per API call or trace, indicating how much a model “talks” to achieve a result. High verbosity can signal inefficiencies like “prompt bloat.”
• Prompt Bloat Detection: By analyzing verbosity, Ward’s system identified instances of excessive prompt length, which, in one case, exposed a bug in the code causing an infinite retry loop and unnecessary token consumption. This effectively turned the observability platform into a “QA tool for the coding team.”
• Context Starvation: Conversely, this metric identifies scenarios where models are provided with insufficient context, leading to repeated failures and retries, which also wastes tokens. Correcting context starvation directly improves model efficiency and reduces spend.

Advanced AI Cost Metrics and Efficiency Beyond Discrete Tasks

While cost per order and cost per automated task are straightforward and provide clear ROI for discrete activities, Ward acknowledges that these represent only a small fraction of C.H. Robinson’s AI workloads. For the more complex and non-discrete tasks, he is developing and tracking advanced, experimental metrics:

• Agentic vs. Conversational Workloads: Ward highlights the necessity of differentiating these two workload types. Agentic workflows, which involve multi-prompt or looping processes, can appear as false positives for “prompt bloat” if not separated from single-shot, conversational prompts. This distinction allows for more accurate metric interpretation.
• Cost per Thought (Experimental): This efficiency metric aims to quantify the cost of a complete AI “thought” or processing cycle. Defined as the total cost divided by the entire trace (encompassing RAGs, tokens, and conversation), a “thought” spans from the first input token to the last output token in a completion. Ward admits this is still an evolving concept requiring more data for robust benchmarking.
• Reasoning Ratio: This metric quantifies the proportion of tokens used for active reasoning versus non-reasoning tasks. Currently, it’s a relatively crude measurement, often based on whether a reasoning-focused model is employed for a session.
• Cache Hit Rate: A critical metric for efficiency, it tracks the percentage of prompts served from a cache, which can yield substantial cost savings (60-70%). Observability platforms automatically monitor this.

Future Directions and General Advice

Ward is exploring dynamic model routing, where an intelligent system would select the most appropriate and cost-effective model for a given task. His overarching advice for FinOps for AI practitioners is to “use your AI to improve your AI.” This means leveraging AI tools to uncover relevant metrics, understand workflows, and optimize resource utilization. He stresses that understanding the underlying workflow and collaborating with development teams are crucial for effective reporting and value demonstration. The ultimate goal is to monitor the bulk of AI operations and demonstrate quantifiable value across all AI initiatives.

• Tags:
• Agentic AI
• Prompt Engineering
• Feature
• LLM Observability
• Vertex AI
• FinOps
• Data Security
• AI Cost Management
• Azure OpenAI
• Cost per Thought
• Reasoning Ratio
• Cache Hit Rate
• Cloud FinOps