How We Cut LLM Latency 70% With TensorRT in Production

The AI Iceberg: From Proof-of-Concept to Production

Moving beyond the initial excitement of AI proof-of-concepts requires tackling the “AI Iceberg”—the vast, invisible work of managing performance, latency, throughput, accuracy, and cost at an enterprise scale. The journey from a simple chatbot to a production-ready, self-hosted AI system involves a steep learning curve, especially for leaders and teams without prior deep AI experience. This requires a mindset of continuous learning alongside the team, focusing on finding the “AI nexus”—the optimal balance between these competing priorities for a specific use case.

GPU Infrastructure and Performance Optimization

A significant focus was placed on optimizing the entire GPU stack to reduce costs and improve user experience. This involved several key strategies:

Latency Reduction with TensorRT-LLM

NVIDIA’s TensorRT-LLM was a critical tool, reducing inference latency by 50-70%. It works by rewiring a model’s neural network to be optimally adapted to the specific hardware architecture of the GPU (e.g., an H100). This optimization provides a massive performance boost on the same hardware.

KV Cache and In-Flight Batching

The team’s strategy for deploying models on GPUs evolved significantly. The initial thought was to fit multiple smaller models onto a single GPU to increase throughput. However, a more effective, counterintuitive approach was discovered:

“Instead of trying to do two or three models in one GPU, I’m going to do one model on the same GPU and use everything else for KV cache, and I’m going to have a huge throughput.”

By dedicating one model per GPU, the remaining video memory is used for a large KV cache. This, combined with in-flight batching enabled by TensorRT-LLM, allows for continuous processing of requests. New requests can be added to a batch as soon as others are completed, without waiting for the entire batch to finish, dramatically improving throughput.

Solving the Cold Start Problem

To support dynamic scaling, minimizing the “cold start” time for new GPU instances was crucial. Two primary solutions were implemented:

1. Faster Storage: Switched from loading models from Amazon S3 to AWS FSx, a much faster storage solution, to accelerate download times.
2. Baking Models into Images: Instead of spinning up a container and then downloading the model, the model was “baked” directly into the container image. This eliminated the download step entirely, cutting minutes off the spin-up time for a new Kubernetes node.

Cost Management and Scaling Strategies

Managing the high cost of GPUs was a top priority, leading to several innovative approaches to prove a positive ROI.

Counterintuitive Cost Savings

The team found that upgrading to more expensive, powerful GPUs actually saved money. The logic was simple:

“If I’m going to go from this GPU to the next GPU, it’s going to be 30% more expensive, but I’m going to use 50% less of the time… it’s going to be like 30% cheaper.”

By processing requests faster and achieving higher throughput, the more expensive GPUs were needed for fewer total hours, resulting in significant net savings. This allowed the team to use larger, more capable models, improving quality while simultaneously reducing costs.

Scheduled and Dynamic Scaling

To align GPU usage with traffic patterns, a two-pronged scaling strategy was implemented:

• Scheduled Scaling: For a predictable HR tech workload, the number of active GPUs is automatically reduced during low-traffic periods like nights and weekends.
• Dynamic Scaling: To handle unpredictable spikes, such as at the end of a quarter, the system can dynamically and rapidly spin up new GPU instances.

This combination ensures that the company only pays for the compute power it needs, when it needs it.

Building an AI Platform

The architectural approach shifted from building siloed AI features to creating a unified, horizontal AI platform.

• Vertical Capabilities: Initially, AI was integrated into specific product verticals like goal setting, performance feedback, and 360 reviews.
• Horizontal Platform: A horizontal layer was built on top of these verticals. This platform can orchestrate the individual AI capabilities to create complex, multi-step workflows. For example, it can generate a 30-60-90 day plan for a new manager by pulling insights from the goals, feedback, and skills data across the entire system.
• Intelligent Routing: The future vision includes an “LLM Proxy” or AI gateway that intelligently routes requests based on AI-specific metadata (e.g., pre-fill vs. decoding needs), rather than just traditional load-balancing metrics.

The Culture of AI in Engineering

Adopting AI coding agents required navigating a significant cultural shift within the engineering team.

The “AI Engineering Lab”

An internal “AI Engineering Lab” was created to allow for open exploration of AI coding tools. This revealed a difference in adoption based on seniority:

• Junior Engineers: Were highly enthusiastic, seeing AI as a way to accelerate their learning and build complex systems they otherwise couldn’t.
• Senior Engineers: Were initially skeptical, concerned that AI would lead to low-quality code that didn’t adhere to established design patterns. They noted that AI-generated code often had a “smell to it.”

Redefining the SDLC

This tension led to a collaborative redefinition of the Software Development Lifecycle (SDLC). The consensus became that engineers must use the time saved by AI to perform more rigorous reviews. The output of work with AI must meet the same quality bar as work done without it. Senior engineers took the lead in creating “skills” and instructions for the coding agents, teaching them to respect the organization’s specific components and design patterns. This process has improved the quality of AI-generated code and led to widespread, effective adoption across the team.

• Tags:
• MLOps
• Cost Optimization
• Agentic Coding
• AI Platform
• TensorRT-LLM
• GPU Optimization
• Self-Hosting
• LLM Inference