9 Lessons Learned from Deploying GenAI at Scale • Garth Gilmour & Stuart Greenlees • GOTO 2025
Drawing from their experience at Liberty Mutual, a Fortune 100 company, Garth Gilmour and Stuart Greenlees share nine hard-won lessons from deploying Generative AI for 5,000 developers. This "from the trenches" talk moves beyond the hype to discuss the real-world challenges of scaling AI, including managing spiraling costs, the complexities of RAG, the difference between shipping and adoption, and the necessity of building a governed platform. They detail their mistakes, solutions, and the evolution of their strategy for architecture, developer education, and model management.
This talk shares nine “warts and all” lessons learned by Liberty Mutual while deploying Generative AI at scale for its 5,000 developers. The speakers, Garth Gilmour and Stuart Greenlees, emphasize that the journey is messy, chaotic, and rarely follows a linear path. Their mantra, inspired by their CTO, was to “get our knees skinned”—accepting failures and learning from them.
1. Learning Never Stops
Even with a research team monitoring emerging technologies, the sudden release of ChatGPT in late 2022 was a surprise. The key lesson is to continuously learn and maintain agility, likening it to paddling to stay on top of a wave. For a large enterprise, it’s crucial to leverage existing partnerships (like their one with Microsoft) to ensure governance, security, and established legal frameworks when adopting new technologies quickly.
2. Shipping Isn’t Adoption
The team successfully rolled out their internal “LibertyGPT” to 70% of the company and GitHub Copilot to 90% of developers. However, they realized that providing access doesn’t guarantee effective use. Initial usage was basic, with employees treating the tools like a better search engine rather than a “thought partner.” To drive deeper adoption, they moved from spontaneous, organic learning events (like promptathons) to a formal, structured “Gen AI Learning Mission” with dedicated learning streams for different roles, from developers to people leaders.
3. Models Aren’t Equal
The initial assumption that Large Language Models (LLMs) were interchangeable commodities proved false. They discovered nuances where different models excelled at different tasks. For example, one model might oversimplify technical jargon while another preserves it, a critical distinction for certain use cases. Their solution was to build a platform that supports a plurality of models (28 at the time), allowing teams to choose the right tool for the job by balancing performance, capability, and cost, rather than defaulting to the most expensive “frontier model.”
4. Scaling is a Beast
Unlike cloud adoption, scaling GenAI presented unique challenges. Access to new models was often gated by providers due to quota limitations. The internal rollout faced hurdles with regional regulations, data sovereignty, security threats like prompt injection, and rate limiting. To manage this complexity, they built a centralized AI platform. This platform provided controlled access to scarce resources, a single pane of glass for observability and cost management, and a consistent way to apply security guardrails. While engineer autonomy remains a priority, it must exist within this safe, governed framework.
5. Today’s Fix is Tomorrow’s Pain
Many custom solutions built today will become obsolete as vendors roll out their own managed services. Liberty Mutual built a “summarizer of summarizers” service to handle large documents, which was later made redundant by models with larger token limits. The lesson is to recognize that much of the current work is temporary scaffolding. The recommendation is to build a composite architecture, be prepared to “kill your darlings,” and focus engineering efforts on unique, business-differentiating use cases.
6. Stay Out of the Weeds
When creating their “GenAI Northstar Architecture,” the team of 12 architects got stuck in endless debates, leading to “13 opinions.” They realized that chasing perfection was counterproductive. The solution was to time-box the debate, publish a “good enough” version of the architecture, and iterate. This allowed them to get alignment, secure investment, and start building, proving that progress is more valuable than perfection.
7. Costs Can Spiral
GenAI’s power to process vast amounts of data can lead to astronomical costs. One use case burned through $20,000 in a few hours. The key takeaway is that “increasing performance increases the ability of your system to incur cost.” The solution is to implement robust observability, including custom dashboards for cost tracking, automated alerts, and educating engineers on the financial implications of their model choices. Reviving old-school patterns like batch processing for non-urgent, expensive tasks is also recommended.
8. RAG is Really Hard
While tutorials make Retrieval-Augmented Generation (RAG) seem straightforward, achieving high-quality, trustworthy results is complex. Moving beyond basic implementations requires deep knowledge of embeddings, vector stores, re-ranking, and chunking strategies. A critical challenge is ensuring “groundedness”—providing citations and traceability to prove the AI’s response is based on the provided source material and not a hallucination. Success requires building an automated evaluation pipeline with a “golden dataset” to systematically test and tune the many variables involved.
9. Latency is Real
The team initially assumed users would tolerate a 15-second wait for a novel GenAI-powered feature. They were wrong; users reverted to the faster, older applications. The solution was a two-pronged approach: improving the user experience with progress indicators and asynchronous processing, and technical optimization (e.g., GPU compute, better vector databases) to reduce the response time to 3 seconds. The lesson is that user perception of speed is critical, and raw capability doesn’t excuse a poor user experience.