You can't just one shot it — Mehedi Hassan, Granola
A product engineer from Granola shares a candid account of the challenges in moving AI features from the playground to production. This talk covers the pitfalls of "one-shot" solutions like web search and generic prompts, and details Granola's strategy of building custom internal tracing and development tooling to create a tight, effective feedback loop for iteration.
The journey of integrating AI features into a production application is fraught with challenges that extend far beyond the initial “it works” stage in a development playground. The idea of “one-shotting” a feature—adding a single line of code for web search or writing one prompt for all users—is seductive but ultimately fails to address the complexities of real-world use.
The Pitfalls of “One-Shot” AI Features
At Granola, a meeting notes application that provides real-time transcription and AI-powered summaries, the team encountered significant hurdles when productizing a seemingly simple chat feature. User feedback quickly revealed the limitations of a generic approach, with complaints ranging from slow web search to a lack of personalization and irrelevant responses.
The Deceptive Simplicity of Web Search
For many LLM providers, integrating web search appears as a simple tool call. However, this simplicity masks underlying issues that emerge at scale:
- Cost and Context Explosion: Complex queries can cause token usage to skyrocket, blowing up the context window and making each chat prohibitively expensive (e.g., 10 pence per chat), which is not feasible for millions of users.
- Provider Volatility: The performance of web search is often at the mercy of the LLM provider. An overnight update from a provider once led to a sudden degradation in Granola’s web search results, an issue that was completely out of their control.
- Inherent Complexity: The existence of billion-dollar companies dedicated solely to web search underscores that it is a far more complex problem than adding a single tool to an LLM pipeline.
The Personalization Challenge: One Prompt to Rule Them All?
A single, generic prompt cannot adequately serve the diverse needs of a varied user base. An LLM’s output needs to be tailored to the user’s role and expectations:
- A salesperson might expect a deal-focused summary.
- An engineer might look for action items, blockers, and Linear tickets.
- An HR manager requires an entirely different type of output.
LLMs can be “stubborn,” and their behavior is often a black box. The key is to find ways to get inside that box and mold the output to specific use cases.
The Solution: Building Machinery for Iteration
Instead of trying to perfect a single prompt, Granola’s approach was to build the internal machinery required for rapid and insightful iteration. The goal was to create a tight feedback loop.
Custom Internal Tracing: Opening the Black Box
To gain visibility into the LLM’s behavior, the team built its own tracing tools. This was made feasible by using LLMs to help build the tools themselves. This system provides end-to-end visibility into the AI’s process:
- Comprehensive Insight: It exposes individual tool calls, the reasoning behind them, search trails, and cost, all structured precisely for Granola’s needs.
- Democratized Data: The UI was built for everyone in the company—product, data, CX—not just engineers. This eliminates the need for complex CloudWatch queries to diagnose a failure, allowing even the founder to trace an agent loop from start to finish to understand what went wrong.
- From “Off” to “Actionable”: This tooling transforms a vague feeling that “the output feels off” into a precise understanding of which component failed, enabling targeted improvements.
Accelerating Development: From Desktop App to Web Shell
LLM features require constant experimentation with multiple variants. Granola’s status as an Electron-based desktop app created friction in testing, as developers could only run one instance at a time and sharing changes with coworkers was cumbersome.
To solve this, they re-architected their application:
- Web-Based Previews: The Electron front-end was converted into a web shell that could be deployed online. Now, every pull request automatically generates a preview link, dramatically speeding up development and testing cycles.
- Technical Implementation: This was achieved by abstracting the IPC APIs (which handle system-level interactions) to fall back to web standards in a web environment. Similarly, React components like routers and session managers were moved to web standards, making the renderer agnostic of its Electron container.
- AI-Powered Testing: A powerful side effect of this change is the ability to use tools like Cursor to automatically test new features in the preview environment and upload screenshots directly to the pull request, further accelerating the feedback loop.
Conclusion: A Feedback Loop for Magical Products
The ultimate lesson is that the answer isn’t to “one-shot better.” It’s to build a robust feedback loop that feels like “playing a tennis game with the LLM.” By investing in custom tooling and a faster development cycle, teams can move from hoping a feature works to having the conviction that what they are shipping will truly connect with users and feel less like a black box and more like magic.