The Latency Goldilocks Zone Explained

iFood’s conversational AI, known as ILO-Agent, is an experience designed to move beyond simple keyword search. It functions as a conversational agent that can handle complex, open-ended requests, such as, “I want a romantic dinner for two, but my wife doesn’t eat onions.” The system understands the user’s preferences, context, and desires to provide the best real-time options. Currently, the agent operates reactively, but proactive recommendations are being developed, especially for the “undecided user” who is browsing without a specific goal.

Recommendation Beyond the Obvious

The core challenge in recommendation is moving beyond a user’s known history. Recommending sushi to a user who only orders sushi is simple, but it traps them in a filter bubble. The real task is to intelligently suggest something new that they might like. This requires understanding the fine-grained characteristics of each user, not just their order history.

iFood tackles this with a model called LCM, which creates a comprehensive user profile by considering numerous factors:

• Taste Profile: What cuisines and items they prefer.
• Economic Profile: Price sensitivity and typical budget.
• Contextual Factors: Time of day, location, and past interactions.

The system uses a combination of traditional machine learning, generative AI, and constant experimentation to find the shortest path between a user, a restaurant, and the perfect dish. It’s acknowledged that there’s no “silver bullet”; it’s a process of continuous trial and error to serve millions of users with vastly different tastes and budgets.

Innovation with “Jet Skis”

iFood employs an “ambidextrous organization” model to foster innovation. The main business is a “big ship,” which is stable but hard to maneuver. To explore new territories, they launch “Jet Skis”—small, fast, and inexpensive projects to test new theses. If a jet ski finds a promising direction, the company invests heavily; if not, the experiment is quickly shelved. This model has successfully launched entire business units, including fintech and grocery services. ILO-Agent began as one of these jet ski initiatives.

Initial results for ILO show it is:

• 16% faster in getting a user from search to checkout.
• Results in a 35% higher probability that a search converts to a cart addition.

Despite these successes, challenges remain in scalability, cost management, and shaping user behavior to leverage the agent’s full potential for complex queries.

The Latency Goldilocks Zone

A fascinating insight from iFood’s experiments is the “Latency Goldilocks Zone.” When building AI systems, the common engineering goal is to minimize latency. However, user perception is more complex.

• Too Fast: If an LLM responds instantly to a complex query, users may not trust the result. It feels like the AI didn’t “think” enough, much like a human giving a snap answer to a serious question.
• Too Slow: If the response takes too long, users lose patience, assume the system is broken, and abandon the interaction.

The ideal latency falls within a specific range (a 2022 study suggested 4-16 seconds for complex tasks) where the response time feels appropriate for the query’s complexity.

Perceived vs. Actual Latency

The user’s perception of latency is often more critical than the actual latency. The experience can be managed to make a 6-second wait feel instantaneous. iFood uses several techniques to achieve this:

• Streaming: Displaying the response as it’s being generated.
• Progress Indicators: Using animations or messages like, “I’m thinking about it for you,” to show the system is working.
• Engaging UIs: Showing an animation of the AI mascot “cooking” while the results are being fetched.

Designing for Multi-Channel Experiences

An AI agent’s interface must be tailored to the specific channel, as user expectations vary dramatically.

• Text (e.g., WhatsApp): Being an asynchronous channel, users are more tolerant of slightly longer delays. The interface can also support richer components like carousels and detailed text.
• Voice: This channel demands near-instantaneous responses. The agent must be concise and assertive. You cannot read a long list of ingredients or multiple options; instead, you might offer one or two curated choices and prompt for a “yes” or “no.” Voice has only a single channel for output, forcing a simplification of the information presented.

iFood’s architecture is built to be multi-channel, allowing for different configurations (e.g., verbosity, number of results) for each channel without rebuilding the core intelligence. The underlying LCM profile provides the foundation, but the final user experience is adapted.

Key Learnings and Future Directions

Building and scaling a production AI agent has yielded several important lessons.

1. Data Dependencies: An agent often relies on APIs from other parts of the organization. Before scaling, it’s crucial to establish clear agreements with data owners to ensure their systems can handle the increased load and avoid creating external bottlenecks.
2. Measuring Product-Market Fit: Standard metrics like the Sean Ellis test can fail in different cultural contexts. In Brazil, it was found that users’ answers to hypothetical questions about the product’s absence were unreliable. The team had to rely on deep, qualitative analysis of conversations to understand true user sentiment.
3. Agent-to-Agent Architecture: Users view iFood as a single entity and often use the ordering agent for customer support queries. This points toward a future where specialized agents collaborate. The system will need to either seamlessly hand off the user to another agent (e.g., “Let me introduce you to my friend who can help with that”) or have a master orchestrator agent that routes tasks behind a single, unified interface. For the user, the experience should be seamless, regardless of the internal architecture.

• Tags:
• Product Management
• LLM
• Latency
• User Experience
• Conversational AI
• Recommendation Systems
• IFood