Beyond the Chatbot: What Actually Works in Enterprise AI

Jay Alammar of Cohere shares insights from his journey of observing Large Language Models (LLMs) transition from research labs to practical enterprise applications. The key for enterprise adoption is to filter the immense hype and focus on established patterns that deliver real-world value.

Initial Enterprise Lessons with LLMs

The industry has learned several crucial lessons since the initial excitement around models like ChatGPT:

• Beyond Chat: The chat interface is not the ideal starting point for enterprise deployment. Instead, companies find more immediate value in simpler, more predictable use cases like summarization and data extraction. These should be treated as building blocks for larger applications.
• The Power of Representation: A significant portion of AI’s value is not generative. The representational power of embeddings is crucial for enhancing classification, categorization, and clustering, enabling companies to make sense of vast amounts of unstructured data.
• Grounding and Security: A primary challenge was overcoming model hallucinations and the tendency for models to confidently state incorrect information. This led to two critical developments:
  1. Retrieval-Augmented Generation (RAG): The practice of retrieving relevant context from a trusted data source before the generation phase to ground the model’s response in factual information.
  2. Private Deployments: For enterprises, grounding data is often internal and sensitive. This necessitates a “model-to-data” approach, where models are deployed within the company’s secure network to ensure data privacy and security, rather than sending data to an external API.

The Evolution of RAG into Agentic Systems

RAG systems have evolved significantly from their initial, simple implementations. This progression reflects an increasing complexity in the types of questions they are expected to answer.

• Simple RAG: The basic implementation involves a single search query against a data source, with the top results fed into the model’s context.
• Query Rewriting: An early enhancement was to use an LLM to rewrite the user’s initial question into a more precise query for the retrieval system. This makes the generative model an active part of the retrieval stack.
• Multi-Query RAG: For comparative questions (e.g., “Compare Nvidia’s financial results in 2020 vs. 2023”), the system learns to execute multiple searches in parallel (one for 2020, one for 2023) and synthesize the results.
• Sequential RAG (Agents): More complex questions require a multi-step process. For a query like, “Who are the top car manufacturers in 2024, and do they make EVs?”, the system first identifies the manufacturers and then sequentially queries for EV production for each company on the list.

This evolution from simple retrieval to multi-step, tool-using systems marks the transition from RAG to what are now considered LLM-backed agents. These agents can decide when they have gathered enough information and can execute tasks successively.

Context Engineering and Evaluation

• Exploiting Document Structure: A key design pattern is to leverage the inherent structure of data sources. During ingestion, an “onboarding phase” can be used where a model analyzes a data source (e.g., a codebase, a database schema) to create a metadata summary. This summary, which might include the tech stack of a repository or table definitions, is then used as a high-level context, making interactions more efficient than stuffing millions of tokens into a context window.
• Evaluation as IP: Evaluation is one of the most critical internal capabilities a company should develop. Creating internal evaluation datasets with real-world examples is core intellectual property. These act as the “unit tests” for AI systems, allowing companies to:
  • Objectively compare different models and vendors for their specific use cases.
  • Detect regressions when a new model version is released.
  • Ground the team’s expectations and define success criteria.

Key Applications and Future Directions

• AI-Assisted Code Generation: This is a major area of focus. The field is maturing from solving simple, LeetCode-style function-writing problems (like the HumanEval benchmark) to tackling complex tasks based on entire GitHub issues (like the SWE-bench benchmark).
• Future of Foundation Models:
  • Multimodality: The next horizon for enterprise applications, particularly for companies dealing with documents like PDFs that contain a mix of text, charts, and images.
  • Text Diffusion Models: An under-the-radar technology that could change the paradigm of text generation. Unlike auto-regressive models that generate token-by-token, diffusion models generate a complete output (initially as noise) and iteratively refine it. This could lead to dramatic increases in generation speed and new model behaviors.
• The Role of Smaller Models: Smaller, more specialized models are catching up in capability at a rapid pace. For many well-defined enterprise tasks, they will prove sufficient. Deploying smaller models for specific tasks can lead to significant savings in cost, latency, and compute resources, making them a vital part of a company’s AI strategy.

• Tags:
• Code Generation
• LLM
• Evaluation
• RAG
• Feature
• Enterprise AI
• Agents
• Cohere