Building an Orchestration Layer for Agentic Commerce at Loblaws

The Challenge: From Agent Prototypes to Production-Grade E-commerce

Prototyping AI agents is straightforward, but deploying them reliably in a complex, mission-critical enterprise environment like Loblaw—one of Canada’s largest retailers—presents significant MLOps hurdles. The core challenges include:

• System Complexity: Integrating with a vast ecosystem of over 50 platform APIs and hundreds of endpoints for services like search, recommendations, cart management, and inventory.
• Production Reliability: Ensuring agents can operate securely, privately, and consistently across diverse and disparate tech stacks.
• Scalability and Reusability: Enabling over 50 internal teams to build and deploy similar agentic applications without reinventing the wheel, ensuring a consistent and scalable approach.

To address these challenges, we developed Alfred, a production-ready agentic orchestration layer designed to serve as a helpful assistant for building and running any agentic workflow across Loblaw’s digital properties, from e-commerce to loyalty and healthcare.

Alfred: A Reusable Architecture for Agentic Systems

Alfred provides a standardized, reusable pattern for agentic applications, built on a foundation of battle-tested open-source tools and Google Cloud Platform (GCP) components.

Core Tech Stack:

• Orchestration: LangGraph for building the agent execution graph.
• Model Abstraction: LiteLLM to act as a gateway to various LLM providers like OpenAI and Google Gemini, allowing for easy switching and budget management.
• Observability: LangFuse for prompt management, traceability, and debugging.
• Backend Service: FastAPI running on Google Kubernetes Engine (GKE).
• State Management: AlloyDB (a managed PostgreSQL on GCP) for checkpointing and managing conversational context.

Key Architectural Components and Responsibilities

Alfred is designed as a foundational layer that provides teams with the necessary tools and guardrails to build applications quickly and safely.

1. Templated Scaffolding: New projects are bootstrapped using copier, a Python templating library. This provides teams with a ready-to-use repository containing the barebones project structure, CI/CD pipelines, testing frameworks, and even a pluggable gradio-based chat UI for easy debugging of conversational flows.

2. Secure and Observable Tool Calling: Alfred enforces a standard schema for communication and provides centralized handling for security patterns like token validation. All tool calls and LLM interactions are tracked through LangFuse, with robust PII (Personally Identifiable Information) masking and encryption applied at two levels: before data is sent to LLM providers and within the observability platform itself.

3. The Model Context Protocol (MCP) Ecosystem: MCP is the cornerstone of the architecture, serving as a gateway to the complex web of internal platform APIs. Instead of having agents call low-level API endpoints directly, MCP provides high-level, task-oriented tools.

   • Abstraction: A single MCP tool might orchestrate calls to multiple underlying APIs. For example, a find_recipe_ingredients tool could interact with the catalog, pricing, and inventory APIs to return a complete list of shoppable products for a given recipe.
   • Reusability: Tools are highly reusable. New tools can be introduced in the MCP layer and become immediately available to any agentic application without requiring code changes in the application itself.
   • Rich UI Components: MCP servers can return rich, reusable UI components (using protocols like AGUI or MCUI) alongside data. This allows an agent to render a product carousel or a store list directly in the chat interface, abstracting UI development away from the agent logic.

MLOps in Practice: Observability, Reliability, and Cost

Running agentic systems in production requires a focus on practical MLOps principles.

• Multi-Faceted Observability: A single tool doesn’t cover all needs. The stack uses a combination of:

  • Grafana and Prometheus for traditional API and service metrics.
  • Custom dashboards for monitoring the health of MCP servers (e.g., successful tool calls).
  • LangFuse for deep insights into the LLM stack, including latency, prompt/response chains, and cost.

• Managing Non-Determinism: To ensure reliable and predictable behavior from non-deterministic agents, we use a curated evaluation set of prompts that mimic critical user workflows. By replaying these prompts against the system, we can validate that changes to tool descriptions, system prompts, or model versions do not cause regressions.

• Cost Optimization: Cost is a critical metric managed through a hybrid approach:

  • Budgeting and Alerts: LiteLLM is used to set budgets and alerts on a per-team, per-project basis for different model providers.
  • Agent Routing: A lightweight “orchestrator” agent, often powered by a smaller model (e.g., Gemini Flash), routes incoming requests to more specialized agents. Simple tasks like follow-up questions or summaries are handled by cheaper models, while complex reasoning tasks are escalated to more powerful models like those from OpenAI.

• Tags:
• Large Language Models
• Feature
• MLOps
• Observability
• LangGraph
• Orchestration
• Tool Calling
• E-Commerce
• Agentic Architecture
• GCP