Why You Should Care About Observability in LLM Workflows

The Challenge: Compliance and Complexity in the Food Industry

At AlwaysCool.ai, which develops and reformulates consumer products for major retail brands, the initial product development lifecycle was mired in manual processes and “a lot of spreadsheets.” The core challenge lies in the highly regulated nature of the food and drug industry. Simple changes in a product’s formulation would trigger a cascade of updates to nutritional facts and ingredient lists.

This process, taking 2-3 hours per iteration, was not only time-consuming but also fraught with risk. Ensuring accuracy is paramount for consumer safety, addressing allergies, and meeting religious preferences, all under strict FDA guidelines where errors can have severe consequences.

Phase 1: From Manual Work to GPTs

The journey into AI began approximately two and a half years ago, mirroring the path of many organizations.

• Initial Steps: The team started by using the ChatGPT console (from version 3.5 to 4.0) to copy-paste and validate nutrition data.
• Custom Tooling: This evolved into building custom GPTs and a Python adapter that interfaced with FDA and USDA databases to generate accurate nutrition labels automatically.
• Internal Platform: To streamline access, they used Open Web UI to create a private, internal console for the team.

However, this console-based approach had significant limitations. It was difficult to scale, enforce consistent workflows, and delegate tasks effectively, as the open-ended nature of a chat console lacked guardrails.

Phase 2: The Synchronicity Trap with Web Applications

The next logical step was to formalize these tools into dedicated applications. The team developed scripts for nutrition generation and label validation against FDA regulations (CFRs) and wrapped them in a Next.js web interface with an OpenAI backend.

This worked well until a major shift in the underlying AI models occurred. As models became more powerful, their response times changed from synchronous to asynchronous, with processing times jumping to 5-15 minutes. This completely broke the user experience of the web applications and stalled their transition from lab experiments to production tools.

Phase 3: A Breakthrough with LangGraph and Asynchronous State Management

The solution to the asynchronous challenge was LangGraph. Initially implemented using the TypeScript SDK in the web tier, LangGraph was a “game-changer” for managing the state of these long-running, multi-step AI jobs. It allowed the team to build robust, asynchronous workflows without losing track of the process.

During this phase, they also integrated LangSmith, which provided crucial, early visibility and tracing capabilities into these complex agentic flows.

Phase 4: Maturing into a Centralized, Observable Architecture

The use of LangGraph “exploded like tribles” within the organization, leading to the current architectural evolution:

• Migration to Python & FastAPI: The team is actively migrating their agentic logic from the TypeScript web layer to a centralized service layer. This new architecture features Python-based LangGraph graphs served as microservices via FastAPI. This provides better governance, control, and a clean separation of concerns between the front-end application and the AI agents.
• The Power of LangGraph: Its key benefits include:
  • State Management: Seamlessly sharing state across all nodes and agents in a flow.
  • Checkpointing: The ability to know exactly where a process failed and to resume from that point.
  • Time-Travel Debugging: The capacity to go back in time through logs to understand an agent’s execution path.

The Cornerstone: OpenTelemetry for Compliance and Governance

The most critical feature of this new architecture is the deep integration with OpenTelemetry. LangGraph allows for the complete instrumentation of agentic flows, which is essential for operating in a regulated environment.

“It’s actually really important for compliance… we can have that business process automated in a simple set of scripts… and we can audit them and have that governance come full circle using open telemetry.”

• Unified Observability Stack: Telemetry data is sent via OpenTelemetry collectors to a full CNCF-compliant stack, including Prometheus, Grafana, Loki, and Tempo. This provides a single, unified view across both the web applications and the backend agentic services.
• Auditability: This stack creates an immutable, auditable log of every automated decision, answering the critical questions for compliance: who, what, when, and how. This is vital for workflows like HACCP, CAPA, and FDA traceability.
• Agent-Specific KPIs: The team can capture specific KPIs that the agent “sees” during its execution and insert them directly into the log stream, providing invaluable business and operational context.

This journey from spreadsheets to a fully observable, agentic microservices architecture has transformed their operations, reducing 2-3 hour manual jobs to accurate, auditable 5-10 minute automated workflows. The key learning is that for complex, mission-critical AI systems, robust state management (LangGraph) and comprehensive observability (OpenTelemetry) are not just important—they are fundamental requirements for success.

• Tags:
• Feature
• MLOps
• Observability
• LangGraph
• Langsmith
• Agentic Infrastructure
• OpenTelemetry
• FastAPI
• Compliance