GenAI Grows Up: Building Production-Ready Agents on the JVM • Rod Johnson • GOTO 2025

The GenAI Paradox: Personal Success vs. Enterprise Failure

While Generative AI has become a powerful personal assistant for developers and knowledge workers, its adoption in enterprise for automating business processes has been plagued by an alarmingly high failure rate. The success of tools like Claude Code, which augment human performance in a “human-in-the-loop” scenario, has not translated to the world of automated business processes. The core issue is the gap between what works for personal assistance—which can tolerate unpredictability—and what is required for production-grade business automation, which demands reliability.

Why Enterprise GenAI Projects Fail

The reasons for failure stem from a combination of inherent technological challenges, flawed integration strategies, and organizational issues.

• Inherent Technology Issues:

  • Non-Determinism: Large Language Models (LLMs) are inherently non-deterministic, a stark contrast to traditional software systems where predictability is paramount.
  • Hallucinations: The tendency for LLMs to generate false information poses a significant financial and legal risk, as demonstrated by an expensive Air Canada chatbot incident.
  • Ambiguity: Natural language, the primary interface for LLMs, is far more ambiguous than programming languages, leading to misinterpretation.
  • Cost and Scalability: GenAI applications are resource-intensive, making them slow and expensive to run at scale.
  • Testability: The non-deterministic nature of LLMs makes them incredibly difficult to test reliably.

• Prompt Engineering: The New Alchemy: The practice of “prompt engineering” is more of a black art than a scientific discipline. Subtle changes in prompts can lead to vastly different outcomes, and prompts are often highly sensitive to the specific model being used. Relying on a single, complex system prompt—a common pattern in personal assistant tools—is not a viable strategy for achieving the high degree of predictability required for enterprise workloads.

• Integration and Organizational Problems:

  • Disconnected Systems: AI initiatives are often developed in isolation from core business systems. GenAI work is frequently done in Python, while critical business logic resides in Java or .NET, creating a significant integration gap.
  • Vendor Influence & Top-Down Mandates: Corporate strategies are often driven by vendor sales pitches and top-down mandates from executives, disconnected from the practical realities of implementation. This mirrors the early, failed days of Enterprise Java, which were only corrected by a bottom-up, developer-led movement focused on practical solutions.
  • Siloed AI Teams: Centralized AI teams can be dangerous if they lack a deep understanding of the core business domain and possess a different skill set from the engineers who build and maintain the enterprise systems.

A Path Forward: Building Dependable AI Systems

To overcome these challenges, a fundamental shift in approach is required, focusing on mitigating non-determinism and integrating deeply with existing enterprise infrastructure.

1. Attack Non-Determinism: The primary goal is harm minimization. This can be achieved through three key strategies:

   • Orchestration: Break complex tasks into smaller, more manageable steps. Each step should have a simpler prompt and fewer tools, reducing the scope for error. This also allows for using smaller, cheaper models for specific tasks.
   • Deterministic Planning: Use code, not an LLM, to orchestrate the sequence of steps. Frameworks like LangGraph (using a finite state machine) or CrewAI (defining a sequence) provide deterministic control over the workflow.
   • Structured Communication: Interact with the LLM using structured data like JSON or objects instead of relying solely on natural language. LLMs are proficient at processing structured formats, which enables type-safe, predictable, and testable interactions that can be reasoned about in code. The core principle is to do as much as possible in predictable, testable, and fast code, reserving the LLM for tasks where it adds unique value.

2. Domain-Integrated Context Engineering: The current discourse on “context engineering” often misses the most critical source of memory in an enterprise: the existing systems of record and the domain model. Instead of trying to create human-like memory analogies, agents should leverage the highly structured, reliable state already present in business applications. This concept of Domain-Integrated Context Engineering posits that the domain model should be the primary source of context sent to the LLM and the primary target for the structured data returned by it.

The Role of the JVM in Enterprise AI

There’s a critical distinction between machine learning/data science and enterprise GenAI. While Python is the undisputed leader for model training and data science, enterprise GenAI is primarily about making API calls to pre-trained models. In this context, the critical adjacency is not to the model training tools, but to the business logic.

• Python vs. JVM: The Python ecosystem lacks the mature tooling, libraries, and frameworks for building large-scale, reliable enterprise applications that the JVM provides. For organizations whose business logic is already on the JVM, building GenAI applications in Python creates an unnecessary and friction-filled gap.
• JVM Strengths: The JVM offers maturity, stability, strong typing, compile-time safety, and a rich ecosystem of libraries and tools. These are not obstacles to AI; they are the foundations of dependable AI systems. This is an opportunity for the Java community to lead again by building something better, rather than imitating Python approaches.

Introducing Embabel: A Framework for Production-Ready Agents

Embabel is a new open-source project built on the JVM designed to directly address the causes of GenAI project failure.

• Core Philosophy: It emphasizes reliability, predictability, and testability, integrating closely with the Spring ecosystem.
• Deterministic Planning: Embabel introduces a novel, deterministic planning step using a non-LLM AI algorithm called Goal-Oriented Action Planning (GOAP). It dynamically works out the optimal sequence of actions to achieve a goal based on the current world state, preconditions, and postconditions. This is more flexible and less brittle than a finite state machine.
• Developer Experience: It provides a modern, type-safe API for both Java and Kotlin. Interactions are strongly typed, using domain objects (like Java Records) to communicate with the LLM, eliminating “magic strings.”
• Integration with Spring: Agents are Spring components, allowing seamless integration with Spring Security, transaction management, and other enterprise features.
• Actions as Code or LLM Calls: The framework treats a call to an LLM and a local code execution as equivalent “actions.” This encourages developers to implement logic in deterministic code whenever possible, reserving LLM calls for tasks that require natural language understanding.
• Testability: Embabel provides extensive support for unit testing actions (including verifying prompt construction) and integration testing entire agent flows.

• Tags:
• Feature
• Enterprise AI
• GenAI
• Agent Frameworks
• JVM
• Java
• Embabel