Kafka for Architects • Ekaterina Gorshkova & Viktor Gamov • GOTO 2026

Apache Kafka has evolved from a messaging broker into a foundational technology for enterprise architecture. The real complexity of adopting Kafka, however, lies not in the technology itself, which “just works,” but in the organizational and architectural shifts required to use it effectively.

The Shift from Monolithic Integration to Kafka Streams

The journey began around 2015 in a bank’s integration team, which traditionally relied on tools like Oracle Service Bus and Tuxedo. These systems involved graphical editors, XML-based rules, and slow release cycles tied to a large monolith.

The first project using Kafka was a simple data filtering task implemented with Kafka Streams. This represented a “breakthrough” for several reasons:

• It was a small, standalone Java application.
• It could be developed and released quickly, decoupled from the monolithic release schedule.
• It shifted the integration logic from a centralized, graphical tool to a more flexible, code-centric approach.

This initial success demonstrated the power of stream processing for replacing cumbersome, traditional integration patterns.

The Real Challenges: People and Processes

While Kafka’s replication and messaging capabilities are reliable, the most significant hurdles are human and organizational. The publish-subscribe pattern is simple in theory but complex in an enterprise setting.

“Publish-subscribe pattern is great. It tells you that publisher publishes something. It doesn’t care about subscribers. But it doesn’t work in this way in the organization. You need to have a kind of social contract. You need to have a kind of technical contract.”

Key challenges include:

• Data Ownership and Contracts: Establishing clear ownership of data and defining technical and social contracts between teams is crucial.
• Schema Evolution: Communicating and agreeing upon schema changes is a major difficulty. Explaining strict compatibility rules to business teams who may want to arbitrarily change an attribute name is a common problem.
• Organizational Architecture: The project’s success depends as much on the organizational structure as it does on the technical architecture.

These topics are so critical that the book “Kafka for Architects” dedicates chapters to data contracts, event-driven design patterns, and strategies for organizing a Kafka project within a company.

Kafka Anti-Patterns: When Not to Use It

Kafka is not a universal solution. While it can be adapted for many messaging patterns, some require a “set of workarounds” and are not native to its design. You should reconsider using Kafka for:

• Synchronous Request-Response: Implementing this pattern requires significant effort to ensure the response is routed back to the correct requester, especially with multiple service instances.
• True Batch Processing: Kafka is designed for continuous data consumption. While you can implement scheduled consumption (e.g., “consume from 10 AM to 4 PM”), it’s not a native pattern.
• Reporting and Complex Dead-Lettering: These use cases often require features that are not smoothly implemented in Kafka.

Kafka’s Role in the AI and ML Landscape

Kafka is poised to be a critical component in the modern AI stack, acting as the nervous system for intelligent applications. The conversation is no longer just about Kafka, but about the ecosystem of products and solutions being built on top of it.

Online Machine Learning

A real-world example of Kafka’s power is a “next best offer” system built for a client. This system provides real-time recommendations based on historical data and live events.

• It is implemented entirely with Kafka and Kafka Streams.
• It uses no external databases.
• The main challenge with such a sophisticated system is knowledge transfer; in this case, “only three engineers know how it works.”

Orchestration for Agentic Systems

Kafka is a natural fit for the emerging paradigm of AI-driven agentic systems. These systems, which integrate with LLMs and external APIs, can be architected like event-driven microservices.

• Orchestration: Kafka can act as the central bus for orchestrating workflows and communication between different intelligent agents.
• Real-Time Context: It provides the mechanism for feeding agents with real-time context from events happening across the business, enabling more relevant and timely actions.

The Future: Beyond Kafka Streams to Flink

While Kafka Streams is a powerful library for building stream processing applications in Java, the industry is moving towards more declarative approaches.

• The Need for SQL: Many users want to perform simple transformations using a SQL-like syntax.
• The Gap Left by ksqlDB: With ksqlDB now being decommissioned, there is a gap in the ecosystem for a user-friendly, declarative stream processing solution.
• Flink as the Successor: Apache Flink is emerging as a “very viable alternative” to fill this space, offering powerful capabilities in a more accessible format.

• Tags:
• AI
• MLOps
• Software Architecture
• System Design
• Event-Driven Architecture
• Kafka
• Kafka Streams