Knowledge is Eventually Consistent // Devin Stein // MLOps Podcast #335

In many organizations, especially within engineering, knowledge is fragmented across Slack, tickets, and documentation, quickly becoming stale. The fundamental challenge is that while code serves as a definitive system of record—a source of truth—the vast amount of contextual knowledge surrounding it is ephemeral.

The Fact-Based Reasoning Agent

Dosu introduces a “fact-based reasoning agent” designed to address this. The agent assists with open-source and internal engineering maintenance by answering questions and triaging issues. It operates on the premise that engineers are often the final arbiters of truth because “code is truth.”

The traditional approach for an agent is to start an investigation from scratch for every query by searching the codebase, pull requests, and conversations. Dosu’s new agent design improves this with a learning loop:

1. Research and Fact Generation: As the agent investigates a query, it identifies and learns “facts,” which are claims supported by evidence from the data sources.
2. Response Generation: It uses these newly found facts to formulate a response.
3. Validation and Knowledge Commitment: If the response is confirmed as correct—either through direct user feedback or validation by a domain expert—the underlying facts are committed to a permanent knowledge base.
4. Fact-Based Reasoning: The next time a similar question arises, the agent first consults its knowledge base. It assesses if the existing facts are sufficient to answer the query. If not, it performs additional research only on the missing pieces, making the process faster and more efficient over time. This creates a “facts flywheel” where the system becomes more knowledgeable and effective with increased usage.

The Lifecycle of a Fact: Ensuring Knowledge Quality

A critical challenge is preventing the propagation of incorrect information. To manage this, Dosu implements a three-stage philosophy for committing facts to the knowledge base, ensuring a balance between automation and accuracy.

• Stage 1: Human-in-the-Loop: An expert, like a project maintainer, explicitly saves a valuable interaction or piece of information to the knowledge base. They can preview and edit what gets saved, ensuring the highest quality of knowledge. The incentive for the expert is that this action reduces their future workload, as the agent will handle similar questions next time.
• Stage 2: Proactive AI Suggestions: The agent acts as a “sleeper agent,” monitoring conversations and pull requests. When it identifies a statement that either conflicts with its existing knowledge or seems important but is not yet documented, it proactively prepares a draft. It then asks the relevant expert for approval, lowering the barrier to entry by doing the preparatory work. The agent identifies importance by looking for signs of significant investigation, back-and-forth discussions, or common troubleshooting questions.
• Stage 3: Full Automation: The eventual goal is for the agent to become confident enough, based on a history of approved suggestions, to automatically generate and save knowledge without human intervention. Less confident suggestions would still be routed for human review.

Knowledge Maintenance in an Ever-Changing Environment

Documentation notoriously becomes stale the moment it’s written. The key to solving this is leveraging AI for tasks it excels at while relying on humans for their unique strengths. Humans are good at identifying what is important, while AI is exceptional at monitoring vast streams of information for changes.

Dosu’s approach hinges on distinguishing between two types of knowledge:

• Temporal Knowledge: Information that is a record of a point in time, like meeting notes. These shouldn’t be updated.
• Canonical Knowledge: The “source of truth” documentation that should always be up-to-date.

By treating code as a verifiable system of record, the agent can monitor changes in the codebase and compare them against the canonical knowledge. When it detects an inconsistency—for example, a documented function signature no longer matches the actual code—it can automatically flag the relevant documentation or fact for an update. This makes knowledge maintenance a manageable, semi-automated process rather than an impossible human task.

The Future of Documentation and Agent Collaboration

The rise of coding agents ironically increases the importance of high-quality written knowledge. Unlike humans, agents don’t have persistent memory and rely heavily on documentation to understand context, such as the architecture of a billing system.

Agents consume documentation differently than humans:

• They prefer information-dense, machine-readable formats. Aesthetics are irrelevant.
• They benefit immensely from documents that explicitly link high-level product concepts directly to their implementation in the codebase (e.g., file paths, directories).

This points toward a future where agents are the primary consumers of documentation. Furthermore, this opens up possibilities for an ecosystem of collaborating agents. Dosu can act as a foundational knowledge provider for other specialized agents, such as SRE agents performing root cause analysis or coding agents needing context before making changes. By providing an up-to-date, trusted knowledge source, Dosu can prevent other agents from acting on stale information, such as an outdated runbook. This vision moves away from simply searching a messy “knowledge sprawl” and towards cultivating a reliable source of truth that grows and adapts with the organization.

• Tags:
• Developer Tools
• RAG
• LLMs
• Feature
• Agents
• Knowledge Management
• Documentation
• Knowledge Base