The Hidden Bottlenecks Slowing Down AI Agents

The primary bottleneck in developing and scaling AI agents is not the tooling itself, but the ability to generate high-quality evaluation sets and establish a robust product feedback loop. While Prosus maintains a strong “buy-first” philosophy to focus engineering resources on what makes the product better, the immaturity and practical limitations of many vendor tools often necessitate building core components in-house.

The Evaluation Bottleneck: It’s About Data, Not Tools

Paul van der Boor emphasizes that the most difficult part of model evaluation isn’t the software, but the process of creating meaningful evaluation datasets. He states, “The hard part is you need an eval set that you curate. You need to have real users that give you new conversations that you can measure against.”

To address this, the team conducts “labeling parties,” inviting diverse employees (not just engineers) to review and score model outputs for various tasks, from image generation to customer service conversations in multiple languages. This human-centric approach is essential for building the flywheel of continuous improvement. While they have tested numerous evaluation tools, the core challenge remains data generation and curation, something a tool alone cannot solve.

Build vs. Buy: A Component-by-Component Breakdown

Bruce Martens, an AI Engineer at Prosus, details the practical decisions behind their “build vs. buy” choices for their Go-based agent platform, Tokon.

Prompting and Evaluation

The team builds its own evaluation flow for two main reasons:

1. Core Intellectual Property: Prompts are so essential to the system’s performance that they should be managed internally.
2. Data Privacy and Security: Sending entire conversations, which may contain Personally Identifiable Information (PII), to external third-party vendors presents a significant legal and security hurdle. Bruce notes, “It’s hard to convince our legal department that we’re going to do that.” While self-hosted solutions can mitigate this, the core sensitivity remains.

Orchestration Frameworks

Prosus built its own orchestrator in Go rather than using popular Python-based frameworks like LangChain or LlamaIndex. The key drivers for this decision are control and reliability. Bruce provides a specific example of debugging a timeout issue with the Vertex AI API:

Because we’re doing the request ourselves… we easily added metrics there to see how long does it take to get the first chunk from Vertex AI… And then we added an extra metric which did the time between chunks. And then we notice all of a sudden that sometimes the time between chunks is super high.

This level of granular control allowed them to implement a custom timeout based on inter-chunk latency and provide Google with a specific request ID for debugging—a level of transparency often abstracted away by frameworks. The choice of Go, while meaning they miss out on some Python-native SDKs, provides compile-time safety and reliability, which is crucial when dealing with the inherent uncertainty of LLM outputs.

Observability

Instead of adopting an agent-specific observability tool (e.g., Langfuse), the team uses a general-purpose tool, Datadog. This decision was made because:

• It consolidates software engineering and LLM observability into a single platform.
• It avoids the significant overhead (legal, security, procurement) of onboarding a new vendor.
• Their existing event-streaming backend already provides capabilities like re-running prompts, which are key features of specialized LLM observability tools.

The team concluded that they could replicate “90-95%” of the necessary functionality in Datadog without the friction of adopting a new, niche tool.

Vendor Tools: The Gap Between Promise and Reality

A recurring theme is the gap between a vendor’s advertised capabilities and their real-world performance, especially concerning documentation and security. Bruce recounts testing a vendor that promised a large suite of OAuth-enabled tools. In practice, many tools were non-functional due to minor issues like missing scopes or undocumented setup steps.

More critically, he highlights major PII concerns with several startups. In one case, he found he had “god mode” access to execute tools using other users’ credentials. In another instance with a long-term memory service, the dashboard displayed entire user conversations and extracted memories front and center. This is in stark contrast to established tools like Datadog or Sentry, which have robust PII scrubbing and access controls built-in. This extra work to ensure compliance often negates the time saved by using the vendor tool in the first place.

The Exception: The Rise of Coding Agents

The one area where the team enthusiastically adopts external tools is coding agents. They see these tools as fundamentally changing the software development experience for a specific persona: the engineer. Their adoption has evolved through several stages:

1. First Generation: Started with GitHub Copilot.
2. Second Generation: Moved the entire team to Cursor, which they felt overtook Copilot with a more integrated, conversational interface and custom-trained models.
3. Third Generation: Now experimenting with autonomous agents like Devon, which can manage entire end-to-end tasks, not just edit code.

To accelerate learning, the team has implemented “Devon duty,” where a developer is assigned to use Devon first for any task they pick up from the Jira board. The goal is to understand where the agent excels and to eventually track its contributions, such as the percentage of PRs it successfully creates.

These tools are also proving invaluable for onboarding new engineers, allowing them to quickly “interrogate a codebase” to understand its architecture, key services, and standards. This new reality changes the build vs. buy calculus itself, as tools like Cursor empower engineers to build custom solutions much faster, making the “build” option more viable than ever before.

• Tags:
• AI Agents
• LLM
• Evaluation
• Feature
• MLOps
• Observability
• Build vs Buy
• Go
• Devon
• Cursor