Open vs Closed Source Agent Infra?

This panel delves into the strategic implementation of open-source, community-driven agentic stacks, balancing the benefits of open-source frameworks and models for rapid iteration, cost-efficiency, and compliance against the complexities of abstraction and debugging in production environments. Experts from NVIDIA, Stacklok, and Prosus discuss when to adopt open source, challenges in framework choice, the critical need for robust observability, and the role of no-code solutions for diverse user groups, emphasizing the importance of aligning tooling decisions with specific use cases and organizational needs.

February 28, 2026 5 min read

Open vs Closed Source Agent Infra? — source: https://www.youtube.com/watch?v=TTAwep2uwto

This panel, hosted by Ben Epstein, delves into the strategic considerations, challenges, and benefits of leveraging open-source, community-driven agentic stacks for machine learning professionals. The discussion brings together experts Adel El Hallak from NVIDIA, Laurel Orr from Stacklok, and Olga Pavlov from Prosus (MLOps Community), offering insights from diverse perspectives as both builders and consumers of MLOps tooling.

When to Embrace Open Source Tooling

The panelists extensively discussed the decision-making process for adopting open-source tooling versus proprietary or closed-source solutions.

Initial Exploration vs. Production: Laurel Orr suggests starting with open-source agent platforms for initial experimentation, as they are “pretty easy to get started” and help identify what works for a specific use case quickly. However, she recommends beginning with a closed-source model for ease, unless one is comfortable with self-hosting. Adel El Hallak concurs, emphasizing that demonstrating the use case first is paramount. Early tendencies often involve using frontier models from providers like Anthropic or OpenAI due to their ease of access.
Scaling and Compliance: For production deployment, open-source models become critical. Adel notes watershed moments like Meta’s Llama 3 and DeepSeek’s open reasoning model, which significantly closed the gap with frontier models. He highlights that once a use case is proven, scaling to production often necessitates open-source solutions due to compliance requirements (e.g., not sending proprietary data to external APIs) and optimization needs.
Community Support and Fine-tuning: Olga Pavlov, from a product management perspective, points out that while closed systems offer a “curated experience” for initial proof-of-concept, the more a case evolves, the more one might need to fine-tune and customize, which is where open source, with its strong community support, becomes invaluable.

Navigating Open-Source Framework Challenges

A significant part of the discussion revolved around the complexities and potential pitfalls of highly abstract open-source frameworks.

Abstraction vs. Control: Ben Epstein raised a critical point: while open-source tools can offer quick “get up and running” capabilities, they often lead to “hitting a wall” due to excessive abstraction, making debugging and understanding underlying processes difficult. Laurel Orr shared similar experiences, stating that “it’s really hard to debug and build and maintain something when you have eight agents talking to each other and if you mess up the message history with one thing everything gets really confused.”
Simplicity vs. Complexity: Laurel’s team initially built their own agent framework but eventually moved to “the simplest framework we could find” that provided essential “nuts and bolts” without unnecessary boilerplate. She advises sticking with “the lowest level abstraction that you can” to avoid future headaches.
NVIDIA’s Approach to Interoperability: Adel El Hallak described NVIDIA’s philosophy of not stifling innovation by dictating specific frameworks. Recognizing that different teams use different tools (LangGraph, Crew AI, AutoGen, raw Python), NVIDIA developed the Nemo Agent Toolkit. This open-source framework enables:
- Interoperability: Through “decoration of all these agents built across all these different frameworks.”
- Observability: By collecting traces from diverse agent systems.
- Full-stack Acceleration: Leveraging NVIDIA’s unique capability to profile and optimize agent performance across the system. This allows for smarter decisions regarding computing paradigms.

Observability and Evaluation in Production

The panelists agreed that the observability and evaluation stack for AI agents is “even less solved” than the agent stack itself.

Open Standards for Observability: Adel emphasized NVIDIA’s commitment to spitting out OpenTelemetry (OTel) traces. This approach avoids vendor lock-in, allowing internal teams to use their preferred observability platforms like Datadog, Weights & Biases, or Langsmith, without being forced off.
Practical Logging for Small Teams: Laurel shared that for her four-person company, they use BAML and store logs in a lakehouse, querying them with SQL. This simple setup works effectively for smaller scales.
Privacy Concerns: Adel highlighted a critical challenge in enterprise environments: “collecting traces is difficult while maintaining privacy.” In sensitive areas like finance or chip design, collecting prompts or traces is often not permissible, necessitating solutions that respect differential privacy.

Standardization vs. Flexibility in Large Enterprises

The discussion also touched upon managing diverse tooling choices within large organizations.

Sandbox vs. Operationalization: Adel explained that NVIDIA differentiates between “a sandbox that allows for exploration” of different frameworks and the standardized components for “operationalizing those agents.” While core infrastructure like Kubernetes (Red Hat) and artifactories (J-Frog Artifactory) are standardized, teams are given flexibility for agent frameworks in the exploration phase.
Balancing Choice with Trade-offs: Olga noted that internal users at Prosus are allowed to choose different solutions “as long as the internal users are very clear on the trade-offs” concerning compatibility, cost, and scale.

The Role of No-Code/Low-Code Agent Builders

The panel explored the utility and limitations of no-code or low-code solutions for non-technical users.

External Customers vs. Internal Productivity: Laurel expressed skepticism about fully no-code solutions for external customers, as users often prefer problems to be solved for them rather than being given tools to build solutions. Debugging AI is a “whole new skill set” that many users don’t have time for. She finds workflow builders and templates more relatable. For external applications, “you oftentimes really need to hold customers hands the whole time and they’re not going to get that in the open-source community.”
Empowering Internal Users: Olga, however, sees “fantastic feeling” in enabling internal users with no-code tools to “improve their own workflow and your own task day,” fostering a high-performance culture, despite the need for guidance and governance.
Tolerance for Failure: Ben summarized this by emphasizing “what’s your tolerance for failure?” Internal tools can tolerate lower precision (60-80%), making no-code tools viable for personal productivity. However, external-facing agents demand “90 plus, 95 plus, 99 plus percent precision,” necessitating more rigorous, often simpler, and highly controlled engineering approaches like those used by his team with BAML and spreadsheet-based evaluations.

The panel concluded by underscoring that the choice of tooling—whether open or closed, simple or complex—ultimately depends on the specific use case, organizational scale, compliance requirements, and the desired balance between flexibility, control, and user empowerment.