On AI and Knowledge — Pablo Castro, Distinguished Engineer & CVP for AI Knowledge, Microsoft

Pablo Castro explores the three pillars of AI knowledge—intrinsic, extrinsic, and learned—explaining how models leverage parametric memory, agents integrate with enterprise data through sophisticated RAG systems like Foundry IQ, and how continuous learning loops, facilitated by tools like the Agent Optimizer, enable self-improving agents, driving the next wave of AI applications.

July 17, 2026 5 min read

On AI and Knowledge — Pablo Castro, Distinguished Engineer & CVP for AI Knowledge, Microsoft — source: https://www.youtube.com/watch?v=RGSFUqzqErE

Pablo Castro, a Distinguished Engineer and CVP at Microsoft, delves into the interplay between AI and knowledge systems, emphasizing their critical role in developing advanced applications and agents. His presentation at the AI Engineer World’s Fair categorizes knowledge into three fundamental types: intrinsic, extrinsic, and learned, each crucial for building sophisticated AI.

Intrinsic Knowledge: The Foundation of AI’s Exponential Growth

Castro defines intrinsic knowledge as the “knowledge that comes with the models… what we train the models on… stored in the models’ kind of parametric memory.” He asserts that this form of knowledge propelled AI into its current “exponential” phase, especially evident in coding tools. He contrasts his manual coding process from 25 years ago with today’s AI-assisted methods, highlighting a rapid evolution:

IntelliSense (1996): First step in AI-assisted coding, reducing the need to remember function signatures.
ML-assisted IntelliSense (2018): Machine learning began ranking options for quicker selection.
GitHub Copilot (2021): A key inflection point, heavily “grounded on this intrinsic memory—what the models already knew.”
Post-ChatGPT (2022 onwards): Tools like Cursor, GitHub Copilot X, and rapidly improving models (GPT, Opus) revolutionized coding.
Open Claw (Early 2023): Demonstrated the apex of intrinsic knowledge, with successful software generated “with not a single line of code written by hand.”

Microsoft’s Foundry platform, integrated with GitHub, aims to make these models and their intrinsic knowledge accessible. It provides an agent platform for contextualization, hosting, observability, and management, alongside a vast model catalog. Castro highlights the recent general availability of Claude in Microsoft Foundry, offering “best of both worlds” for developers.

Extrinsic Knowledge: Grounding Agents in Real-World Data

While intrinsic knowledge initiated the AI revolution, Castro argues that “Intrinsic model got us here, but it only gets you so far if you’re building a system that or an agent that needs to participate in what’s happening in an organization.” This necessity introduces extrinsic knowledge, which refers to private, ambient organizational data. The industry responded with the Retrieval Augmented Generation (RAG) pattern, evolving from simple techniques to sophisticated context engineering.

He elaborates on two primary dimensions of this evolution:

Company-Wide Grounding: Moving beyond isolated datasets to leverage an organization’s entire data fabric.
- Microsoft IQ: A unified entry point for ambient data, comprising:
  - Work IQ: Connects agents to documents (SharePoint), emails, calendars, chats, and people connections.
  - Fabric IQ: Provides access to analytics assets like data warehouses, data lakes, and Power BI reports.
  - Foundry IQ: For agents to push and ground on their specific data.
  - Web IQ: For grounding on public web information to complete an agent’s view.
Evolution of Retrieval Systems: From basic vector search to advanced, combined methods.
- Initial belief that “computing cosine similarity between vectors, we were all set for retrieval” proved insufficient.
- Evaluations, particularly from Azure AI Search, consistently show that “if you combine methods, you just get better results.”
- Foundry IQ’s Design: A layered system that simplifies usage for common scenarios but offers deep control for experts. Users can start with high-level commands (“Just deal with them” for PDFs) or dive into fine-grained control over vector indexes, quantization, and lexical retrieval.
- Agentic Retrieval: For complex cases, this system allows the agent to “reflect on what’s in the data set and decide whether or not we’ve satisfied the information need” before returning results. This approach significantly improves “actual evidence recall or answer completeness” in difficult scenarios.

A demo of Foundry IQ showcased the creation of a knowledge base, integrating unstructured data (PDFs in blob storage), structured data (parquet tables), and web data. The system allows configuring agentic retrieval effort (balancing latency and quality) and connecting this knowledge base to Foundry agents or external harnesses via an MCP server. It also provides granular control over index structures, quantization approaches, and data exploration, ensuring “token efficiency” by delivering “the most information dense answer that has the fewest tokens.”

Learned Knowledge: The Agent Learning Loop

The final category, learned knowledge, stems from observing and reflecting on daily work processes. Castro highlights a profound shift: “The idea that we can actually observe the processes and get better at them by reflecting and improving every step of it is something that is really changed now.” He references Satya Nadella’s insights on how “people and agents can really compound in how they do the work and how they can create this learning loop” to capture unique organizational capabilities.

Microsoft Foundry offers a “materialized version” of this concept through its Agent Optimizer. This component facilitates a continuous improvement cycle:

Evaluate Baseline: Assess current agent performance.
Generate Candidates: Create new agent configurations.
Evaluate Candidates: Test new configurations against defined rubrics.
Deploy: If successful, deploy the optimized configuration to production.

A demo of the Agent Optimizer in VS Code illustrated how an agent’s configuration (instructions, tool definitions) can be optimized. By using eval generate, the system can produce task adherence-focused evaluations based on agent traces. Then, optimize performs a “hill climbing” process, iterating on candidates using a J-power-style loop to improve metrics. The resulting optimized configurations feature “a bunch of instructions that are not handwritten but that that they emerged out of the hill climbing process,” reflecting on actual agent traces. Castro concludes that this represents “a real learning loop materialized in practice,” empowering companies to “capture this differentiated capability.”