How MasterClass Builds Multi-Agent AI That Learners Can Trust
MasterClass engineers Mandar and Priya explain how they built NCX, a multi-agent AI tutoring system, and the CoreWeave-based tracing, MCP server, and AgentOps loop that keeps it reliable and self-improving in production.
The talk starts from a simple premise: students learn better with personalized attention, but personal tutoring has never scaled. Teachers are scarce, students are many, and teaching quality varies widely. The speakers point to decades of data on one-to-one tutoring, then to two recent studies. A World Bank experiment in South Africa used Copilot to augment English classroom teaching, and a Harvard physics experiment produced similar results. In both, students who had AI conversations after class covered the same material in less time, stayed more engaged, and retained more. MasterClass’s own research found the same pattern. The lesson they draw is that a chatbot alone is not enough. The teaching system has to be pedagogically sound, managing cognitive load, using periodic recall, and pushing active learning instead of handing over answers. That takes a clear product point of view on what good education looks like.
From AI-augmented to AI-native learning
MasterClass’s AI work has moved through three stages. The first was AI-augmented learning, starting about two and a half years ago after GPT-3 appeared. The Class TA sat beside a video and answered questions about its content, and Certificates followed a similar pattern. The second stage was AI-native products. On Call lets learners phone AI personas of instructors such as Mark Cuban, and MasterClass Coach listens in on real meetings and gives executive-coaching feedback, with suggested articles and videos. The third stage is MasterClass Executive, built with Chicago Booth and OpenAI. It has 10 curated business subjects and 12 hands-on AI labs with instructor calls. Its core is an AI teacher that draws on hundreds of videos and about 20 tools and adapts to each learner’s background. Mandar gives an example. He found cost amortization dry, but the teacher noticed his disengagement, reframed the lesson around content cost amortization because that was his real job, and he became far more engaged. Early feedback from the founding cohort is strong: learners lean in, learn faster, and retain more.
Why production reliability matters more here
Mandar notes that the team writes rubrics, evals, and red-team tests before launch. The harder question is whether the architecture holds up once real learners are using it. Learning raises the reliability bar because the failure modes that matter most fall on a person trying to learn, so failures are not just bad outputs but lost learning.
Architecture of the NCX multi-agent system
Priya describes NCX as four cooperating agents. The planner designs a curriculum that will teach each concept fully, using a range of tools. The orchestrator acts as a head teacher and router. It decides whether to stay on a topic or move on, based on what the learner is doing, how the teacher is teaching, and the completion criteria for each concept, and it sends the teacher feedback such as “we’ve covered this well enough, but this aspect hasn’t been touched.” The teacher delivers the personalized experience and follows the pedagogical principles, “pushing people the right amount” rather than spoon-feeding answers. The memory agent stores the learner’s background, strengths, weaknesses, and preferences, and the teacher uses this to personalize. Two people taking the same course can have very different journeys. Sessions can run for hours and learning journeys for months.
Observability on CoreWeave with Weave
Every input, output, tool call, and piece of reasoning from every agent is stored in CoreWeave’s Weave. Priya calls the resulting data “a real goldmine,” one that was previously very hard to mine for improving individual learners’ experiences. The team uses it in three ways. First, it answers hard questions, such as why a learner had a bad experience. Second, it supports QA over the whole dataset through natural-language questions. Third, once a failure is found, it surfaces other instances of the same failure to build issue datasets for testing fixes.
The tracing skill: MCP server, skill logic, and memory book
The foundation is a tracing skill with three parts. The first is an MCP server built on CoreWeave with four generic tools: one infers the trace schema, one counts traces matching a query, one runs a shallow scan, and one runs a deep scan. The second is the skill logic, which describes how to use these tools to search traces. The third is a memory book that makes the skill self-learning. Any agent that has the skill, such as Claude Code or Cursor, can use it to investigate traces.
Priya walks through a concrete case. A learner says, “Did you actually teach me this? I don’t remember that.” The phrasing varies widely and the search is fuzzy, so the agent generates 10 to 15 hypotheses about the phrasing, uses the count tool to prune paths with very low or very high counts, runs shallow scans to read sample conversations, and then runs deep scans to examine what the agents said to each other. A single investigation can take anywhere from 20 to 100 tool calls, and it ends with a hypothesis that a human can take over and explore.
The memory book records what worked and what did not in past searches. The team went a level further by sending the tracing skill’s own tool-use traces to a separate CoreWeave project. The agent’s search behavior is now traced too, so periodic review of those paths reveals new patterns that get written back into the memory book. Priya calls this a flywheel of continuous improvement, a pattern she thinks others can reuse for building self-improving skills.
From one complaint to datasets and automated monitoring
Once a pattern is confirmed, for example “when A leads to B and the agent does C,” the team searches the last six months of traces for every instance. Usually a single learner complains, but many others have had similar suboptimal experiences without saying so. The resulting dataset includes the agents’ input forms, so fixes can be replayed and tested immediately.
Monitoring has two stages. Automated monitoring already runs on a regular cadence, with agents extracting trends from production traces and surfacing them. The team is moving toward autonomous monitoring with a larger meta skill that connects traces, GitHub, Jira, and the community pages where learners post feedback. When a learner flags a concern, the skill investigates the traces, finds similar cases, builds an issue dataset, checks whether engineering is already working on it, and flags it to the team if not. The goal is a system that clusters experiences that could be improved, creates datasets, and helps solve them with less human effort.
Key takeaways
First, trace everything, including the middle steps between user and assistant messages, because that is where the reasons for system behavior live. Second, connect those traces to an agent-ops loop so the system can diagnose and improve itself. Third, make skills self-learning by attaching a memory book they can write to. Fourth, use findings to build datasets that drive iteration on fixes. Fifth, move toward automated, and eventually autonomous, monitoring. The session closes with a promotional video contrasting Aristotle’s one-on-one teaching of Alexander the Great with MasterClass’s program for enrollment.