Scaling Meta's Multi-Agent Systems to a Billion Videos

Meta's approach to solving modality misalignment and content theft in short-form video using a multi-agent system of smaller, specialized models instead of a single large LLM. The talk covers the architecture (Perceiver, Retriever, Reasoner), evaluation stack, and key cost-saving optimizations.

May 25, 2026 5 min read

Scaling Meta's Multi-Agent Systems to a Billion Videos — source: https://www.youtube.com/watch?v=psC2-iEvXCg

At the scale of hundreds of millions of short-form videos per day, user-generated content (UGC) presents unique and dynamic challenges. The content patterns are constantly changing, creators actively try to game the system, and the data is far messier than the clean web data used to train most large models. Two of the most difficult problems in this space are modality misalignment and content theft.

Modality Misalignment: This occurs when different modalities within a video contradict each other. A simple case is text saying one thing while the video shows another. A more complex and common problem is intra-modality intrusion, where a video about a benign topic suddenly inserts a few frames of unrelated ads, political content, or other policy-violating material before returning to the original topic. This is a “needle in a haystack” problem.
Original Content Attribution: In the copycat creator economy, it is trivial to download, slightly edit, and re-upload someone else’s viral content, passing it off as original to gain followers and revenue.

Why a Single Large Model Fails

Attempting to solve these problems with a single, massive Vision Language Model (VLM) is impractical for several reasons:

Cost: The cost of running inference with a 200B+ parameter model at a scale of 100 million+ videos per day makes it economically unfeasible. The ROI is impossible to justify.
Lack of External Context: A standalone LLM cannot compare a new video against the entire corpus of existing videos or access external databases and knowledge graphs about creators, past violations, and content similarity. This context is essential for attribution and theft detection.
Modality Bias: Large, general-purpose VLMs often exhibit bias towards one modality (e.g., text over video), which can cause them to miss subtle visual cues or intrusions.
Lack of Specialization: These problems don’t require a model that can also solve complex math problems. They require highly specialized, fine-tuned models that are experts in this specific domain.

A Practical Multi-Agent Architecture

A more effective and efficient solution is a decomposed, multi-agent system where each agent is a specialized, fine-tuned model that functions like a microservice. This architecture consists of three core agents.

1. The Perceiver Agent

This is the first agent in the pipeline, responsible for signal acquisition and deep content understanding. It takes the raw short-form video and breaks it down at a granular level.

It uses a suite of fine-tuned VLMs and discriminative models to process all modalities (video, audio, text).
It identifies scene boundaries and splits the video into coherent segments.
It generates last-layer embeddings from a VLM for each segment and stores them in vector databases.
It performs OCR to extract text, identifies topics, and generates natural language descriptions of the video’s content.
Essentially, it does all the necessary pre-processing to create a rich, structured representation of the video.

2. The Retriever Agent

Given the processed signals from the Perceiver, the Retriever’s goal is to gather all relevant context from the broader ecosystem to maximize recall.

It takes the embeddings, topics, and OCR data from the input video.
It performs K-Nearest Neighbor (KNN) searches across vector databases to find semantically similar videos.
It queries knowledge graphs to pull context on creator-to-creator interactions and user engagement patterns.
It constructs similarity matrices to identify other videos that share topics, styles, or visual elements.
The objective is to find any and all content that might be related to the input video to aid in the final decision.

3. The Reasoner Agent

The Reasoner is the decision-making component, equipped with a slightly larger, fine-tuned LLM (e.g., 11B parameters) capable of more complex logic.

It synthesizes all the inputs: the granular description from the Perceiver and the contextual documents from the Retriever.
It uses a chain-of-thought process to analyze the evidence and determine if a policy violation (misalignment or theft) has occurred.
It outputs a final decision, a confidence score, and a detailed explanation of its reasoning.
If the Reasoner determines it has insufficient evidence, it can request more context from the Retriever, asking it to expand its search parameters.

A Comprehensive Evaluation Framework

Evaluating such a system requires a multi-faceted approach beyond simple accuracy.

Performance: Standard precision, recall, and F1 scores are the primary product metrics for identifying misalignment and theft.
Retrieval Quality: The Retriever agent is evaluated in isolation to ensure it is surfacing all relevant documents. A failure here cascades to the entire system.
Reasoning Quality: The chain-of-thought process is evaluated using an LLM-as-a-judge, calibrated against a golden dataset reviewed by human-in-the-loop (HITL) labelers.
Robustness: The hallucination rate is closely monitored by sampling production outputs and having human reviewers check if the model’s reasoning is grounded in the video’s actual content.
System Efficiency: Every hop in the pipeline is logged to track latency, token usage, and cost for each agent and tool call, providing a 360-degree view of system performance and scalability.

Four Key Optimizations to Reduce Cost by 10x

Spatial and Temporal Frame Merging: To reduce token usage, the Perceiver agent intelligently samples frames. For a video of clouds moving slowly, it doesn’t need to send 300 nearly identical frames to the VLM. It can merge temporally similar frames and spatially similar patches within a frame, drastically reducing input tokens without losing semantic meaning.
Semantic Hashing of Viral Content: Viral content is often re-uploaded with minor changes. By pre-computing and caching hashes of viral videos, the system can perform a quick lookup. If an incoming video is an 98%+ match to a known piece of unoriginal content, the entire multi-agent pipeline can be bypassed, saving significant compute.
Dynamic Routing and Reasoning Budgets: Not all content requires the same level of scrutiny. Simple, safe content (e.g., a nature video) can be routed through a lighter pipeline, while complex or controversial topics (e.g., political news) are allocated a higher “reasoning budget,” allowing the Reasoner agent to use more tokens and perform a deeper analysis.
Metadata Pruning: The system uses signals about the creator and topic to skip processing entirely. A reputable creator with a long history of original content is trusted. Likewise, topics known to be safe (e.g., “dogs playing in a garden”) can be allow-listed, freeing up resources for higher-risk content.