How Grounded Synthetic Data is Saving the Publishing Industry // Robert Caulk
Robert from Emergent Methods discusses how grounded synthetic news data can solve the publisher revenue crisis in the AI era. He details the process of 'Context Engineering' news into token-optimized, objective data for high-stakes AI agent tasks, covering their open-source models for entity extraction and bias mitigation, and the on-premise infrastructure that protects publisher content.
The traditional advertising model for news publishers is fundamentally broken in the age of AI. When an AI agent visits a website on a user’s behalf, it doesn’t generate ad revenue, threatening the financial viability of journalism. This issue is compounded by the use of original publisher content to train large language models (LLMs), leading to high-profile copyright lawsuits. The core challenge is to create a new, scalable model that allows publishers to monetize their content while providing AI developers with the high-quality, real-time information they need.
The Bridge: Grounded Synthetic News Data
The proposed solution is grounded synthetic news data, which serves as a bridge between publishers and AI developers. This approach protects publishers from copyright infringement and reduces hallucination rates for AI agents.
The needs of both parties are distinct but complementary:
- Publishers want to: Monetize their information, protect their original expression, and have low user acquisition effort.
- AI Developers want: Accurate and reliable data feeds, token cleanliness (i.e., no raw HTML), a diversity of sources, and a natural language API.
Crucially, for high-stakes decision-making tasks like geopolitical forecasting or financial analysis, AI developers do not actually want the original journalistic expression. They need objective, clean, and structured data. The original narrative voice can introduce bias and is not optimized for machine consumption.
Context Engineering the News
The process of creating this valuable synthetic data is called “Context Engineering.” It’s not about inventing a new way to understand news but about systematically structuring it for an LLM. This is achieved by breaking down articles into the foundational components of journalism: the “who, what, why, where, and when.”
A synthetic data point is enriched with:
- Core Information: People, organizations, events, statements, and attributions.
- Essential Metadata: Precise publication dates, geocoordinates for locations, and identified motivations.
- AI-Specific Enrichments: Sentiment analysis, topic classification, and analysis of the reporting voice (e.g., objective vs. provocative).
- Structured Relationships: Knowledge graphs that map the connections between entities across thousands of sources.
The goal is to produce what is described as “token optimized, extra content, LLM ready, objective information,” in stark contrast to raw, unstructured HTML which is a “seed for hallucination.”
Building the Synthetic Data Pipeline
Creating high-quality synthetic data requires a sophisticated MLOps pipeline and specialized models.
- Entity Extraction: A small, efficient, in-house-trained entity extraction model (based on Gliner) is used for high-throughput processing. This model was specifically trained on a diverse dataset to counteract the common problem of Western-centric bias found in many off-the-shelf extractors, ensuring it performs well on names from all regions.
- Bias Mitigation: A significant research effort is dedicated to understanding and mitigating bias. This includes analyzing models like DeepSeek to prove and understand their biases. A newly released open-source model, “bias expert,” can identify bias in content and recommend improvements.
- Infrastructure: To protect publisher content, all processing is done on-premise. No data is sent to third-party APIs like OpenAI or Anthropic, which is a key part of the agreement with publishers. The use of small, fine-tuned models is critical for managing the cost and latency of this high-throughput, 24/7 operation.
- Storage and Retrieval: Quadrant is used as the vector database to store the synthetic text and its rich metadata, enabling high-level filtering and retrieval for agentic tasks.
Validation and Business Model
The value of this approach is validated by real-world performance. On Metaculus, a platform for real-time forecasting tournaments, bots using Ask News data consistently dominate the leaderboard. Users switched from other data providers because the quality of the data led to more accurate forecasts and financial wins.
The business model is designed to be a partnership:
- Royalty Distribution: 50% of all royalties are paid directly to publishers, weighted by how often their content is surfaced.
- Sustainable Research: The other 50% allows Emergent Methods to maintain the platform and continue its open-source research into bias and entity extraction.
Over 800 publishers, including major sources like Agence France-Presse (AFP), have partnered with Ask News, alongside users from universities (Columbia, George Mason) and risk assessment firms. This demonstrates a viable ecosystem where publishers are compensated for their work, and AI developers get the reliable, high-quality data needed to build next-generation applications.