From Raw Documents to AI-Ready Data — Leo Platzer & Jeff Koss

This presentation, delivered by Jeff Koss and Leo Platzer from Deasy Labs (acquired by Collibra), addresses the crucial challenge of transitioning AI projects from small-scale pilots to production-ready systems, particularly when dealing with vast amounts of unstructured data. While a chatbot might perform excellently on 40 hand-picked files, it often “falls apart” when exposed to 80,000+ real-world documents, primarily due to inherent data quality issues.

The Stalled POC to Production Dilemma

The speakers introduce a common scenario: North River Manufacturing, a large company with millions of documents in SharePoint, aims to build a legal-ops AI chatbot to answer questions about supplier contracts and partnership agreements for their legal and procurement teams. Initial pilot projects on 40 pre-curated files, involving parsing, chunking, OCR, embedding, and vector database integration, showed promising results. However, scaling this to over 80,000 files in SharePoint revealed significant hurdles:

• Difficulty discovering relevant files.
• Unawareness of sensitive data locations.
• Lack of trust in data quality.

Jeff Koss highlights that traditional data quality dimensions (completeness, conformity, accuracy) differ for documents. For unstructured data, critical quality concerns include:

• Duplicate information: Redundant content.
• Conflicting information: Contradictory facts across different versions or documents.
• Freshness: Ensuring data is up-to-date and not stale.
• Context: Leveraging metadata for more relevant chatbot responses.

These issues manifest as incorrect answers from the chatbot, sensitive information leaks, prolonged time to production, high maintenance costs, and inefficient use of AI engineers’ time.

Deasy Labs’ Approach to Unstructured Data Curation

Deasy Labs (now Collibra’s unstructured data solution) provides a platform to tackle these challenges, enabling organizations to access, understand, and curate large unstructured datasets for AI. The goal is to efficiently identify relevant, high-quality, up-to-date files, enrich them with metadata, and filter out sensitive information.

AI-Assisted Taxonomies and Metadata Tagging

The platform allows users to ingest documents from sources like SharePoint. It offers robust capabilities for building and applying taxonomies:

• Manual Tagging: Users can define tags directly.
• Reusable Tags: Leverage existing tag libraries.
• AI Auto-Suggestion: The system can suggest relevant tags based on document content, driven by user-defined prompts sent to an LLM.
• Conditional Tagging: Create hierarchical taxonomies where sub-tags are applied only to specific categories (e.g., only “Legal and Contracts” documents get “Contract Type” sub-tags).
• Metadata Generation: Once tags are defined, the system generates metadata for selected files. Crucially, it provides:
  • Evidence: Shows why AI tagged a document a certain way, highlighting specific text snippets.
  • Confidence Scores: Quantifies the AI model’s certainty for each tag, reducing guesswork.
  • Human-in-the-Loop: Users can provide feedback (thumbs up/down) for few-shot learning and adjust tags.
  • Efficient Filtering: Rapidly narrow down large corpuses based on tags, e.g., from 339 files to 20 relevant ones.

Sensitive Data Detection

The platform can scan for Personally Identifiable Information (PII), Protected Health Information (PHI), and other sensitive data. It supports:

• Pattern-based detection: Using regular expressions (regex) with context.
• AI-based detection: Identifying sensitive information even in foreign languages (e.g., Finnish national IDs) and suggesting relevant patterns. This aims to reduce false positives.

Data Quality Checks

A dedicated dashboard provides insights into data quality:

• Duplicates: Detects redundant documents, with AI suggesting which versions to keep.
• Conflicts: Identifies conflicting information, crucial for documents that evolve over time.
• Freshness: Allows filtering documents based on creation/modification dates or specific date ranges, ensuring only current information is used.
• Mandatory Metadata: Ensures critical metadata fields are present, flagging documents that lack essential information.

These quality checks enable the creation of “data slices” or AI-ready data products, which are subsets of the corpus that meet specific quality criteria.

Automated Delivery and Maintenance

To ensure continuous relevance and accuracy, the platform supports:

• Scheduled Auto-Refresh: Data slices can be configured to automatically update on a schedule, incorporating new files or removing deleted/stale ones from SharePoint.
• SDK Integration: An SDK allows chatbots and vector databases to always load the freshest metadata, preventing stale responses.

For North River Manufacturing, this process reduced data preparation time from four months to days, significantly increasing data utilization for AI, improving chatbot accuracy, and reducing compliance risks by effectively managing sensitive data.

The “So What”: Impact on RAG Performance

Leo Platzer elaborates on the critical “so what” of data quality for AI agents, especially in Retrieval Augmented Generation (RAG) systems.

• Poisoning Retrieval: Duplicated or conflicting data clutters the vector space. When an agent queries, the “top K” retrieved results will be filled with redundant or irrelevant information, hindering accurate responses.
• Stale Data Impact: If a raw corpus has 30% stale or duplicative files, up to 80% of the actual context an agent uses can be useless. This dramatically impairs the agent’s ability to provide correct information.
• Quantifiable Improvement: Through multi-hop RAG evaluations, cleaning unstructured data (removing duplicates and ensuring freshness) roughly doubled recall and improved overall task completion by 10-15%.
• Use-Case Agnostic: These improvements are fundamental and hold true regardless of the specific AI use case, making data quality a universal prerequisite for effective AI.

Context Files for Coding Agents

Leo demonstrates how the Deasy Labs SDK can prepare a context repository for coding agents (e.g., Claude Code, Codex). Instead of these agents sifting through individual files, the SDK creates a context.md file that contextualizes each folder in SharePoint. This context file includes:

• Document types within the folder.
• The folder’s purpose and key topics.
• When to use the folder.
• Example questions it can answer.

This approach significantly improves the performance of coding agents and reduces token costs by providing highly relevant, pre-summarized context.

The presentation concludes by emphasizing the transformative impact of robust unstructured data curation on AI project success, offering a path from stalled POCs to scalable, reliable AI solutions.

• Tags:
• RAG
• LLMs
• Data Curation
• Feature
• Unstructured Data
• AI Engineering
• Data Quality
• Legal AI
• Chatbots
• Metadata Management
• Taxonomy
• Sensitive Data Detection
• SharePoint