Unlocking Unstructured Data with LLMs

Shreya Shankar of UC Berkeley discusses DocETL, a MapReduce-style framework that leverages LLMs to extract, analyze, and structure insights from unstructured enterprise data. The conversation covers practical architecture patterns, the role of non-determinism, strategies for model selection (including fine-tuning and multi-LLM pipelines), and the importance of user experience in this emerging field.

July 3, 2025 4 min read

Unlocking Unstructured Data with LLMs — source: https://www.youtube.com/watch?v=BsQvMtCD814

Large Language Models (LLMs) are creating a paradigm shift in how organizations process massive amounts of unstructured data, such as text documents, PDFs, and logs. Traditional approaches relied on bespoke NLP models or expensive crowd-sourcing pipelines, but LLMs offer a more flexible and powerful way to extract semantic information and structure from these sources.

The DocETL Framework: MapReduce for Semantic Data

DocETL is a dissertation project from UC Berkeley that applies the familiar MapReduce paradigm to unstructured data using LLMs. The workflow allows users, including non-coders like analysts or doctors, to build powerful data processing pipelines.

Map Operation: The map operation is defined by an LLM prompt to extract one or more insights per document. For example, a prompt could instruct the LLM to “extract all customer pain points and any associated quotes.” The LLM executes this prompt on each document, producing a list of structured outputs.
Reduce Operation: The reduce operation performs a semantic group-by and aggregation. Since LLMs can be inconsistent in their phrasing, the grouping is done semantically, not by exact string matching. After grouping, another LLM can be used to summarize the collected insights for each theme, generating a report for stakeholders.

This framework is designed to be extensible, with pluggable operators that users can contribute. It handles task decomposition automatically; for instance, if a user wants to extract 50 different fields, DocETL can break that down into multiple sub-tasks to improve accuracy, as LLMs perform better with more focused requests.

Navigating Non-Determinism, Observability, and Architecture

A key challenge with LLM-based pipelines is their probabilistic nature. This non-determinism can be viewed in two ways:

A Bug for Accuracy: For tasks with a clear ground truth, like well-defined entity extraction, non-determinism is undesirable. Users often set the model’s temperature to zero to create the impression of deterministic output and focus on accuracy.
A Feature for Creativity: For creative or exploratory tasks, like “find interesting insights in this data,” non-determinism can be a feature. Users may run the pipeline multiple times to glean a wider range of novel insights.

To manage these pipelines, the DocWrangler project provides a specialized IDE. It offers greater observability into the pipeline’s execution, tools for prompt engineering (including automatic prompt writing), and the ability to run pipelines incrementally. It also facilitates feedback loops, where LLMs can be used to edit prompts based on user feedback on the outputs.

In an enterprise setting, these pipelines can establish a “semantic medallion architecture.” A data engineering team can use DocETL to process millions of PDFs, for example, and materialize the output as a structured table in a relational database or a knowledge graph. This structured data can then be queried by analysts or used in downstream RAG applications. Reliability is enhanced through guardrails, which can be code-based (e.g., checking if an extracted entity is actually in the source document to prevent hallucination) or LLM-based (using an “LLM as judge” to validate outputs).

Model Strategy and the Evolving Ecosystem

Choosing the right model is critical and nuanced. It’s common practice to use multiple LLMs within a single pipeline, leveraging their respective strengths.

Extraction vs. Generation: A common pattern is to use models like Gemini or OpenAI’s for extraction tasks and then switch to a model like Claude for summarization and report generation.
Reasoning Models: Advanced reasoning models are most valuable for bridging the gap between a user’s high-level, vaguely-defined intent and a well-specified pipeline. For example, they excel at translating a natural language request into a full DocETL pipeline. However, for the actual extraction and summarization operations within the pipeline, a standard model with a well-crafted prompt is often sufficient and more cost-effective.
Fine-Tuning: A powerful enterprise pattern is to use a DocETL pipeline to generate a high-quality labeled dataset from unstructured documents. This dataset can then be used to fine-tune a smaller, specialized, or open-source model, which can then be plugged back into the pipeline for a more efficient and customized workflow.

The entire field is in its infancy (“day zero”), and the tooling ecosystem is evolving rapidly. The key challenge lies in optimizing the complex interaction between the user’s intent, the semantic pipeline’s behavior, and the nature of the data itself.