AI on Your Lakehouse: Context Comes in Shapes, Not Queries — Zach Blumenfeld, Neo4j
AI agents often struggle with context in lakehouses, leading to confident but incorrect answers. This workshop by Zach Blumenfeld introduces three Neo4j graph shapes built on lakehouse data—Connections (semantic layer), Trees (document outlines), and Communities (themes)—to provide essential context, enabling agents to accurately navigate structured and unstructured data, answer complex estate-level questions, and overcome limitations of traditional Text2SQL and vector search.
Zach Blumenfeld highlights that while modern AI agents, using tools like Text2SQL and vector search, can access data from lakehouses, they often lack the necessary context to provide accurate answers, leading to confident but incorrect responses. He posits that the missing piece is not a better model or query, but rather structured context provided through graph shapes built atop lakehouse data using Neo4j.
The workshop introduces three reusable graph shapes:
1. Connections (Semantic Layer for Structured Data)
For structured data in a data warehouse (e.g., BigQuery), a graph acts as a semantic layer, providing a metadata-rich understanding of table relationships. This layer helps the agent understand how tables should be joined correctly, especially in environments with hundreds of tables or similar-looking schemas. Neoarta, a Neo4j Labs project, is used to build this metadata graph. Instead of ETLing data into the graph, Neoarta ingests metadata about tables, columns, and join paths. The agent can query this graph to generate accurate Text2SQL queries, leveraging primary-foreign key relationships or query logs to inform joins. This approach allows an agent to correctly answer questions like, “Which vehicle received part IC 2042?” by guiding its understanding of the underlying relational schema. The speaker emphasizes that Neo4j’s strategic focus is on these semantic layers and virtual graphs, allowing agents to reason about data while it resides in its native system, addressing challenges of data volume, security, and ETL complexity.
2. Trees (Table of Contents for Unstructured Data)
For unstructured documents (e.g., manuals, bulletins, recalls from cloud storage), a graph creates a hierarchical “table of contents” or outline. This structure enables agents to navigate documents much like a human would, understanding containment (library > folders > documents > sections) and explicit links between documents or sections. This graph is constructed using a deterministic, idempotent loading process based on regex and inherent document structure, rather than LLM-based entity extraction. Each node in the graph (representing a folder, document, or section) receives a hierarchical URI (e.g., /technical_library/bulletins/doc_name), allowing for efficient traversal and scoping. The agent uses a Cypher skill and a predefined specification to generate queries that build and navigate this outline. This tree structure is augmented with a Lucene full-text search index, allowing for both precise lexical search and semantic expansion (where the LLM broadens search terms based on world knowledge), scoped by the hierarchical URIs. This facilitates human-like navigation and targeted search, offering a complementary approach to traditional vector search, especially for questions requiring document context or understanding non-existence.
3. Communities (Themes for Unstructured Data)
This shape surfaces hidden patterns and groupings, or “themes,” within unstructured documents that interlink (e.g., one manual referring to another). Using Neo4j’s Graph Data Science (GDS) library, the Leiden community detection algorithm is applied. This algorithm identifies natural clusters of highly interconnected documents. The process involves projecting a part of the graph into memory for efficient computation. The themes are derived directly from the document structure and link metadata, without relying on LLM-based labeling, ensuring stability and cost-effectiveness. The output includes information on how tightly interconnected documents are (conduciveness metric), top shared link targets, and most linked documents within each theme. These themes are invaluable for estate-level questions, such as identifying concentrations of issues, documentation gaps, or areas where new documentation might be needed. The granularity of theme detection can be adjusted using parameters like gamma.
Agentic Application and Benefits
The workshop culminates in demonstrating how AI agents leverage these three graph shapes to answer complex, real-world questions:
- Technician’s Query: Answering “For this VIN with this DTC code, what fix have we done on similar vehicles?” The agent combines full-text search for grounding, the Tree shape for cross-links and causes within documents, and the Connection shape to query the BigQuery warehouse for historical repair data. This multi-tool approach helps avoid hallucinations and improves accuracy compared to purely semantic search.
- Supervisor’s Query: Addressing “Are there mismatches between our documented procedures and field problems? What documents are missing or not leveraged?” This complex question, akin to proving a negative, is challenging for traditional search. The agent correlates document insights from the Tree and Community shapes with work order history from the Connection shape to identify undocumented field issues or documented procedures that are underutilized, leading to high comeback rates.
- Analytics Query: Responding to “What are common patterns across all our bulletins and recalls, and how many cars do they affect?” The agent utilizes the Theme shape to identify high-level themes and then employs the Connection shape to correlate these themes with work order history, providing insights into coverage and impact by grouping relevant work orders under specific themes.
The Neo4j CLI, along with Cypher and GDS skills, empowers the agent to perform flexible graph reasoning, access schemas, and efficiently write custom Cypher queries, dynamically adapting to complex information needs. While the workshop illustrates a lightweight, deterministic approach to graph construction, it acknowledges that more complex scenarios might benefit from LLM-driven entity extraction for even richer semantic understanding.