Distilling 200+ Hours of NeurIPS: What’s Next for AI // Nikolaos Vasiloglou // MLOps Podcast #336

Nikolaos Vasiloglou, VP of Research-ML at RelationalAI, undertook the monumental task of manually distilling the NeurIPS 2023 conference, which featured nearly 8,500 publications and hundreds of hours of video content. He describes the process as becoming a “human language model,” aiming to identify and synthesize key research trends that might otherwise be overlooked. This effort resulted in a comprehensive 12-part, 650-slide presentation. Vasiloglou notes that this was likely the last year such a manual synthesis is feasible, and future efforts will require the assistance of purpose-trained LLMs.

The State of LLMs: Everywhere, But Not for Everyone

A primary takeaway from the conference is that Large Language Models (LLMs) are now pervasive. The open-source movement is gaining significant traction, with many papers presenting models that are fully open, including checkpoints, datasets, and pre-training capabilities.

A key talk by the inventor of the LSTM highlighted a sentiment Vasiloglou translates as, “Everybody loves to hate language models.” The volume of complaints and critiques signifies deep engagement and widespread use, and these very complaints are constructive feedback that will drive the next wave of innovation.

The conference also marked the 10th anniversary of two foundational papers:

• The first sequence-to-sequence model, the precursor to modern LLMs, authored by Ilia Sutskever.
• The first Generative Adversarial Networks (GANs), which kicked off the era of synthetic image generation.

The Rise of Agentic AI

AI agents were a dominant theme at NeurIPS. A key point of discussion was the performance of smaller, open-source models versus the large, frontier models from providers like OpenAI and Google. Six months post-conference, Vasiloglou’s assessment is that the frontier models still hold a significant lead, despite the buzz around smaller alternatives.

Two significant trends in agentic AI stood out:

1. Agents in Scientific Discovery: There were numerous papers on building agent-based labs for biology, materials science, and antibiotics discovery. In this context, the “hallucination” or creativity of LLMs is a feature, not a bug, as it allows for out-of-the-box thinking where one novel idea out of ten incorrect ones can be a breakthrough.
2. Decision Support vs. Decision Making: Vasiloglou draws a critical distinction. Using agents for decision support—running simulations, generating scenarios, and exploring “what-if” questions—is a powerful and appropriate use case. However, there’s a rushed effort to use them for automated decision making and execution (e.g., booking flights, managing finances). He cautions that entrusting deterministic tasks to probabilistic systems is risky and difficult to debug, comparing it to the early, chaotic days of multi-threading.

Conquering the Last Frontier: Tabular Data

For years, deep learning has struggled to outperform traditional shallow models like XGBoost on tabular data. NeurIPS 2023 marked a turning point. For the first time, a series of papers demonstrated deep learning models that could beat XGBoost on predictive analytics tasks, albeit at a higher computational cost.

This signals that foundational models for tabular data are on the horizon. A major obstacle has been the lack of large, public tabular datasets equivalent to the web-crawled text corpora used for LLMs. However, concerted efforts are now underway to create these datasets, which will accelerate progress in this domain. The enterprise demand for a single, consolidated platform that can handle text, images, and tabular data is a strong driver for this research.

Verification Systems and the Role of Math

As AI models become more powerful, making them reliable is a central challenge. A key theme was the combination of generative models with verification systems. The ideal use case for an LLM is a task where its output can be formally verified.

“If the task you’re trying to solve you can attach a verifier, that’s the perfect combination.”

Examples include:

• Theorem Proving: An LLM can creatively generate a proof, and a symbolic system can then check if the proof is logically sound.
• Software Generation: An LLM generates code, but a verifier is needed to ensure the code does what is intended. This goes beyond simple unit tests to proving abstract properties, such as a variable never becoming negative.

The AI community is intensely focused on mathematics for two reasons:

1. Human Aspiration: It aligns with the grand challenge of solving difficult, long-standing problems.
2. A Proxy for Reasoning: It has been shown that training an LLM on mathematical problems and code improves its general reasoning capabilities, making it a better agent overall. It’s analogous to how advanced math is used as a measure of intelligence in university admissions, even for humanities students.

The Power of Knowledge Graphs

Vasiloglou defines a knowledge graph as “a language that both humans and machines understand.” It serves as a structured, intermediate representation of an organization’s truth, containing facts, entities, rules, and relationships.

Historically, building a knowledge graph was a massive, multi-million dollar undertaking. LLMs have dramatically lowered this barrier, making it possible to extract and structure knowledge from disparate sources like documents, databases, Slack messages, and emails.

At RelationalAI, the focus is on relational knowledge graphs, which act as a semantic layer over existing databases. This approach ensures that data security, access roles, and retention policies are respected. It also solves a common enterprise problem where business logic (e.g., the definition of an “active customer”) is inconsistently replicated in countless queries across different applications. By centralizing this logic in the knowledge graph, companies can build cleaner, more maintainable, and more consistent applications.

Future Directions: Data Attribution and Composable Models

Two forward-looking topics from the conference are poised to have a major impact:

1. Data Attribution and Monetization: For the first time, papers were presented on models that can trace a generated output (e.g., a sentence) back to the specific source document in the training data. This technology is a crucial step toward creating fair data monetization and royalty systems, allowing creators to be compensated for their contributions to training data.
2. Composable Language Models: This concept involves building smaller, specialized LLMs and then composing them like LEGO blocks to create a larger, more capable model. For example, one could combine a Greek-to-English model with an English-to-Italian model to create a Greek-to-Italian translator on the fly. This approach contrasts with today’s monolithic models and could lead to more collaborative, efficient, and modular AI development.

• Tags:
• Agentic AI
• LLM
• Knowledge Graph
• NeurIPS
• Tabular Data
• Model Verification
• Data Attribution
• RelationalAI