Why Physical AI Needs a new Data Set | Rerun CEO

Nikolaus West, CEO of Rerun, argues that the primary obstacle to advancing “physical AI”—encompassing robotics, AR/VR, and spatial computing—is the inadequacy of current data tooling. He compares Rerun to Weights & Biases, noting that while both provide logging and system-of-record capabilities, Rerun is specifically designed for the complex, multimodal, and time-aware data inherent in physical systems.

Rerun’s Technical Approach: An Entity Component System for Data

The core of Rerun’s innovation is its data model, which was redesigned four times to achieve the right balance of flexibility and performance. Instead of a traditional object-oriented approach, Rerun adopted an Entity Component System (ECS), a model borrowed from game development.

• How ECS Works: Rather than defining rigid objects like an Image with fixed properties, data is broken down into composable components (e.g., a memory buffer, image size, a confidence score). Systems then interpret these components. For example, a visualization system will render an image if it finds an entity with the necessary image buffer and size components. This allows users to log and visualize arbitrary, custom data without the tool needing prior knowledge of their specific data structures.
• Tech Stack: The system is built from the ground up in Rust and uses Apache Arrow. It features a custom in-memory database and a specialized file format, described as a “sparse version of Parquet,” optimized for the multi-rate, episodic nature of sensor data from robots.
• Open Source Strategy: The core logging SDK and visualizer are open source to maximize adoption and trust. West believes visualization is too critical at every stage—from notebook prototyping to live debugging—to be behind a paywall. Monetization is handled through a cloud-based data platform designed for managing and querying massive datasets.

The State of AI in Robotics

West highlights incredible recent progress, particularly in complex manipulation. Tasks like folding laundry have transformed from “impossible to boring” within the last year, driven by end-to-end learning methods.

• Imitation Learning: Described as the “robotics version of supervised learning,” this involves a human teleoperating a robot to perform a task. The system records all sensor data and actions, training a neural net to replicate the behavior. This began showing serious results a few years ago.
• Reinforcement Learning (RL): Traditionally used for locomotion, RL is now being combined with imitation learning to create more robust and adaptable manipulation capabilities.
• Challenges: A major hurdle is the lack of standardized benchmarks. Unlike the LLM space, robotics performance is often tightly coupled with specific hardware, making universal benchmarks difficult. Most companies rely on internal benchmarks on their physical robot fleets.

From Demos to Deployed Products

There’s a significant gap between viral video demos and real-world products. The challenge lies in the “whole systems aspect”—servicing, user onboarding, and handling the long tail of real-world variability.

While high-end humanoid robots are still far from mass consumer adoption, practical robotics are making inroads. Scrappy companies are deploying robots in warehouses for pick-and-place tasks using open-source models. In the consumer space, the most successful application remains the robot vacuum, with newer models like Matic showing significant advances in autonomy and mapping over older ones like Roomba.

The humanoid form factor is beneficial for data collection, as humans can easily teleoperate or “puppet” them, but West suggests a semi-humanoid form (a wheeled base with arms) is more practical in the short term.

The Data Pipeline Problem

The biggest challenge for robotics teams is data management. The ecosystem for text data (LLMs) is mature, with tools like Parquet and Spark enabling flexible, high-performance data processing. In contrast, robotics data formats (like ROS bags) were designed for fast logging, not flexible querying.

This forces teams to write complex, imperative code for every data query—for example, finding all instances where a gripper failed at night. Rerun aims to solve this by providing a unified data platform with a powerful query engine, turning these complex jobs into simple, declarative queries and drastically simplifying the data pipeline.

• Tags:
• Robotics
• Imitation Learning
• Reinforcement Learning
• Feature
• Computer Vision
• Data Visualization
• Embodied AI
• Physical AI
• Data Logging
• Rerun
• Entity Component System
• ROS