Building Planetary-Scale Data Systems with Venice • Felix GV & Olimpiu Pop

Félix GV, one of the architects behind LinkedIn’s Venice database, describes it as a planetary-scale, “unbundled” distributed database. This architectural pattern means each core piece of the system is a complete, standalone distributed system in its own right.

Unbundled Architecture

The main components of the Venice system include:

• Pub/Sub Layer: This acts as the write-ahead log (WAL) and replication mechanism between replicas. While internally they were exploring replacements, this is conceptually similar to a system like Apache Kafka.
• Server Fleet: A large cluster of servers, each running a Java process with an embedded RocksDB instance for local data storage and computation.
• Control Plane: A separate distributed system, using Apache ZooKeeper and Helix, responsible for managing metadata, partition placement, and overall cluster orchestration.
• Client Ecosystem: Venice offers multiple ways to access data:
  • A standard client-server model where an application sends requests over the network.
  • An “eager cache” client that embeds RocksDB directly within the application process. This client pulls data ahead of time and effectively acts as another follower replica of the database, offering significantly lower latency at the cost of higher resource consumption.

Reliability Through Chaos Engineering

To ensure the system could withstand large-scale failures, the team regularly performed “load tests” inspired by the chaos engineering pattern. This involved:

• Simulating Data Center Failures: Several times a week, during peak morning traffic, they would intentionally fail traffic out of one of their three data centers.
• Exceeding Normal Failure Capacity: The tests were designed to be more extreme than a standard outage. In a normal failure of one data center, the remaining two would each need to handle over 50% of their typical load. The load tests would concentrate traffic even further, pushing a single data center to handle well over its 50% peak capacity to ensure it could sustain the load and accommodate future growth.
• The Importance of Testing: This rigorous and frequent testing was crucial. As Félix states, “having reliability mechanisms that you do not regularly test basically means you don’t have reliability mechanisms.”

Derived Data, Asynchronous Writes, and Consistency Trade-Offs

Venice is classified as a “derived data system,” meaning it’s designed to host machine-generated data from batch or stream processing jobs, rather than user-generated primary data. This design leads to a specific set of trade-offs:

• Asynchronous Ingestion: Data is written to Venice asynchronously, either from a pub/sub stream (like Kafka) or from batch jobs that generate file dumps. This allows for extremely high ingestion throughput.
• Sacrificing Strong Consistency: The major trade-off for this high throughput is the loss of strong consistency, specifically linearizability (often described as “read-your-own-writes”). When a write is acknowledged, it confirms durability in the write-ahead log (Kafka), but the data is not yet indexed or readable in the database servers. This is a fundamental design choice, unlike a traditional database like PostgreSQL, which guarantees that once a write is acknowledged, the data is immediately consistent and readable.

The CAP Theorem in Practice

The discussion provides a practical perspective on the CAP theorem (Consistency, Availability, Partition Tolerance):

• The Real Choice is CP or AP: Since partition tolerance (handling network splits) is a necessity in any distributed system, the only real choice is between consistency (CP) and availability (AP).
• Venice as an AP System: Venice is designed as a highly available (AP) system. Because of its asynchronous write architecture, it continuously operates in a state where strong consistency is sacrificed. It’s not just a temporary loss during a network partition; it’s a permanent architectural trait.
• Multi-Region Analogy: This behavior is also seen in multi-region primary databases. If a user in Europe writes to a local data center, that data must be asynchronously replicated to a data center in the US. During this replication lag, the system is not strongly consistent across regions. In a sense, systems like Venice operate as if they are in a continuous state of high-latency replication.

Experimental Integration with DuckDB

While RocksDB serves as the primary key-value storage engine, the team experimented with integrating DuckDB, an in-process SQL analytics engine. The motivations were:

• Data Exploration and Debugging: It provides a powerful way to run ad-hoc SQL queries (e.g., aggregations, cardinality checks) on data within a single partition of Venice. This avoids the slow, high-inertia workflow of setting up an ETL job to move data into an offline system like Spark just to debug or understand the shape of the data.
• Leveraging the DuckDB Ecosystem: Integrating DuckDB provides “for free” access to its rich ecosystem of extensions, which could enhance Venice’s capabilities. For example, Venice has had basic vector math features since 2019, but DuckDB’s extensions could offer more advanced vector search and analysis functionalities.

• Tags:
• Distributed Systems
• LinkedIn
• VeniceDB
• CAP Theorem
• Chaos Engineering
• DuckDB
• RocksDB
• Kafka
• Database Architecture