Real-time Feature Generation at Lyft // Rakesh Kumar // MLOps Podcast #334

Lyft’s business-critical operations, such as surge pricing and real-time forecasting of supply and demand, are powered by machine learning models that require features generated with sub-second to second-level latency. Serving stale data can lead to inaccurate market predictions and significant financial impact, making low-latency feature generation paramount. The majority of these use cases rely on data aggregated over one-minute windows.

The Evolution of Lyft’s Real-Time Pipeline

Lyft’s infrastructure for real-time feature generation has undergone significant transformation, moving from a simple but latent system to a highly optimized, scalable streaming architecture.

Phase 1: From Cron Jobs to Streaming

The initial approach was based on a series of cron jobs, each triggering at the top of the minute. A job would pull data from Kinesis, process it, and store it in a temporary location like Redis for the next job to pick up. This system was functional at a small scale but suffered from inherent latency; a downstream process had to wait for the next minute to start, even if the upstream job finished in seconds. This latency negatively impacted model performance.

To address this, Lyft re-architected the system using a streaming-based solution with Apache Beam on top of Apache Flink. The team chose the Apache Beam Python SDK to align with their existing Python-based ecosystem and the skill set of their engineers and data scientists. This shift to asynchronous stream processing, where data is processed as it becomes available, was a major step forward.

Phase 2: Solving the “Hot Shard” Problem with Geohashes

While the initial streaming proof-of-concept was successful, scaling it to multiple cities revealed a classic “hot shard” issue. The pipeline was designed to route all events for a specific city to a single node for processing. However, a few major cities like New York and San Francisco account for a disproportionately large volume of traffic (~80%). This resulted in some nodes running at 90% CPU utilization while others sat nearly idle at 10%. In a stream processing system, the entire pipeline’s throughput is dictated by its slowest node, so this bottleneck created a significant performance drag.

The solution was to change the sharding key from city to geohash (a representation of a city block). This strategy broke down the traffic from a few hundred cities into millions of nearly uniformly distributed geohashes. This allowed the load to be spread evenly across all available nodes, eliminating the hot shard problem. The most computationally expensive work, like filtering and aggregation, is performed at the granular geohash level. The resulting, much smaller, aggregated data can then be efficiently re-sharded by region or sub-region to be fed to the models.

System Design and Architecture

A Custom Geospatial Feature Store

Traditional feature stores, often designed for point-in-time lookups on a single key (e.g., a user ID), were insufficient for Lyft’s needs. Lyft’s models operate on aggregated data over geographical areas. To serve this, the team built a custom geospatial feature store.

This store is designed to handle hierarchical data. A region contains multiple geohashes (e.g., GH4), which in turn contain smaller geohashes (GH5, GH6), forming a tree-like structure. The API allows a data scientist to request data for a specific region and specify the desired level of granularity (e.g., GH4 level). This design enables a single, efficiently generated feature to be shared and consumed by multiple models with different geographical scopes.

The Data Contract: Bridging Data Science and Engineering

A strong data contract exists between the data science consumers and the platform engineering producers. The typical workflow is as follows:

1. A data scientist develops a new feature using offline data stores.
2. They validate the feature’s impact on model performance in an offline environment.
3. Once the value is proven, they bring the feature logic (often an SQL query) to the platform team to be productionized.
4. The platform team’s contract is to convert the offline query into a real-time streaming pipeline, guaranteeing that the online feature values match the offline ground truth with ~99.9% accuracy.

An in-house framework continuously samples and compares the real-time features against their offline counterparts to monitor for quality degradation and alert on any divergence. The small gap in parity is attributed to the realities of real-time data, such as delayed events from users with poor connectivity.

Optimization and MLOps Best Practices

Centralized Processing and Pipeline Bundling

As the number of pipelines grew, the team observed that many performed similar initial processing steps (e.g., data filtering, user state creation). To reduce redundant computation and cost, they abstracted these common tasks into a centralized, two-stage pipeline.

A single pre-processing pipeline now handles initial data filtering and builds a state for each user (e.g., booked_ride, dropped_off). This state is then emitted for various specialized downstream pipelines to consume for their specific aggregation logic. This “divide and conquer” approach, where common logic is centralized and specialized logic is branched out, significantly improves efficiency.

From Code to Configuration: The YAML-Based Pipeline

To further accelerate development and empower data scientists, Lyft created a system for declarative pipeline generation. Instead of writing complex streaming code, users define their feature pipelines in a simple YAML configuration file. This file, often just 20-30 lines long, specifies the desired operations. The platform abstracts away the underlying complexity of windowing, joining, and other streaming concepts, reading the YAML file to construct and deploy the full pipeline automatically. This innovation reduced the code required to generate a feature from thousands of lines to a short configuration file, cutting the codebase by as much as 60% in some cases.

Strategic Technology Adoption

Decisions to adopt new technologies are weighed carefully against several factors:

• Problem Fit: Is it the right tool for the job, both now and in the future?
• Development & Operational Cost: How much effort is required to build and maintain solutions with this technology?
• Ecosystem Integration: How well does it integrate with Lyft’s existing Python-heavy environment? This was a key reason for choosing Apache Beam’s Python SDK over native Flink/Java.
• Open Source Support: Is the technology well-adopted and supported by the community?

• Tags:
• Feature
• MLOps
• Real-Time ML
• Feature Generation
• Streaming
• Apache Flink
• Apache Beam
• Feature Store
• Lyft