Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face

The Hugging Face platform currently serves over 14 million users and hosts 3 million public models, 1 million datasets, and 50,000 organizations. This represents a massive growth, with models increasing 150x in a few years (from 20,000 to 3 million). Datasets also grew from 10,000 in 2022 to 1 million today. This rapid expansion necessitates robust infrastructure decisions, particularly concerning search functionality. While 20,000 models made any query fast, 3 million models demand a re-thinking of search, as users expect instant results. A slow search for even 1% of 14 million users means 140,000 people having a poor experience, emphasizing the priority of P99 latency over P50. This drives investments in pre-computed tokens, denormalized read collections in MongoDB, Apache Lucene-based full-text search, Kubernetes autoscaling, and future database sharding.

Search Architecture and Data Storage

The Hugging Face Hub’s architecture separates metadata from model artifacts to optimize scaling:

• Metadata Storage: MongoDB Atlas serves as the source of truth for metadata, including user info, repository details, model and dataset metadata, configuration, billing, and access control.
• Artifact Storage: Actual model artifacts, tokenizer files, and associated assets are stored separately in cloud object storage like AWS S3. This decoupling allows independent scaling and optimization of metadata, binary storage, and compute.
• Request Flow: User requests flow from the frontend through the Hub API, running on Kubernetes. This API interacts with an optimized, denormalized read collection in MongoDB Atlas.

Full-Text Search Implementation

To ensure instant search results, Hugging Face implemented a sophisticated full-text search mechanism:

• Pre-computed Tokens: Model names are tokenized at insertion time, not query time. For example, “meta-llama/llama-3-8b” is split into tokens like “meta”, “llama”, “3”, “8b”, and stored in an array within the MongoDB document.
• Atlas Search with Apache Lucene: Atlas Search, which leverages Apache Lucene under the hood, is used for instant autocompletion and search. MongoDB’s core does not offer full-text search; instead, it uses a wrapper process called mongot around Lucene, transparent to the end-user.
• Evolution of Search Queries:
  • Past Approach: Initially, a classical find method on the models collection with a regex operator was used to search within search_tokens arrays, with results sorted by a trending score (downloads and likes over the last seven days). This approach failed to scale as the dataset grew due to regex’s poor performance at scale.
  • Current Approach: The system migrated to Atlas Search using the $search operator within an aggregation pipeline. This unifies the query API, pointing to a specific Lucene index (e.g., model_search_autocomplete), and still allows sorting by the trending score. This solution is significantly more efficient and has resolved latency issues.

MongoDB Cluster Architecture

Hugging Face utilizes a seven-node MongoDB replica set cluster to handle millions of queries across hundreds of services:

• Primary Node: All inserts, deletions, and updates are routed exclusively to the primary node. Queries requiring strong consistency also target the primary.
• Secondary Nodes: Reads are distributed across multiple secondary machines. Queries that do not require the absolute latest data are directed to secondaries. Secondaries continuously tail the primary’s oplog to stay in sync.
• Workload Distribution on Secondaries:
  • Complex Aggregations: Heavy aggregation pipelines involving large data scans, sorting, grouping, or transformation are offloaded to secondaries to prevent impacting primary performance.
  • Change Streams: Operations reacting to real-time data changes (e.g., cache invalidation, syncing to other data stores like AWS Redshift, event-driven workloads) are also run on secondaries due to their resource intensity.
• Hidden Analytics Node: A dedicated “hidden” node exists that is invisible to application drivers. It replicates data from the primary but does not interact with production traffic. This node is used for heavy reporting, ad-hoc queries, and experimental queries, completely isolating them from the production workload and ensuring consistent P99 performance for user-facing applications.
• Principle: The core principle is to allow the primary to focus solely on operations only it can perform, pushing all other workloads to different machines.

Future Database Scaling with Sharding

With the continuous growth of users and models, a single MongoDB replica set will eventually become insufficient. Sharding is the next step for horizontal scaling:

• Mechanism: Sharding involves splitting the dataset into pieces, with each piece (shard) residing on a separate replica set cluster. Each shard has its own primary and secondaries.
• Difference from Replica Set: Unlike a replica set where each node holds a full copy of the data, a sharded cluster distributes parts of the data across multiple shards.
• Scalability: Adding more shards allows the database to scale horizontally for CPU, memory, storage, reads, and writes. A shard key is crucial for effective sharding and data distribution.

Frontend and Infrastructure Autoscaling

The Hub’s frontend infrastructure also employs robust autoscaling mechanisms:

• Kubernetes Horizontal Pod Autoscaler (HPA): The Hub runs on Kubernetes and uses HPA to scale pods based on CPU and memory thresholds. During traffic spikes, new pods automatically scale up (from 10 to 500 pods) and scale back down when traffic drops, ensuring health and cost-efficiency.
• CastAI for Node Autoscaling: If HPA needs to add new pods but Kubernetes lacks available nodes, CastAI is used for node autoscaling. CastAI adds new nodes to the cluster, allowing the Kubernetes scheduler to place pending pods, providing a second layer of scaling at the infrastructure level.
• Migration to KEDA (Kubernetes Event-Driven Autoscaling): Hugging Face is migrating from HPA to KEDA:
  • HPA Limitation: HPA scales primarily on CPU and memory utilization, which might not always reflect actual application load (e.g., low CPU but a high request queue).
  • KEDA Advantage: KEDA scales based on real application metrics like requests per second or event loop utilization, directly reflecting the workload. This enables more precise and responsive scaling, as KEDA can identify bottlenecks like a high request queue that HPA might miss.

Conclusion

The overarching goal of these architectural decisions and scaling strategies is to maintain a seamless and instant user experience, regardless of the underlying complexity and massive scale of the Hugging Face Hub. The user should not have to think about the infrastructure; it “just works” whether they are pushing a model, searching the Hub, or downloading content.

• Tags:
• Distributed Systems
• Cloud Architecture
• Kubernetes
• Scalability
• Hugging Face
• MongoDB Atlas
• Apache Lucene
• KEDA
• Autoscaling
• Database Sharding
• Full-Text Search
• AWS S3
• Infrastructure Engineering