The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

Filip Makraduli from Superlinked discusses the common infrastructure gaps and profiling mistakes encountered when deploying small embedding and transformer models. He introduces the Superlinked Inference Engine (SIE), an open-source solution designed for dynamic model loading, hot-swapping, and memory-aware eviction to maximize GPU utilization and streamline the path from development to production.

May 5, 2026 4 min read

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked — source: https://www.youtube.com/watch?v=qdh_x-uRs9g

The journey into building a specialized inference engine began with a realization: while deep knowledge of model internals like flash attention is valuable, the practical challenges of deploying models in production, especially small ones, represent a significant and often overlooked blind spot. This led to the creation of the Superlinked Inference Engine (SIE), an open-source project designed to bridge the gap between model development and scalable, efficient production deployment for AI search and document processing.

The “Why”: Solving Context Rot in Agentic Workflows

A primary motivation for focusing on small model inference is tackling the problem of “context rot,” where the quality and performance of large models degrade as the input context size increases. Small, specialized models are a powerful tool for context management and data preprocessing before feeding information to larger agents.

Preprocessing Data: Small models can be used to filter, rank, or extract entities (e.g., using Named Entity Recognition to build knowledge graphs) from large documents, providing a more concise and relevant context for agents.
Tool Calling: These models can be framed as “tools” that an agent can call upon. For example, in an e-commerce setting, a taxonomy classification model can be used as a tool to navigate and retrieve data efficiently.
Community Adoption: This approach is validated by leaders in the field, such as Andrej Karpathy building knowledge bases and vector databases like Chroma releasing models specifically for context filtering.

What Inference Is Not About

Common misconceptions about inference can lead to inefficient and costly systems, particularly for smaller models.

It’s Not Just About Adding More GPUs: Small models like embedding models (e.g., Stella), rerankers, or NER models (e.g., Glyner) often occupy only a few gigabytes of GPU memory. Dedicating an entire GPU to each model results in significant underutilization and wasted resources. The better approach is to hot-swap multiple models on a single GPU, using a memory-aware eviction policy (like Least Recently Used) to maximize hardware utilization and reduce costs.
It’s Not Just a Server: A complete solution requires more than just an API wrapper around a model. There is a significant gap in the open-source ecosystem for an end-to-end solution that handles routing, auto-scaling, monitoring (e.g., with Prometheus and Grafana), and GPU provisioning out of the box.

The Yin and Yang of Inference: A Holistic Approach

A robust inference solution must balance two critical components: comprehensive model support (the Yin) and production-grade infrastructure (the Yang).

The Yin: Broad and Flexible Model Support

The open-source AI landscape is exploding with millions of models on platforms like Hugging Face, many of which outperform managed services on specific, narrow tasks. Supporting this diversity is essential but technically challenging, as there is no universal engine that can handle the architectural variations.

Architectural Differences: Models like BERT, Qwen, and ColBERT have different implementations for key components:
- Normalization: Implemented differently across model families.
- Attention: Some models use grouped query attention, preventing the fusion of query, key, and value (QKV) matrices.
- Positional Embeddings: Vary from absolute lookups (BERT) to rotary positional embeddings (Qwen).
The Solution: To handle this diversity, SIE re-implements the forward pass to normalize these differences. This includes adapting attention mechanisms and implementing variable-length flash attention, which allows for proper padding and batching without wasting compute on empty tokens. This flexibility is crucial for supporting not only standard embedding models but also late-interaction models like ColBERT (which outputs multiple vectors) and cross-encoders/rerankers (which output scores instead of vectors).

The Yang: Production-Grade Infrastructure

The second half of the solution is providing the entire infrastructure cluster, not just the inference code. This allows users to go from configuration to production quickly.

End-to-End System: The architecture includes API primitives (encode, score, extract), a router, and a queuing mechanism that adapts to the load.
Intelligent Resource Management: It manages different pools of GPUs, including spot instances and larger GPUs, to distribute workloads efficiently.
Built for Scale: The system uses KEDA for auto-scaling based on Prometheus metrics, ensuring resources are provisioned dynamically. The entire stack is delivered via Terraform, Helm charts, and Docker images, making deployment and model management as simple as changing a config file and running terraform apply.

By combining deep model support with a comprehensive, scalable infrastructure, SIE provides a complete, open-source solution for the unique challenges of small model inference.