Large clusters for small models — Daniel Svonava, Superlinked
Daniel Svonava argues that small open-source models, when specialized and properly served, can achieve frontier performance for specific tasks, offering significant cost savings and latency improvements over large, generalized models or managed endpoints. He highlights challenges with existing serving infrastructure for a fleet of diverse small models and introduces Superlinked's open-source solution, which leverages a pull-based, centralized queuing system, a Rust sidecar for runtime abstraction, and an autoresearch loop for automated model tuning, enabling high throughput (e.g., half a million embedding tokens/sec on one GPU) and efficient deployment of specialized models and LoRAs.
