From Scratch to SOTA: Training a 3B State-Space Vision Model — Krishna Prasad Srinivasan, Sarvam

Sarvam introduces Vision India, a 3B parameter Vision Language Model achieving state-of-the-art in document AI for English and 22 Indian languages. It leverages a State Space Model (SSM) backbone for efficiency with long documents and a novel block-level OCR approach. A four-stage training curriculum, including advanced RL with verifiable rewards, and a robust data engine for low-resource languages underpin its superior performance, enabling the digitization of millions of pages for critical sectors.

September 23, 2026 4 min read

From Scratch to SOTA: Training a 3B State-Space Vision Model — Krishna Prasad Srinivasan, Sarvam — source: https://www.youtube.com/watch?v=T72nqdC92PM

Krishna Prasad Srinivasan introduces Sarvam’s Vision India VLM, a 3-billion parameter model that achieves state-of-the-art (SOTA) in document AI, outperforming models 100 times its size. Developed entirely in India, this model supports English and 22 official Indian languages, addressing a significant gap where less than 1% of Common Crawl data is in Indian languages, despite India being a major market. Sarvam, a “sovereign foundation model company,” highlights the challenge of Indic document intelligence: the need for knowledge extraction over pure text, the complexity of Indian Unicode scripts, and the low-resource nature of most Indian languages.

Sarvam made a “contrarian bet” in late 2025 by focusing on block-level OCR with a layout and reading order harness, rather than monolithic page-level VLMs—an approach now widely adopted. Crucially, the model’s backbone is a State Space Model (SSM) instead of a transformer. While transformers incur quadratic compute and memory costs with sequence length, making them expensive for long documents (5,000-10,000 visual tokens), SSMs maintain constant memory and linear compute growth by updating a single state token by token, proving more efficient for document processing.

Training Curriculum

The training employs a four-stage curriculum:

Text-only Pre-training: 13 trillion tokens (English, Indian languages, math, code) build a strong 3B parameter language backbone, enabling the model to resolve ambiguous text using a strong language prior before seeing any pixels.
Continual Pre-training: 300 million image-text pairs develop general vision capabilities.
Supervised Fine-tuning (SFT): 100 million OCR samples make the VLM robust for OCR across diverse languages and document types (tables, equations, handwritten text).
Reinforcement Learning (RL): Pushes performance further. OCR correctness is “machine checkable,” allowing for scalable “verifiable reward RL” using unit tests on metrics like character error rate (CER), table structure, or grammar.

Data and Evaluation

A key “moat” for Sarvam is its data engine for low-resource languages, which creates synthetic data and “doc data” from real documents, continuously improving based on evaluation. The eval layer features extensive, curated benchmarks to ensure SOTA and practical usefulness. The team is exploring “RLMs” for agentic vision capabilities.

Results and Impact

Sarvam’s model achieved SOTA on English benchmarks (CRB bench 84.3%, Omnidoc bench 93.2%) and holds an “unbeaten lead” across 22 Indian languages, surpassing frontier models like Gemini and ChatGPT. It powers “Aure,” an agentic document intelligence workbench, and is currently digitizing over 35 million pages for various sectors including insurance, banking, and government. A public “Sarvam Indic benchmark” from the 1800s to today, covering diverse layouts and document types, will be released soon.

Key Takeaways from Q&A

Low Resource Transfer: The model’s multi-lingual capability significantly enhances English performance and is applicable to any low-resource language.
Synthetic Data in RL: While synthetic documents aid pre-training, SFT, and RL, using complex real-world documents is preferred for RL to set up unit tests for specific rewards.
Sovereignty: This aspect is vital for India, enabling control over data and deployment in a country with significant digitization needs. Sarvam’s model addresses this, demonstrating strong market demand by digitizing 35 million pages in four months, highlighting the importance of local AI capabilities, quality data creation, and SOTA models for national digitization efforts. The training data comprises 40% Indic languages.