How Transformers Finally Ate Vision – Isaac Robinson, Roboflow

Isaac Robinson, the research lead at Roboflow, explains the fascinating evolution of computer vision models, detailing how Transformers ultimately “ate” a field long dominated by Convolutional Neural Networks (CNNs).

The Initial Contenders: CNNs vs. Vision Transformers (ViT)

The talk begins by contrasting the two primary architectures:

• Convolutional Neural Networks (CNNs): These models have a strong, built-in inductive bias inspired by the human visual cortex. The convolution operation is inherently translation-invariant, meaning an object is recognized the same way regardless of its position in the image. This makes them efficient and a natural fit for vision tasks, scaling with N^2 complexity.
• Vision Transformers (ViT): In contrast, ViTs are architecturally simpler but less intuitive for images. They treat an image as a set of patches (tokens) and apply a set-to-set operation (self-attention) with no inherent inductive bias. A pattern in the top-left corner can have a completely different activation than the same pattern in the bottom-right. This results in a computationally expensive N^4 scaling with image resolution.

The central question is: how did the less-biased, more computationally demanding ViT architecture win? The answer lies in two key factors: massive, ViT-specific pre-training and the leveraging of infrastructure and optimizations from the booming world of Large Language Models (LLMs).

The Architectural Evolution: A Full Circle

The journey to ViT’s dominance was not linear. It involved a series of innovations and counter-movements as researchers tried to find the optimal architecture.

1. Swin Transformer: The first major iteration on ViT attempted to re-introduce some of the beneficial biases from CNNs. Instead of global attention across all patches, Swin used localized attention within windows. By shifting these windows in subsequent layers, it allowed information to mix across the image in a manner reminiscent of convolutions. This added a locality bias and improved computational efficiency to N^2.

2. ConvNeXt: This model represented a pushback, arguing that the lessons from Transformers could be applied back to the proven CNN architecture. The researchers took a standard CNN and modernized it by incorporating a Transformer-like block structure (spatial mixing via convolution, followed by a feed-forward network), patchification, and other modern techniques like LayerNorm. When benchmarked, ConvNeXt successfully beat both ViT and Swin on ImageNet.

3. Hiera: This work from Meta explored the trade-off between inherent architectural bias and learned bias from pre-training. The researchers started with a highly biased model and systematically stripped away the inductive biases one by one. They found that with sufficient pre-training, the model could learn the necessary biases, often resulting in a faster, more general architecture.

The Decisive Factor: ViT-Specific Pre-training

The ability to learn biases, rather than hard-coding them, became ViT’s trump card. This was enabled by pre-training techniques that are uniquely suited to the Transformer architecture.

• Masked Autoencoding (MAE): Similar to BERT in NLP, MAE randomly masks a large portion of the image patches and tasks the model with reconstructing them from the surrounding context. This forces the model to develop a deep, contextual understanding of visual data, effectively learning the inductive biases that CNNs possess by default. This method is difficult to apply to CNNs due to the nature of the convolution operation.
• DINOv2 & DINOv3: These more advanced self-supervised methods took pre-training to another level. Without any human labels, these models produce incredibly rich and semantically meaningful feature maps. A PCA decomposition of features from a DINO-pretrained ViT can cleanly segment the paws of a cat or differentiate between roads and buildings in satellite imagery. The performance of these self-supervised models, measured by linear probing, began to approach that of fully supervised methods.

With these techniques, the simple, scalable ViT architecture could learn everything it needed from data, while optimizations like Flash Attention—borrowed from LLMs—mitigated its N^4 computational complexity, making it practical at scale.

Consequences and Deployment Challenges

This architectural evolution is mirrored in the development of practical foundation models like the Segment Anything Model (SAM).

• The original SAM used a ViT backbone with MAE pre-training.
• Subsequent attempts like MobileSAM and SAM 2 experimented with hybrid architectures (TinyViT, Hiera) to improve efficiency.
• Finally, SAM 3 returned to the simplest approach: using a massively pre-trained ViT backbone, accepting that the raw power from pre-training outweighed architectural tweaks.

However, this creates a new problem: a lack of deployment flexibility. A model like SAM 3 is incredibly powerful but is a “one-size-fits-all” solution. At 800 million parameters and 300ms inference time on a T4 GPU, it is impractical for the many real-world vision applications that require deployment on low-power, resource-constrained edge devices.

To solve this, Roboflow developed methods like RF-DETR, which use Neural Architecture Search (NAS) to transform a single, fixed foundation model into a whole family of high-performance models. This allows for tailoring the architecture to specific target data and hardware, achieving up to a 40x speedup over fine-tuning SAM 3 for the same accuracy. This combination of massive pre-training with flexible, hardware-aware deployment marks the final step in the Transformer’s conquest of computer vision.

• Tags:
• Foundation Models
• Pre-Training
• Feature
• Computer Vision
• Model Deployment
• CNN
• Vision Transformers
• ViT
• SAM