Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face

Merve Noyan discusses the pitfalls of directly using large Vision Language Models (VLMs) at runtime due to performance limitations and restrictive licensing. She introduces "Web Vision," a toolkit that enables efficient training of smaller, task-specific computer vision models using a novel "vibe training" methodology. This approach leverages VLMs for initial labeling and judgment, followed by training efficient models like RF-DETR, all while adhering to Apache 2.0 licenses and costing only a few dollars.

September 23, 2026 6 min read

Skill issue: stop deploying vision language models, use them with Skills — Merve Noyan, Hugging Face — source: https://www.youtube.com/watch?v=dKcTBQzR7jI

Merve Noyan advocates for a shift away from direct, runtime usage of large Vision Language Models (VLMs) in favor of more efficient and ethically licensed solutions. While acknowledging her affection for VLMs—even authoring a book on them—she highlights two primary concerns with their direct application by developers.

Challenges with Direct VLM Usage

Performance Limitations: Direct VLM inference struggles to achieve real-time performance. Even on basic hardware (“a toaster”), task-specific detectors like RF-DETR can achieve 30-40 frames per second (FPS) and consistently outperform VLMs in robustness and speed for tasks like image classification or instance segmentation.
Licensing Issues: Many popular object detection models, such as YOLO, are distributed under copyleft licenses (e.g., AGPL 3.0). Developers often deploy these without fully understanding the licensing implications, potentially leading to compliance issues or unexpected payment obligations. Noyan strongly recommends migrating to Apache 2.0 licensed models.

Web Vision: A Toolkit for Efficient Computer Vision Development

To address these challenges, Noyan introduces “Web Vision,” a toolkit designed to empower developers to build robust vision applications end-to-end, leveraging smaller, Apache 2.0 licensed models. This toolkit is inspired by the concept of giving powerful models as “tools” to a “clueless computer vision engineer”—a coding agent.

The toolkit prioritizes:

License: Strict adherence to Apache 2.0, MIT, or other non-commercial licenses.
Performance: Models must offer competitive performance relative to their size and architecture, as validated by benchmarks and academic research.
“Vibes”: Personal preference and the practical utility derived from experience.

Vibe Training: A Novel Data Labeling and Model Training Pipeline

The most innovative part of Web Vision is “vibe training,” a methodology for training task-specific models from unlabelled image datasets, making VLM capabilities accessible without direct runtime deployment. The pipeline consists of three main stages:

VLM as a Labeler: An initial large VLM, specifically Qwen 3.5 9B, is used to label the raw image dataset. Instead of outputting bounding boxes as tokens, the toolkit overlays these predicted bounding boxes directly onto the images.
VLMs as Judges: The “labeled” images (with overlaid bounding boxes and associated label descriptions) are then passed to two smaller VLM judges: Gemma 4 8B and LFM 2.5VL 2B. These judges evaluate the quality of the initial VLM-generated labels.
- Judgment Merging: Crucially, judgments are merged based on minimum agreement rather than consensus. If at least one judge approves an example, it is kept. This approach counteracts a significant judge imbalance observed, where LFM tends to reject a large proportion of examples, which would otherwise lead to insufficient training data if consensus were required.
- Prompt Generation: The label descriptions and judge prompts are themselves generated by coding agents, though human approval is still required to ensure prompt quality.
Task-Specific Model Training: The refined, VLM-labeled dataset is then used to train a specialized, efficient model, such as RF-DETR (medium or large). This model is designed for high performance and robustness for specific computer vision tasks.

Infrastructure and Cost-Effectiveness

The entire pipeline is designed to run efficiently on Hugging Face infrastructure, leveraging:

Hugging Face Jobs: For one-off batch processing and training.
Inference Providers: For serverless routing and scalable inference.
Data Storage: Buckets for intermediate data, alongside dataset and model repositories.

The full end-to-end process, from labeling to training, is remarkably cost-effective, estimated at $3-4. This is achieved by strategically using different services: DeepInfra for Qwen 3.5 labeling (due to its low cost), Hugging Face Jobs for judging and training (e.g., on L4 GPUs for RF-DETR).

Performance and Generalization

Noyan validated the “vibe training” pipeline on two distinct problems:

Road Sign Detection: The trained RF-DETR model achieved a good mean average precision (mAP) against ground-truth annotations, demonstrating its effectiveness in learning from VLM-generated labels. While a gap existed compared to direct Qwen annotations, the results were promising.
Document Parsing: The trained RF-DETR model exhibited strong generalization capabilities. In one notable instance, it successfully detected a signature in a document that the initial Qwen VLM labeler had missed, showcasing the benefit of the refined training approach.

Key Findings and Lessons Learned

Beyond the technical pipeline, Merve Noyan shared critical insights gleaned during the development:

Judge Imbalance: As mentioned, the difference in rejection rates between judges (LFM being more conservative) necessitated the “minimum agreement” strategy to avoid data scarcity.
Human-in-the-Loop for Prompts: While coding agents can generate judge prompts, human oversight and approval remain essential to ensure their quality and relevance.
Coding Agent Limitations (“Clueless CV Engineer”): Even advanced coding agents (like Opus 4.8) lack common sense in computer vision contexts. For example, they might suggest inappropriate data augmentations like horizontal flipping of traffic signs or color jittering of traffic lights, which can corrupt the dataset. This highlights the need for explicit instructions or patches to guide augmentation strategies.

Preferred Models as Tools

The second part of the Web Vision toolkit comprises a curated selection of efficient, Apache 2.0 licensed models for various computer vision tasks, acting as specialized tools for the coding agent:

Zero-shot Segmentation: Falcon Perception (600M parameters, Apache 2.0) is highlighted for its ability to perform complex, open-ended referential segmentation (e.g., “segment the red car next to the orange car”), surpassing SAM’s capabilities in this area.
Pose Estimation: Models from the Sapiens family for human-centric tasks.
Zero-shot Detection: Moon Dream 3 and MM Grounding DINO (a smaller, Apache 2.0 alternative).
OCR: Various models selected from OCR benchmarks, catering to different sizes and performance needs.
Depth Estimation: Specific Apache 2.0 licensed models, noting that some larger commercial models lack open licenses.
Integration: The toolkit also supports Roboflow’s supervision and tracker libraries for instance tracking.

Future Plans

Noyan’s future work includes:

Image-Guided Detection: For industrial use cases where natural language descriptions are insufficient for parts (e.g., detecting a specific part from an example image across other images).
IoU-based Judgment Merging: Exploring Intersection over Union (IoU) as a more nuanced metric for merging judge verdicts, where judges generate bounding boxes, and their overlap determines acceptance, rather than binary approve/reject.
Segmentation Support: Expanding the toolkit to fully support segmentation tasks.