Your Agent Can Now Train Models — Merve Noyan, Hugging Face

Open Source Models Have Caught Up

The landscape of AI has shifted, with open-source models now matching and even exceeding the performance of their closed-source counterparts. The Artificial Analysis intelligence index shows models like GLM 5.1 leading the charts, demonstrating that the performance gap has closed.

This parity brings significant practical advantages beyond benchmarks. With full access to model weights, developers gain crucial capabilities:

• Quantization: Models can be shrunk to run on less powerful hardware.
• Fine-tuning: Models can be adapted to specific tasks and datasets.
• Privacy & Control: Deploying models to edge devices or browsers ensures that user data never leaves the local infrastructure, guaranteeing privacy.
• Transparency: With open models and code, there are no surprise performance degradations or unexpected changes, as everything is public.

The Hugging Face Hub: An Ecosystem for Open AI

The Hugging Face Hub serves as the central infrastructure layer for the open-source AI workflow, hosting millions of models, datasets, and Spaces. The platform has evolved to specifically support the growing agentic ecosystem.

Finding and Comparing Agentic Models

Navigating millions of models can be challenging, but the Hub provides several tools to simplify the process:

• Model Filtering: You can directly filter for agentic models, which are often trending. A key trend is the “Day Zero” release of Large Language Models (LLMs) with vision capabilities (VLMs), such as Gemma 4 and Qwen 3.5.
• Benchmark Datasets: To objectively compare models, the Hub integrates benchmark datasets like SWE-bench (for coding) and AIME. This allows users to see ranked lists of open models based on their performance scores, making it easier to select the best one for a given task.
• Inference Providers: This service routes requests to the best provider for a given model, offering the cheapest or fastest option. It also includes a “tool use” column, helping users select models specifically for agentic use cases.

Running Local and Remote Coding Agents

The ecosystem provides robust support for running coding agents both locally and remotely.

• Local Agents: Tools like Pi and llama.cpp make it simple to set up a local coding agent. llama-agent is baked directly into llama.cpp as a binary, allowing you to start a local agent with a single command pointing to a Hugging Face Hub model ID.
• Hermes Agents: A particularly powerful option is Hermes Agents, which offers advanced features like sophisticated memory management. It can be run locally or with Hugging Face Inference Providers and integrates easily with platforms like Slack or WhatsApp. The speaker shared a personal success story of using GLM 5.1 with a Hermes agent to autonomously debug and fix a faulty Slack integration.
• Agent Traces: Hugging Face has introduced a new repository type called “traces.” This allows developers to store, explore, and parse agent sessions from tools like Claude Code or Pi. These structured traces can later be used as a dataset to fine-tune other models, creating a powerful feedback loop.

Supercharging Agents with Hugging Face Skills

Hugging Face Skills are a transformative feature that plugs the Hub’s capabilities directly into coding agents, allowing them to perform complex MLOps tasks.

> “You just go to your agent and say, ‘Train Qwen 3.5 on this dataset for me,’ and then it just trains, which to me is like a sci-fi at this point.”

Key skills include:

• LLM Trainer Skill: Enables an agent to fine-tune LLMs and VLMs on a specified dataset. The agent handles the entire backend process, including calculating the required VRAM, selecting the right compute instance, and launching the training job.
• Hugging Face CLI Skill: Allows agents to manage repositories, run jobs, and launch demos on the Hub.
• Gradio Skill: Lets agents build and launch interactive demos.
• Dataset Skill: Gives agents the ability to explore datasets using the Dataset Viewer API.

This new paradigm was demonstrated by asking Claude Code to fine-tune a Qwen2-VL model on the lava-instruct-mix dataset. The agent intelligently prompted the user for necessary details (like the validation split) and handled the rest automatically, transforming a task that once required a full day of “napkin math” and manual setup into a single prompt.

A Practical Example: OCRing 30,000 Papers

To illustrate the power of this agent-driven workflow, a real-world project was presented: OCRing 30,000 AI research papers on the Hub to make them searchable. The entire process was orchestrated by prompting an LLM:

1. Select a Model: The agent used the OCR-bench benchmark to choose a cost-effective and performant OCR model.
2. Write the Code: The LLM was prompted to write the processing script for the job.
3. Execute the Job: The agent calculated the required instance type and cost, then kicked off the processing job on the Hugging Face infrastructure using products like Jobs and Buckets (a cheaper, faster alternative to S3).
4. Profit: The task was completed with minimal human intervention, showcasing an end-to-end, prompt-driven MLOps workflow.

• Tags:
• LLM
• Open Source
• VLM
• Fine-Tuning
• Agents
• Hugging Face
• Hugging Face Hub
• Hugging Face Skills