Fine-Tuned Models Are Getting Out of Hand

Fine-Tuning and RAG: A Complementary Approach

Fine-tuning and Retrieval-Augmented Generation (RAG) are not competing but complementary technologies for building intelligent systems. The choice between them depends on the nature of the data and the task at hand.

• RAG for “Fast Data”: RAG excels when dealing with factual, rapidly changing information that requires retrieval. This is analogous to “fast data.” A customer support system is a prime example, where an agent can retrieve answers to semantic queries from an ever-growing knowledge base. It’s about recovering static, factual information.

• Fine-Tuning for “Slow Data”: Fine-tuning is better suited for emulating a decision-making process, or “slow data.” This involves capturing the nuanced, intrinsic patterns of how an individual thinks and makes decisions based on multiple variables. It’s less about retrieving a specific fact and more about replicating a thought process. For example, you use a generic model to summarize a document, but you ask a trusted colleague for their opinion on whether a new tool complies with company policy. The colleague’s knowledge is ingrained, similar to a fine-tuned model.

The ideal system uses both. A fine-tuned model possesses the contextual awareness to ask the right questions of the RAG system. A vanilla base model might struggle with framing intermittent queries for effective retrieval, but a model fine-tuned on enterprise data understands the context and can therefore query the RAG system more effectively to retrieve the most relevant information.

The Anatomy of a Personalized AI Agent

An AI agent can be broken down into three core components:

1. Intelligence Layer: The decision-making brain, ideally a fine-tuned Small Language Model (SLM).
2. Memory Layer: The system for learning and remembering information, often powered by RAG or other memory frameworks.
3. Actions Layer: The tools and capabilities the agent can use, such as reading/writing files, sending messages, or controlling a browser.

A significant challenge with current AI systems is that they are “low trust systems in a high trust environment.” People are often skeptical of AI-generated content. To overcome this, the focus must be on interaction design, making agents not only think but also behave more like a human collaborator.

Building Personalized Agents with Small Language Models (SLMs)

SLMs (in the 1-7 billion parameter range) are excellent candidates for the intelligence layer of personalized agents. While not as powerful off-the-shelf, they are like “good students”—fresh and ready to learn.

The benefits of using SLMs include:

• Trainability: They require significantly less compute, making them cheaper and faster to train.
• Deep Knowledge Instillation: Unlike larger models which often only get LoRA adapters, SLMs can be fully fine-tuned. This allows for instilling much deeper, more nuanced process knowledge and intrinsic patterns into the model’s weights.
• RL Capabilities: Reinforcement Learning (RL) can be run on them more effectively to improve reasoning.

The process of creating a personalized agent involves:

1. Data Collection: The system observes the user to learn their unique workflows and decision-making patterns. This data can come from Slack conversations, documents, notes, or even screen recordings that capture the steps taken to complete a task. Data privacy is paramount, ensuring users have full control over where their data goes and how it’s used.
2. Fine-Tuning: This “process knowledge” is distilled into a personalized SLM. The training objective is for the model to accurately predict the user’s next action given a specific situation and context.
3. Action and Deployment: The fine-tuned agent is given access to tools (e.g., calendar, email, GitHub). It can then act as a digital sidekick, joining a Slack channel or managing an email inbox, responding and acting just as the user would.

The Future of Agentic Workflows

The next evolution for AI agents is moving beyond conversational interfaces to taking direct action within a user’s digital workspace.

• Human-in-the-Loop for Creative Tasks: For complex, creative, or “information-sparse” domains like video editing or tasks involving custom software without APIs, agents won’t be fully autonomous initially. Instead, they will act as human-augmented assistants. A user might collaborate with the agent on a plan of execution, review its progress at key checkpoints, and provide feedback, rather than delegating the entire task and hoping for the best.

• The Rise of Computer Use: Since many consumer and specialized applications lack robust APIs, agents will need to interact with them via their graphical user interfaces (GUIs). While current “computer use” frameworks can be slow or unreliable, their performance will improve dramatically with purpose-specific training data. This is where fine-tuning becomes critical; you cannot simply prompt a model with the immense context required for long-tail GUI automation tasks.

• An OS-Level Orchestrator with a Family of Models: The future architecture will likely involve a family of specialized, fine-tuned models rather than a single monolithic one. You might have a “Demetrios Photoshop model” and a “Demetrios Google Ads model,” each an expert in its domain. These models would be managed by a dynamic, OS-level orchestrator that hot-swaps the appropriate model into memory at runtime based on the application in use. This vision leads to a future with a “billion models fine-tuned for specific people and applications,” enabling truly personalized and effective AI collaboration.

• Tags:
• AI Agents
• RAG
• Fine-Tuning
• Computer Vision
• Agentic Workflows
• SLMs
• Personalized AI