Upwork's Radical Bet on Reinforcement Learning: Building RLEF from Scratch | Andrew Rabinovich (CTO)
Andrew Rabinovich, CTO and Head of AI at Upwork, details their strategy for building AI agents for digital work. He introduces a custom reinforcement learning approach called RLEF (Reinforcement Learning from Experience), explains why digital work marketplaces are ideal training grounds, and shares his vision for a future where AI delivers finished projects, orchestrated by a meta-agent named Uma.
Upwork’s AI Strategy: From Matchmaking to Work Delivery
At Upwork, the AI strategy is centered around a meta-agent named Uma (Upwork’s Mindful AI). Initially, Uma’s role is to facilitate the connection between clients and freelancers by understanding a client’s project needs and recommending the right talent. This represents a shift from a traditional marketplace to an AI-guided platform. The long-term vision, however, extends far beyond matchmaking to a point where a client describes a project to Uma, and Uma delivers the completed work.
A Hybrid AI Architecture: MoE, RAG, and Knowledge Graphs
Upwork employs a sophisticated, hybrid AI architecture rather than relying on a single monolithic model. The system is designed as a Mixture of Experts (MoE), where Uma possesses various specialized “skills,” each fine-tuned for a specific task:
- Creating a detailed job post from a client conversation.
- Identifying and ranking suitable freelancers.
- Assisting freelancers in drafting compelling proposals.
- Helping clients evaluate and select freelancers based on those proposals.
To ground these skills in real-time platform data, Upwork heavily utilizes Retrieval-Augmented Generation (RAG). A crucial component of this is a knowledge graph that serves two purposes:
- Routing: It directs queries to the appropriate data sources and RAG systems.
- Inference and Query Expansion: It understands relationships between concepts, allowing for more intelligent search. For example, a search for a “front-end developer” can be automatically expanded to include related skills like “JavaScript” or “React,” which are then used to retrieve a richer context for the language model.
A key advantage for tuning this RAG system is Upwork’s vast amount of “self-curating” data. A successful contract between a client and a freelancer serves as a strong positive label, validating the effectiveness of the search and matching process. This feedback loop allows for continuous optimization of data sources and retrieval algorithms.
The Thesis: Digital Work as the Ultimate Agent Training Ground
Digital work marketplaces provide a unique and powerful environment for training AI agents, superior in many ways to traditional methods like simulation or game-based self-play.
If you allow agents to learn from an environment that is realistic, relevant relatively to their real world, then the results are really incredible… digital work is the kind of domain where it is almost okay to make mistakes so long that you can learn from them.
Unlike training self-driving cars where mistakes have severe consequences, or game environments like Go where the reward function is clearly defined, digital work offers a real-world setting with low-stakes failure. This allows agents to explore unconventional solutions—the equivalent of AlphaGo’s “Move 37”—in creative and business tasks. The challenge in the real world is the absence of a predefined value function. Upwork solves this by leveraging its network of human experts who can evaluate the agents’ outputs and provide the necessary reward signals.
RLEF: A New Paradigm Beyond RLHF
To train these agents, Upwork is developing a novel framework called Reinforcement Learning from Experience (RLEF), which diverges significantly from the more common Reinforcement Learning from Human Feedback (RLHF).
- RLHF: Typically involves humans ranking a set of machine-generated outputs (e.g., A is better than B). This confines the model’s learning to the boundaries of human preference and imagination.
- RLEF: Allows an agent to freely explore a vast landscape of possible solutions. A human expert then provides a direct reward signal on the final output, similar to classical reinforcement learning. This encourages the agent to discover solutions that a human might never have conceived.
To overcome the sample inefficiency of RL, Upwork’s approach leverages the strong correlations between different digital work tasks. For instance, skills learned in graphic design can be applied to website design. This multi-task learning framework creates regularization, allowing policies for one task to inform and constrain the learning for another, thereby reducing the number of samples needed. This has necessitated the development of a custom RL library from scratch, as existing tools do not fit this new paradigm.
Practical Implementation: UX, Evaluation, and Vision
The current focus of this initiative, which has been in development for about five months, is on creating a multi-modal user experience for human-agent collaboration. The goal is to move beyond text prompts and allow experts to provide rich, actionable feedback—for example, by directly annotating an image to indicate that a specific area has the wrong color saturation.
Agent evaluation begins with human experts. The data from these evaluations is then used to train a specialized “LLM as a Judge” for each category of work. The underlying principle is that verifying a solution’s quality is a much simpler problem than generating the solution from scratch.
Future Vision:
- 5-Year Goal: A client will state their objective to Uma, and after a specified time and budget, Uma will deliver the finished product. In the background, Uma will have orchestrated a team of human experts and AI agents to complete the work.
- 6-12 Month Impact: While the types of work won’t change dramatically, projects will be delivered significantly faster and more affordably.
- Freelancer-Built Agents: Upwork plans to empower freelancers to build their own “second-party agents” using their deep domain expertise. These agents, after passing a quality evaluation, can be deployed on the marketplace.
A Call for Smaller, Physics-Informed Foundation Models
Looking ahead, the ideal foundation models would be smaller, faster, and less reliant on pure scale. There is a belief that current large models contain immense redundancy and that simply scaling them is not the path to AGI. Instead, progress will come from integrating more symbolic AI constructs and real-world knowledge, such as the laws of physics, into the models, moving beyond statistical pattern recognition on internet data.