Research to Reality: Bringing Frontier ML Research to Production - Vaidas Razgaitis, Higharc

Vaidas Razgaitis, a Senior Research Engineer on Higharc’s labs team, outlines three tactical tips to accelerate the transition of novel AI/ML research into production-grade, customer-ready features. Higharc, operating in home building and spatial reasoning, heavily leverages various AI technologies, including computer vision, reasoning agents, custom transformers, and diffusion models, necessitating a robust process for integrating cutting-edge research.

The Core Challenge: Bridging Research and Production

The fundamental problem lies in the handoff between ML researchers and software engineers. ML researchers are adept at exploring frontier AI/ML concepts and developing novel features but often lack experience in building production-grade APIs. Conversely, software engineers excel at creating robust, production-ready code but may not be familiar with the methodologies, research, and domain specifics of areas like computer vision, LLM training, or other specialized AI/ML fields. The goal is to facilitate this baton pass productively.

Razgaitis identifies this as a systems and process problem, proposing three main focus areas to improve the velocity of bringing research into production:

1. Research Legibility
2. Codebase Structure
3. Decomposition and PR Review Plan

1. Research Legibility: The Research Prototype Taxonomy Document (RPTD)

To make a researcher’s prototype digestible for software engineers and product managers, Higharc uses a “Research Prototype Taxonomy Document” (RPTD). This is analogous to a technical design document (TDD) but tailored for machine learning contexts. Key components of the RPTD include:

• Domain Context: Explaining domain-specific lingo, data representations (e.g., party diagrams, circulation graphs, embedding models, latent space representations) crucial for a new engineer to understand.
• Business Goal: Articulating the ‘why’—the problem being solved and the value proposition of the ML tool.
• Type Safety: Defining the type contract between the core product repository and the ML repository, ensuring types are shared and synchronized.
• Persistence Layer: Mapping out the data persistence strategy, which serves as a natural entry point for software engineering involvement.
• System Architecture: Describing the overall anatomy of the research prototype, including workflows, chaining, external LLM calls, and dependencies.
• Merge and Decomposition Strategy: Outlining how the prototype will be integrated and broken down for production.

2. Codebase Structure: A Monorepo of Decoupled Microservices

Higharc maintains a separate, Python-based monorepo for all its AI/ML components, distinct from the core product repository. This monorepo is designed around cleanly isolated and fully decoupled microservices. The philosophy is often a one-to-one ratio of researcher to microservice, allowing independent iteration and growth.

• Deployment Architecture: A gateway guards requests from clients (e.g., web applications) and routes them to the appropriate microservice within a Docker Bridge network. Clients do not call microservices directly.
• Microservice Internal Structure: Each microservice follows a simple layered architecture:
  • API Layer: Exposed via FastAPI applications.
  • Controller Layer: Wraps the business logic.
  • Services Layer (Business Logic): Contains core business logic, handles external LLM calls to foundation models, and manages ML model weights via CI/CD.
  • Data Layer: Handles data interactions.
• Root Directory Contents: Each microservice’s root directory contains essential metadata, build instructions, a Dockerfile, project dependencies (using poetry or UV lock files), and clear documentation.
• Supporting Infrastructure: The monorepo includes GitHub Actions for automated testing, linting, formatting, and type checks. Jupyter notebooks run on Modal for GPU compute, and a custom CLI aids ML engineers in bundling these microservices for production.

3. Decomposition and PR Review Plan

Moving a proven research prototype from a monolithic state into the structured monorepo requires a thoughtful decomposition strategy, viewed as a design problem. The goal is to intelligently slice large prototypes into manageable, production-ready components.

• Slicing and Dependency Mapping: Teams analyze the axes for slicing projects and map out dependency graphs to understand how to break down the prototype.
• Stack Diffs with Graphite: Higharc utilizes Graphite for stack diffs, which facilitates the decomposition of large prototypes into smaller, interdependent pull requests (PRs).
• Asynchronous Review and SME Engagement: Graphite enables asynchronous PR reviews, allowing different domain specialists to review different PRs concurrently. This permits tapping subject matter experts (SMEs) across the organization for specific, tightly decomposed PRs.
• Informed by RPTD: The initial RPTD, with its detailed mapping of layers, architecture, persistence, and types, directly informs the decomposition strategy.

Diagnosing Challenges

To assess the effectiveness of these processes, Razgaitis suggests diagnostic questions for each area:

• Research Legibility: When staffing new research initiatives, is it clear to product people, software engineers, and AI engineers where to focus their efforts and what tasks to undertake? Ambiguity indicates a need to revisit this process.
• Code Repository: When integrating prototypes, is it obvious where new code should reside? Are there templates and clear patterns to follow? If the team is fighting old abstractions or experiencing headaches due to repository limitations, the codebase might be outgrown.
• Decomposition: Can timelines and delivery dates for moving research consistently be estimated? Is it clear which SMEs should be involved in review and productionizing? Inconsistencies or lack of clarity often point to issues in research coordination (RPTD) or the underlying codebase. The ultimate goal is to foster an environment where research can rapidly evolve into impactful, customer-facing features.

• Tags:
• Microservices
• Feature
• MLOps
• Production ML
• System Design
• FastAPI
• Research Engineering
• Software Engineering Best Practices
• Technical Documentation
• Monorepo
• Code Decomposition
• PR Review
• Higharc
• AI/ML
• Research to Production