Compilers in the Age of LLMs — Yusuf Olokoba, Muna

The daily reality for an AI engineer involves a complex web of tools: OpenAI clients, Hugging Face models, multiple “playground” repositories, and agentic workflows that are essentially brittle chains of HTTP calls. While the industry buzzes about voice agents and multi-modal models, the foundational challenge for many engineering teams is simpler yet more pervasive: How can we use more models in more places without constantly rebuilding our infrastructure?

The current process for deploying a new open-source model is cumbersome, often requiring custom Docker files and new infrastructure provisioning. The ideal developer experience is far simpler: an OpenAI-style client that “just works,” capable of pointing to any model—whether it’s running locally or remotely, using Llama.cpp or TensorRT—with minimal code changes.

This talk presents a solution: a compiler that transforms plain Python code into a tiny, self-contained native binary that can run anywhere, from the cloud to Apple Silicon. This approach strategically uses LLMs within the compiler pipeline to achieve this goal.

The Vision: A Compiler for Hybrid AI Inference

A Python compiler was chosen as the long-term solution for AI deployment for two primary reasons:

1. Standardization and Simplicity: To create an extremely simple, standardized method for developers to execute any AI model (internal or open-source) with an experience that mirrors the ease of changing the model argument in an OpenAI API call. The compiler ingests Python inference code and outputs an artifact that is easily executable in any user environment.
2. Preparing for Hybrid Inference: The future of AI deployment is hybrid, where smaller, specialized models run locally on user devices or at the edge, working in tandem with larger, more powerful models in the cloud. This architecture requires a shift away from Python and Docker containers toward low-level, hardware-proximate, and highly responsive code.

How It Works: From Python to a Native Binary

Let’s walk through the process using a practical example: a Python function that runs Google’s Gemma-2B embedding model. The goal is to take this function, generate equivalent C++ and Rust code, compile it into a self-contained binary, and consume it using a familiar OpenAI client interface.

# Example Python function for an embedding model
def generate_embeddings(text: list[str]) -> list[list[float]]:
    # ... model loading and tokenization ...
    # ... model inference ...
    return embeddings

Step 1: Tracing the Python Function

The first step is to generate a graph representation—an Intermediate Representation (IR)—that describes every operation within the function.

• Initial Approach (PyTorch torch.fx): The team first experimented with torch.fx, which traces Python code by running it with “fake” inputs. This was abandoned for two reasons:
  • It is heavily focused on PyTorch code and struggles with other common libraries like NumPy or OpenCV.
  • Creating “fake” inputs is trivial for tensors but becomes incredibly difficult for complex data types like images or dictionaries.
• Second Approach (LLMs): Using LLMs to generate structured trace outputs was explored. While this was nearly 100% accurate, it was far too slow for a compiler pipeline.
• Final Solution (Custom Tracer): The team built a custom, “old-school” tracer. It analyzes the code’s Abstract Syntax Tree (AST) and uses internal heuristics to build a proprietary IR. For the embedding function, this IR consists of nodes representing inputs, function calls (e.g., to the tokenizer and the model), and return values.

Step 2: Type Propagation and Code Generation

The next challenge is translating the high-level, dynamically-typed Python IR into low-level, statically-typed languages like C++ or Rust.

• The Challenge (Dynamic vs. Static Types): A Python variable x can be an integer one moment and a string the next. In C++, a variable’s type is fixed upon declaration. The compiler must therefore deduce and enforce a static type for every variable.

• The Solution (Type Propagation): The compiler uses a technique called type propagation. It starts with known types from the function signature (e.g., the input text is a list[str]) and global constants. It then “flows” this type information through the graph of operations.

  • For example, if the code concatenates a known string prefix with an input string, the compiler can deduce the output must also be a string. It generates a C++ function operator.add(string, string) which returns a string. This process is repeated for every operation, allowing the compiler to assign a static C++ type to every intermediate variable.

• The Scalability Problem: This implies that every unique Python operation needs a corresponding manual implementation in C++. While this seems daunting, it’s made tractable for two reasons:

  1. The vast variety in source code comes from the permutation and combination of a relatively small set of elementary operations, not an infinite number of unique operations.
  2. LLMs are used to mass-produce the C++/Rust code for these elementary operations. Instead of manually rewriting every function, developers can use LLMs to generate the necessary translations from Python to native code, dramatically accelerating the process.

With a fully typed IR, the compiler can now generate correct C++ source code that maps directly to the original Python logic.

Step 3: Compiling to a Native Binary

The generated C++ source code is then compiled into a native, self-contained binary (e.g., a shared object .so file). Because virtually every platform in existence has a C or C++ compiler, this artifact is now universally portable and can run anywhere without dependencies like a Python interpreter or Docker.

Step 4: Invoking the Model with an OpenAI-Compatible Client

The final step is to use this compiled binary. Using JavaScript and Node.js as an example:

1. Load the Library: A Foreign Function Interface (FFI) is used to load the compiled native library from JavaScript. The FFI bindings declare the names and signatures of the functions available inside the binary.
2. Create the Client: A wrapper class is created to mimic the official OpenAI client.

   class Client {
     constructor() {
       this.embeddings = {
         create: async ({ model, input }) => {
           // 1. Resolve model name to the path of the compiled binary
           const libraryPath = resolveModelPath(model);
   
           // 2. Use FFI to load the library and invoke the function
           const output = invokeNativeFunction(libraryPath, input);
   
           // 3. Massage the output to match the OpenAI response schema
           return formatAsOpenAIResponse(output);
         }
       };
     }
   }
   

3. Execute: When a user calls client.embeddings.create, the client resolves the model name to the path of the compiled binary, loads it, executes the native code, and formats the result.

This completes the pipeline, delivering the original promise: an OpenAI-compatible client that can run any open-source model, compiled into a high-performance, portable binary, on any device.

• Tags:
• Code Generation
• LLM
• Feature
• Python
• C++
• Model Deployment
• Compiler
• Hybrid Inference
• FFI