CI/CD Breaks at AI Speed: Tangle, Graphite Stacks, Pro-Model PR Review — Mikhail Parakhin, Shopify
Shopify CTO Mikhail Parakhin details their aggressive internal AI adoption, revealing the internal stack—Tangle, Tangent, and SimGym—that powers reproducible ML, auto-research, and customer simulation. He also discusses how AI shifts the software bottleneck to PR review and CI/CD and explains their use of Liquid AI for low-latency, non-transformer workloads.
The New Bottleneck: From Code Generation to PR Review
The widespread adoption of AI for code generation has fundamentally shifted the bottleneck in the software development lifecycle. While an individual AI model might write code with fewer bugs than an average human, the sheer volume of code produced leads to a net increase in bugs making it to production. This creates a new, critical challenge: the pull request (PR) review process and the stability of the continuous integration and deployment (CI/CD) pipeline.
The real problem is no longer the time spent writing code, but the increased probability of test failures due to the volume of PRs. This can lengthen deployment cycles as teams work to identify and revert problematic changes. Parakhin argues for a counter-intuitive solution: spending more time and budget on review by using large, powerful models (e.g., GPT-4 level or higher) in an automated, multi-agent critique loop. While this increases latency at the PR stage—sometimes taking an hour or more—it yields significant downstream time savings by preventing bugs and stabilizing deployments. Shopify has built its own internal tools for this, as existing solutions often don’t support the required large models or the necessary multi-turn “debate” process between agents.
This flood of AI-generated code is straining traditional Git-based workflows, suggesting that the entire paradigm may need to be re-evaluated for an “agentic world.”
Shopify’s Internal AI & ML Stack
Shopify has developed a powerful, compounding suite of internal tools to manage and accelerate its ML and data workflows.
Tangle: Reproducible ML and Data Workflows
Tangle is Shopify’s third-generation system for managing data processing and ML experiments. It is designed for collaborative, iterative work, addressing the common pains of “digital archaeology” where developers struggle to reproduce past results.
Key features of Tangle include:
- Reproducibility: Ensures that rerunning a workflow produces the exact same results, eliminating ambiguity.
- Collaboration and Composability: Allows teams to easily clone, share, and modify existing pipelines. A user can change one small component in a large workflow without needing to understand the entire system.
- Efficiency through Content-Based Caching: Tangle uses content hashes to avoid re-running computations. If a process’s output hasn’t changed, it’s not re-run. This creates a powerful network effect where experiments from different teams can automatically benefit from each other’s pre-computed steps, dramatically saving time and resources.
- Production-Ready by Design: Unlike workflows developed in notebooks, Tangle experiments are production-ready from the start, streamlining the path from development to deployment.
Tangent: Democratizing Experimentation with Auto-Research
Built on top of Tangle, Tangent is an “auto-research” agent that automates the process of experimentation and optimization. It analyzes a Tangle workflow, identifies potential changes, and iteratively runs experiments to maximize a target metric or loss function.
This approach has been transformative, moving beyond simple hyperparameter tuning to modifying code, swapping components, and discovering novel optimizations. It has been successfully applied across Shopify to:
- Increase search throughput from 800 QPS to 4200 QPS on the same hardware.
- Optimize prompt compression (“gisting”) techniques.
- Reduce data storage costs by identifying and eliminating redundant datasets.
Crucially, Tangent democratizes ML experimentation. Product managers and other non-ML engineers are among its highest users, as it allows them to leverage their domain knowledge to guide the optimization process without needing deep expertise in the underlying algorithms.
SimGym: Customer Simulation as a Competitive Moat
SimGym is a sophisticated customer simulation environment that provides a significant competitive advantage. It addresses the core challenge of simulation: without grounding in real-world data, agents will simply do what they are prompted to do.
Shopify’s advantage lies in its decades of historical merchant and buyer data. SimGym uses this data to train and calibrate agents to accurately replicate real customer behavior. The system achieved a high correlation (0.7) with real-world “add to cart” events in A/B tests.
Key aspects of SimGym include:
- Data as a Moat: The system’s effectiveness is difficult to replicate without access to a comparable longitudinal dataset of e-commerce interactions.
- Modeling Trajectories and Counterfactuals: Using techniques like Hierarchical State-Time Units (HSTU), SimGym models entire user and merchant journeys. This allows Shopify to run counterfactuals—simulating what would have happened if an intervention (like offering a discount) occurred at a different point in time—to optimize outcomes for both merchants and buyers.
- Complex Infrastructure: SimGym runs agents in a real, simulated browser environment to capture visual and interactional friction, requiring a complex, expensive infrastructure of multi-modal models and optimized GPU usage.
Liquid AI: A Bet on Non-Transformer Architectures
Shopify is a significant user of Liquid AI, a non-transformer architecture based on state-space models (SSMs). While transformers are dominant, Liquid AI’s architecture offers compelling advantages for specific, high-value use cases.
- Low Latency and Long Context: Liquid models are sub-quadratic in context length, making them highly efficient. Shopify uses a small, 300M parameter Liquid model to power its search query understanding, achieving an end-to-end latency of under 30 milliseconds.
- Ideal for Distillation: For large-scale, offline tasks like product catalog categorization and analysis for Sidekick Pulse, Shopify distills knowledge from the largest available frontier models into more compact, efficient Liquid models (e.g., 7-8B parameters). This provides a cost-effective way to deploy specialized intelligence at scale.
While not a frontier model competitor in general reasoning, Liquid AI has proven to be a superior architecture for these specific niches, and its share of Shopify’s internal workloads is growing.