You're Shipping 10x More Bugs and Don't Know It

The explosion in code generation, driven by powerful coding agents, is dramatically outpacing our ability to verify and test that code. This creates a critical bottleneck, shifting the challenge from code creation to code validation. While developers are producing an order of magnitude more code, the tools and processes to ensure it works as intended have not kept pace.

The Widening Gap Between Code Generation and Verification

Two primary approaches to Quality Assurance (QA) are struggling to keep up with AI-assisted development:

• Code-Based Testing: Traditional methods like unit, integration, and end-to-end tests involve writing more code to test auto-generated code. This is an incremental improvement at best, often leading to fragile, slow, and expensive CI pipelines that hinder developer experience.
• Manual Testing: The burden of QA has largely shifted to developers, who are now expected to test their own work. This manual process is a significant bottleneck, as a developer creating a pull request (PR) can’t realistically check every part of an application their changes might affect. For example, a developer might check the three most important places a modified button appears, but not the other 47.

Even if coding agents become near-perfect, the need for verification will not disappear. Human requirements are rarely “one-shot” and perfect. Product and engineering in the future will resemble a “writer’s room,” involving intense collaboration, rapid prototyping, and collective “taste-making” to align on product direction and technical design. This collaborative and iterative process inherently requires a robust verification loop.

Ito AI’s Approach: Runtime Execution Testing on Every PR

Ito AI focuses on providing “automated QA on every PR” by targeting the point of maximum leverage: the developer authoring the pull request. The goal is to act as a force multiplier, automatically checking all the affected user flows that a developer can’t cover manually.

The core focus is on runtime execution testing, which is compared to test-driving a car rather than just “looking at the engine really hard” (the equivalent of static analysis). While static analysis is valuable and complemented by Ito AI, it misses the behavioral side of how users actually interact with an application.

How It Works: Infrastructure, Trust, and Security

Ito AI is built on top of open-source frameworks like Playwright and Appium but adds a significant layer of value through its agentic pipelines and infrastructure.

• Ephemeral “Single-Use” Code: We are in the “plastic age of code,” where code can be generated, run once for a specific purpose (like testing a PR), and then discarded. Ito AI runs a high volume of these ephemeral tests, avoiding the need for developers to maintain a massive, slow, and fragile regression test suite in their CI.
• Trust and Visibility: To build trust with developers, the platform provides extensive visibility into the testing process. This includes real-time summaries of the agent’s actions, video recordings of UI interactions, logs, and other artifacts. Developers can also interact with the agent by tagging it in a PR comment to request specific additional tests.
• Security by Design: Security is paramount. Ito AI operates on the principle of an “iron cage.” It temporarily pulls down the source code into a completely isolated and sandboxed environment for each customer. Within this sandbox, the agent is fully empowered to interact with the database (using mock/seed data), mock services, and spin up servers. After the run, the environment and the code are completely deleted. Customer environments are never run on shared instances.

The Evolving Role of QA and the Enterprise Bottleneck

The QA profession is evolving from a separate, siloed team to a more integrated, strategic function. In the age of AI, QA’s role will shift from chasing individual bugs to providing a global, “taste-making” perspective on quality. They will help define organizational standards for handling errors, prioritizing fixes, and ensuring a consistent user experience.

While solo developers feel the magic of AI coding, large organizations are struggling to see the same benefits. Their complex release processes, higher verification standards, and the significant cost of bugs (tens of thousands of dollars per bug when accounting for meetings, discovery, and process) mean they cannot simply “push out garbage and fix it later.” The primary bottleneck for these organizations in adopting AI is the lack of a corresponding verification layer to trust the output of LLM-generated code. The demand for verification is so high that the market for manual testing services is currently exploding. Ito AI aims to automate this layer, unblocking enterprises and allowing them to truly leverage the power of AI in their development cycles.

• Tags:
• AI Agents
• Code Generation
• Feature
• CI/CD
• QA
• Software Testing
• DevTools
• Pull Request