Beyond the Prototype: Using AI to Write High-Quality Code - Josh Albrecht, Imbue

The primary challenge with current AI coding tools is the significant gap between the code they generate and the robust, production-ready software required for established codebases. The core issue is not just about generating code, but about building user trust in that code. The definition of high-quality software is the absence of defects, so the goal should be to leverage AI not just for generation, but for identifying and mitigating potential problems as early as possible in the development lifecycle.

Preventing Problems in AI-Generated Code

Proactively guiding an AI agent to produce better code is the first line of defense. This involves several key strategies integrated into the development workflow.

1. Learn and Plan First

Before writing code, agents should be encouraged to research existing solutions and formulate a clear plan. In Sculptor, this is facilitated by allowing users to change the agent’s system prompt to first enforce a planning phase without any code generation. This structured, multi-step workflow—plan first, then code, then run checks—can be enforced to guide the agent’s behavior more strongly than a simple instruction.

2. Treat Specifications and Docs as First-Class Citizens

AI agents lack the implicit context humans have from emails, Slack, or design meetings. Therefore, detailed specifications and documentation are crucial for them to understand a project’s requirements. A common obstacle to writing docs is the maintenance burden. Sculptor aims to lower this barrier by automatically detecting inconsistencies between the code and the documentation, highlighting conflicting specifications, and helping to keep them synchronized.

3. Enforce a Strict Style Guide

A consistent style guide is essential for team collaboration and becomes even more critical when working with AI agents. Sculptor helps enforce this by making suggestions that keep the agent on a reasonable path. For example, it can suggest making a class immutable to prevent race conditions, reflecting a style guide that favors a functional, immutable approach. The development of a style guide custom-tailored to AI agents can help them avoid their most common and egregious mistakes.

Detecting Problems in AI-Generated Code

No matter how good the prevention is, mistakes will happen. The next step is to have robust mechanisms to detect these errors before they reach production.

1. Run Linters and Automated Tooling

Tools like Ruff, mypy, or pylint are excellent for automatically detecting certain classes of errors. While fixing minor linting issues can be tedious for human developers, AI agents excel at this task. Sculptor integrates this by allowing the agent to detect and automatically fix these issues. It also distinguishes between pre-existing linting errors in a codebase and new ones introduced by the agent, ensuring the AI doesn’t degrade code quality.

2. Write and Run Comprehensive Tests

The advent of AI largely removes the traditional objections to writing tests, as agents can generate them easily. With AI-assisted coding, tests are more important than ever to ensure an agent doesn’t unexpectedly alter system behavior.

Key principles for writing effective tests for AI-generated code include:

1. Write Code in a Functional Style: Code with no side effects is significantly easier for an LLM to test and reason about. It’s critical to isolate side effects (e.g., database writes, API calls) from the core logic.
2. Use Happy and Unhappy Path Tests: A few “happy path” tests can confirm the code works as intended. The real value comes from “unhappy path” tests, where LLMs can generate hundreds of potential edge-case inputs to uncover bugs.
3. Consider Throwing Away Unit Tests: Since generating tests is now cheap, there’s less need to maintain an exhaustive suite of unit tests for behavior you don’t care deeply about. It can be better to refactor or regenerate tests as needed to avoid confusing the LLM with outdated test logic.
4. Focus on Integration Tests: These tests verify the system works from a user’s perspective (e.g., “when the user clicks the add button, the item appears in the cart”). AI agents are particularly effective at writing these tests from high-level, descriptive test plans.
5. Measure Test Coverage: Test coverage is a critical metric for building trust. A pull request with 100% test coverage that passes is far more trustworthy than one without.
6. Use Sandboxes: Running tests in secure, sandboxed environments without access to secrets is crucial for safety and preventing flaky tests.

3. Ask an LLM for Analysis

An LLM can act as an automated reviewer to check for a wide range of issues, such as:

• Violations of style guides or architectural patterns.
• Inconsistencies between the implementation and the specification.
• Inadequate test coverage.
• Logical errors or potential race conditions.

Sculptor is being designed to allow developers to extend these checks with their own custom best practices.

Fixing Identified Issues

Once a problem is clearly identified and stated, it is “half-solved.” AI coding agents are surprisingly effective at fixing well-defined problems. Simple strategies, like attempting a fix multiple times with different parameters in a safe, sandboxed environment, often succeed. If any attempt yields a valid solution, it can be used.

The future of AI in software development will see an ecosystem of integrated tools for debugging, logging, QA, design-to-code, and contextual search, making the entire development experience more seamless and efficient.

• Tags:
• AI Agents
• Code Generation
• Software Engineering
• LLM
• Testing
• Code Quality
• Sculptor
• Imbue