AI Assistance for Software Teams: The State of Play • Birgitta Böckeler • GOTO 2024

The AI Tooling Landscape

The ecosystem of AI tools for software delivery can be categorized into three main archetypes:

1. Coding Assistants: Tools integrated into the IDE, like GitHub Copilot. This is currently the most mature and widely adopted category.
2. Delivery Toolchain Extensions: AI features being added to existing tools like Jira, Confluence, Figma, and CI/CD platforms. Adoption is slower as it depends on vendors rolling out meaningful features.
3. Other Specialized Tools: Tools for specific tasks beyond coding, such as requirements analysis or architectural design. This area is still nascent.

Due to its maturity, the primary focus is on coding assistants.

Deep Dive into Coding Assistants

Modern coding assistants are more than just a connection to a large language model (LLM). Their value lies in the sophisticated tooling built around the model.

Core Functionality: The Swiss Army Knife

Assistants provide two main features that work in tandem:

• Inline Suggestions: As you type code or comments, the assistant offers completions, filling in details and boilerplate. This is most effective when you have a clear plan and need to execute it.
• Integrated Chat: This feature transforms the assistant into a versatile “Swiss Army knife.” Unlike a separate tool like ChatGPT, the integrated chat has access to the codebase. It can be used for a wide range of ad-hoc tasks for which a dedicated tool would never be built, such as:
  • Explaining unfamiliar code.
  • Generating SQL queries or cron expressions.
  • Creating test data and stubs.
  • Scaffolding new components.
  • Building small, throwaway tools (e.g., a browser extension for a specific bug, a quick data visualization script).
  • Finding hard-to-google features in a technology.

The Power of Context Orchestration

The most significant differentiator between assistants is not the model but how they orchestrate context. This is essentially an implementation of retrieval augmented generation (RAG) tailored for software development. The tool composes a detailed prompt to send to the LLM by gathering relevant information from various sources:

• Current Codebase: Tools index the local workspace to answer questions like, “Where is this API being called?” They can identify relevant files to provide context for a query.
• All Company Codebases: Enterprise-level tools (e.g., Copilot Enterprise, Codium, Sourcegraph Codi) can index all of an organization’s repositories, making suggestions more specific to internal libraries and conventions.
• External Context Providers: An emerging ecosystem allows assistants to connect to other knowledge sources. By using commands like @azure, @atlassian, or @jira, developers can pull in context from cloud infrastructure, wiki pages, or the specific ticket they are working on.
• Custom Instructions: Some tools allow teams to define rules (e.g., “never use unwrap in Rust,” “always use functional React components”) that are included with every prompt to enforce coding standards.

This rich context orchestration is why the common question of fine-tuning a model on a private codebase is often misguided. The vast majority of customization and value comes from RAG, while effective fine-tuning is extremely expensive, requires massive amounts of high-quality code, and provides only a marginal lift.

The Next Frontiers: Promising but Not Production-Ready

Several areas show immense promise for AI assistance but do not yet work reliably.

• Testing: While AI is great for brainstorming edge cases and generating test data, fully automated test generation is risky. For legacy codebases that were not designed to be testable, it is nearly impossible, as the AI struggles with complex setup and dependencies. We as a profession need to develop new practices that leverage AI safely, perhaps by writing the initial test cases via TDD and then using AI to augment the suite with edge cases.
• Solving Larger Problems (Agents): The concept of “developer agents” that can take a ticket and implement it is a hot topic. However, current implementations are experimental and often fail. While a general-purpose problem-solving agent seems unlikely in the near future, there is potential for specialized agents focused on narrow, repetitive tasks like migrating tests between frameworks (e.g., Enzyme to React Testing Library).
• Code Review & Refactoring: Models are currently much better at adding new code than changing existing code. A CodeScene paper found that LLMs provided functionally correct refactorings in only 37% of cases, often introducing subtle but dangerous bugs. This weakness may contribute to faster codebase growth, as it becomes easier to add new code rather than refactor existing logic—a trend highlighted in a GitClear study.
• Documentation & Knowledge Discovery: Instead of just generating documentation that can become stale, a more powerful application is unlocking “code as knowledge.” By parsing code into an Abstract Syntax Tree (AST) and loading it into a knowledge graph, we can allow non-technical team members to ask natural language questions about the system’s current behavior. This can significantly accelerate reverse-engineering legacy systems and improve staff mobility.
• Debugging: Some IDEs like Cursor are experimenting with “chain of thought” debugging, where the AI reasons through an error, inspects relevant files, and hypothesizes solutions. While still experimental, this points to a powerful future use case.

A Guide to the Tooling Ecosystem

• The Leader (GitHub Copilot): The most widely adopted and solid product.
• The Challengers (Amazon Q, Gemini Code Assist): Currently feel like they are rushing to catch up.
• The Original (Tabnine): A mature product that predates Copilot.
• The Developer Favorite (Cursor): An IDE known for its excellent UX and innovative, experimental features.
• The IDE Expert (JetBrains AI): Focused on deeply integrating AI with its already powerful IDE features.
• The Open Source Leader (Continue): A highly configurable plugin for testing various models, including local ones.
• The Librarian (Unblocked): Focuses on creating a unified knowledge graph from code, Slack, Jira, and Confluence.
• The Confidence Booster (Codo): Specializes in improving code quality through better testing.

Mental Model and Final Takeaways

Think of an AI coding assistant as a teammate who is eager to help, very well-read, but inexperienced. They have high intelligence but low wisdom. They can be stubborn, won’t admit when they don’t know something, and will always try to please you.

1. Understand the Basics: Differentiate between the model and the tool. The tool’s ability to manage context is where the real power lies.
2. Embrace Imperfection: AI is non-deterministic. Don’t dismiss it for being imperfect; instead, adapt your workflows to leverage its strengths, like the versatility of a Swiss Army knife.
3. Don’t Overthink Tool Choice: It’s easy to switch between assistants. Experiment and find what works for you.
4. Reflect on Practices: These tools are here to stay. As a profession, we must reflect on our practices in areas like testing, refactoring, and design to use them effectively and safely.

• Tags:
• Code Generation
• Developer Tools
• RAG
• Feature
• GitHub Copilot
• AI Assistant
• Software Delivery
• Coding Assistant
• Refactoring