Multiplayer agentic engineering — Arjun Singh, Superconductor

Arjun Singh, co-founder of Superconductor, presents “Multiplayer Agentic Engineering,” a framework for enabling entire teams and AI agents to work together productively. Drawing from Superconductor’s aggressive integration of agents into their decade-long software development experience (which included co-founding GradeScope), Singh outlines six key lessons to overcome bottlenecks and friction points in agentic workflows, ensuring human-agent collaboration remains effective.

Lesson 1: Be Model and Harness Agnostic

• Rationale: The optimal AI model and its operating environment (harness) are constantly evolving. New models emerge, existing ones are modified or even retired.
• Benefits:
  • Avoids workflow disruption when changes occur.
  • Allows leveraging open-weight models (e.g., GLM 5.2) which are often good and more cost-effective.
  • Aligns incentives: Users aim to deliver value, while token providers seek to sell more tokens. Agnosticism empowers users to control token consumption, paying only for what’s necessary.
• Impact: Superconductor emphasizes that model agnosticism enables seamless switching between different models without significant workflow disruption, maintaining control over performance and cost.

Lesson 2: Turn Every Human Interface into an Agent and Human Interface

• Problem: Traditional agent interaction is often confined to a single laptop or, at best, a specific platform like Slack, limiting collaboration and context sharing.
• Solution: Design systems where the exact same agent session is accessible from all relevant interfaces simultaneously (e.g., Slack, desktop/mobile applications, GitHub).
• Workflow Example: A task can be initiated in Slack, continued in a specialized desktop application, and finalized within GitHub, with the agent retaining full context throughout. This prevents the agent from “forgetting” prior interactions or requiring users to manually transfer context.

Lesson 3: Make Agent Work Visible and Collaborative Across the Team

• Enhancing Collaboration: Beyond just Slack, comprehensive visibility is crucial. Superconductor’s app view displays all interactions with a task, showing who’s involved, notified, and has seen the progress.
• Benefits:
  • Transparency for Non-Technical Stakeholders: Critical when non-technical team members (e.g., customer support) initiate work, allowing them to easily verify if an engineer has vetted the agent’s output.
  • Efficient Review: Reviewers can directly query the agent within the existing thread for clarifications (e.g., “Hey, why did you do it this way?”) without waiting for the original author, as the answer is likely within the agent’s context.
  • Ubiquitous Artifacts: Regardless of where work started or ended, the agent can produce and display artifacts (screenshots, videos) accessible from all interfaces, eliminating the need to search for specific files or images across different platforms.

Lesson 4: Turn Every External Signal into Code Your Team Can Quickly Evaluate

• Challenge: Valuable external signals (Slack conversations, customer meetings, sales calls, bug reports, emails, feature requests) are fragmented across various systems, requiring significant human coordination to convert into actionable code.
• Superconductor’s Approach: Develop automated mechanisms to ingest these signals, prioritize them, and enable agents to act upon them.
• Meeting Bot Example:
  • Functionality: A bot can be invited to Google Meet, Zoom, or Teams to listen to conversations.
  • Intelligent Processing: It links to existing work if relevant, preventing duplication. It identifies emergent ideas from discussions.
  • Automated Action: In one instance, a participant mentioned the need for coding agents to have clear acceptance criteria. The bot independently identified this, created a new ticket, began working on it, and subsequently modified the product’s ticket form to include two new acceptance criteria fields.
  • Rapid Prototyping: The bot provided a screenshot of its modification upon request. While not necessarily shippable as-is, this demonstrates how ideas are concretized, prototyped, and made ready for evaluation.
• Outcome: This capability transforms customer calls, onboarding sessions, and team meetings into sources of dozens of prototyped ideas and often several genuinely shippable pull requests, drastically reducing the manual effort of translating requests across systems.

Lesson 5: Set up Workflow, Codebase, Project in an Isolated Cloud Environment

• Central Premise: Running agents in an isolated cloud environment, rather than on individual developer machines, is paramount for unlocking advanced workflows and ensuring security.
• Key Reasons:
  1. Eliminate “Lid Anxiety”: Developers can close their laptops, knowing that ongoing agent tasks continue unhindered in the cloud. This significantly improves work-life balance and flexibility.
  2. Least Privilege Security: This is deemed the most critical reason. Developers’ laptops often contain sensitive credentials or data not intended for LLM access. Highly autonomous and resourceful agents, eager to comply with instructions (e.g., “wipe this agent database”), could potentially find production credentials and inadvertently cause significant damage. A dedicated, isolated cloud environment prevents such unauthorized access and misuse.
  3. Prevent Data Exfiltration: A configurable network sandbox restricts an agent’s network access to predefined locations, preventing unauthorized exfiltration of sensitive code, secrets, or data. Any attempt to access a new, unapproved endpoint triggers a prompt for explicit permission.
  4. Empower Non-Technical Team Members: This isolation enables non-technical personnel (e.g., customer support, growth teams) to trigger real development work without requiring local development environments. They can simply make requests (e.g., “Fix this” in Slack), and agents can execute the task, with the output reviewed by engineers for merging. This bypasses traditional, slower triage processes.
• Implementation: While traditionally challenging, modern agents (like Cloud Coder or CodeX) and Superconductor’s own environment setup assistant can help configure projects to run effectively within these sandboxed cloud environments.

Lesson 6: Benchmark Agents on Your Codebase

• Methodology: Instead of relying solely on public benchmarks (e.g., SWE-bench, Terminal Bench), Superconductor evaluates agents against its own codebase. They select existing pull requests (human-created, agent-created, or hybrid) that exemplify quality engineering work.
• Evaluation Criteria: Agents are benchmarked based on quality, cost, and time taken to complete these real-world tasks.
• Why Custom Benchmarking is Essential: Public benchmarks may not reflect the specific nature of a team’s tasks or technology stack (e.g., SWE-bench is Python-centric, while Superconductor uses Ruby on Rails). Results can vary drastically.
• Superconductor’s Findings:
  • Anthropic agents consistently showed better quality but were slower and significantly more expensive on their codebase.
  • Codex agents were fast, good quality, and cheaper.
  • Open-source models improved over time but were generally slower.
• Impact: These data-driven insights led Superconductor to switch their default agent to Codex (and briefly to Fiable when available), demonstrating how agnosticism and internal benchmarking allow for rapid adaptation to new models without workflow disruption. This process also alleviates “model anxiety” by providing concrete data on what works best for a specific context.
• Future Direction: Superconductor aims to automate the routing of tasks to the most suitable model based on these internal benchmarks, optimizing for specific project requirements.

Results and Recommendations

• Agent-Generated Code: Nearly 100% of Superconductor’s pull requests are heavily agent-generated, with all changes undergoing human review for quality, reliability, and security.
• Usage Metrics (One Month):
  • Approximately 1.5 billion tokens processed.
  • 3,300 Claude Code runs, equivalent to ~$10,000 in token value.
  • Codex ran four times as many sessions for a lower overall cost.
• Recommendations for Teams:
  1. Sandbox Your Codebase: Get your codebase and agents working within an isolated cloud sandbox to unlock new workflows.
  2. Integrate Agents into Human Interfaces: Embed agents into all relevant human interfaces to foster seamless team and agent collaboration, reducing friction and context switching.
  3. Benchmark and Be Model Agnostic: Establish a method for benchmarking agents against your own codebase and maintain model agnosticism to consistently operate at the optimal frontier of cost, speed, and quality.

• Tags:
• AI Agents
• Developer Productivity
• Cost Optimization
• LLM Workflows
• Software Development Lifecycle
• Team Collaboration
• Agentic Engineering
• Cloud Development
• Performance Benchmarking
• Security Best Practices