Prototyping as Leadership: How a CTO Ships with AI Agents — Hursh Agrawal, The Browser Company

Hursh Agrawal, CTO and co-founder of The Browser Company, presents a compelling argument for why “building” has become an integral part of modern leadership, particularly for managers in the AI/ML space. Despite a demanding schedule filled with 15+ recurring meetings, managing seven direct reports, and personal commitments like a toddler at home, Agrawal consistently ships between two and ten pull requests per week—a feat he attributes directly to the emergence of autonomous AI agents. He emphasizes that this newfound ability to build isn’t about working longer hours but about intelligently leveraging AI to transform the traditionally meeting-heavy “manager schedule” into productive “building time.”

Why Building is Now Essential for Leaders

Agrawal articulates two primary reasons why hands-on building is no longer a luxury but a necessity for leaders:

1. Calibration with Rapidly Evolving Frontier Models: The core technologies driving many products today are frontier models that undergo significant changes every three months. This rapid evolution generates considerable “noise” and conflicting opinions about new models’ capabilities. Agrawal asserts that it’s “impossible to tell what a new model is good for unless you have your hands in it and you’re using it all day long.” Direct engagement allows leaders to understand the contours of new model families, inform engineering direction, assess strategic impact, and anticipate future developments (e.g., open-source models catching up). This visceral understanding forms crucial intuition.

2. Effective Communication and Vision Setting: Even with strong intuition, conveying complex possibilities to engineers who haven’t experienced the models firsthand can be challenging. Building prototypes with new models provides a tangible, playable artifact that bypasses lengthy debates. “Hey, I built this prototype with this new model family. It works in the product. Here’s an actual prototype you can play with.” This approach is “so much faster and more efficient than trying to convince people every 3 months.”

Leaders are uniquely positioned to leverage AI agents for building due to their inherent advantages:

• Business Context: Leaders possess the broadest context regarding business goals, strategy, trade-offs, and imperatives. This context makes their “steering” (i.e., prompts given to agents) significantly more impactful per token than an individual contributor’s.
• Delegation Skills: The core leadership skill of delegation—setting goals, providing context, checking in, and even coaching—translates effectively to managing AI agents.
• Judgment in an Execution-Focused Landscape: While current AI models excel at execution, they often lack judgment. Leaders provide this critical judgment, guiding agents through complex problems or suggesting alternative approaches when an agent hits a perceived blocker.

Agrawal also references Julie Zhuo’s insights on what leaders should build:

• Internal Tools: Enhancing internal efficiency and quality of life.
• Codebase “Gardening”: Improvements and maintenance.
• Celebration Artifacts: Recognizing team members.
• Vision Pieces: Most importantly, exploring new model capabilities, understanding their business fit, and demonstrating product possibilities through prototypes.
• Crucially, avoid critical path work. Leaders’ schedules are unpredictable, and critical path dependencies should not rest on them.

The Overnight Loop: Mechanics of Building

Agrawal outlines a practical daily routine centered around a powerful “one overnight loop” mechanism:

• Morning Block (1 hour): Reviewing and refining the code generated by agents overnight.
• Interspersed Steering Blocks: Short periods throughout the day to guide agents or set up smaller tasks.
• The Critical 5 PM Block: Dedicated to setting up the overnight run.

The overnight loop process involves:

1. Context Gathering (5 PM): The “mindset shift” is to provide the frontier model with all the context it needs to make decisions, akin to how a human would. Agrawal suggests using a “co-work agent” (connected to Slack, Jira, Notion, code repositories) to spend 20 minutes researching and synthesizing a comprehensive prompt, including business goals, trade-offs, past attempts, and rationale.
2. Setup the Run: This detailed prompt is fed to a coding agent (e.g., Claude Code, Cursor, CodeX).
3. Autonomous Execution (Overnight): The coding agent works for 4-8 hours.
4. Morning Report & Shipping: In the morning, the leader reviews the agent’s output, including a detailed report on decisions and trade-offs, and prepares to ship.

Agrawal provides three key examples of tasks effectively handled by this overnight loop:

1. Building Features

The agent is prompted to “implement this whole feature” with the gathered business context. Key instructions for the agent include:

• Test-Driven Development: “Write the tests first so they capture what you do” to ensure higher quality tests.
• End-to-End Verification: Instruct the agent to “test the end-to-end flow with computer use,” ensuring the feature works and aligns with business goals.
• Reviewer-Friendly PRs: Generate “reviewer friendly PRs with clear descriptions,” manage CI, and ensure “CI green.”
• AI Code Review Integration: Integrate internal or external AI code reviewers, instructing the agent to “run this AI code review skill in a clean sub agent, and then fix those things… resolve the threads, run this autonomously.”
• Morning Report: Provide a comprehensive report on trade-offs and performance. This process can convert “weeks of work” into a single overnight run.

2. Hill Climbing AI Features

To optimize initial LLM calls or agent-driven features (which are rarely perfect), Agrawal implements a feedback-driven hill-climbing process:

• Feedback Collection: Add a simple feedback button to prototypes. Collect JSON dumps of runs, system prompts, inputs, and feedback (even 4-10 samples are useful, but 20-30 are better).
• Overnight Optimization (5 PM): The agent is given the feedback JSONs and instructed to:
  • “Turn this into an eval set” using local tools like SQLite.
  • Interactively design “evals or scoring functions to optimize for this.”
  • “Build a harness that runs this call against the evals and hill climbs until the score goes up.”
  • Instruct to “not overfit” and “keep it general” with small datasets.
  • Provide a full report in the morning.
• Skill Reusability: A valuable tip is to “save this flow as a generic skill” for continuous improvement. This approach significantly enhances feature quality, as demonstrated by internal usage.

3. Training Custom Classifiers

A more recent and powerful capability is training custom ML models overnight:

• Use Case Example: Training a BERT PII classifier after existing frontier models proved insufficient due to cost, latency, or precision/recall.
• Overnight Training (5 PM): Provide business context and training data. Instruct the agent to:
  • “Clean up the training data… bolster it with synthetic data.”
  • Use provided API keys (OpenAI, Anthropic).
  • “Use an ensemble of frontier models to push the quality.” Let the agent decide the best model class, and even “train two separate ones.”
  • Manage cloud resources: “Give it AWS access… provision sandbox (not prod!), pick the right GPU and EC2 cluster, train it, test against eval examples, deprovision it.”
  • Return “the trained models” and a “report in the morning,” including guidance on “how I would host this with my codebase on inference.” This results in production-ready ML models trained autonomously overnight.

Scaffolding and Hygiene for Success

Agrawal emphasizes that pushing the scope of AI agents requires robust organizational scaffolding and strict personal hygiene:

• Organizational Scaffolding:
  • AI Code Reviewers: Essential for maintaining code quality (internal or external).
  • Agent Configuration Hygiene: Clear documentation or configuration for agents (e.g., agents.md).
  • Trustworthy CI: Continuous Integration systems that developers can rely on.
  • Sophisticated Feature Flags: For controlled rollout and experimentation.
  • Prototype Branch: A dedicated environment for testing new features with employees without impacting production.
• Personal Code Hygiene (from experience):
  • Test Everything: Leaders must still personally test the agent’s output in the morning, even with automated computer use.
  • Small, Readable PRs: Model good development practices for the team.
  • Read Your Code Before Review: “Never add reviewers to code you have not read.” Failing to do so can lead to embarrassing situations and reflects poorly on leadership. Agrawal candidly admits his own code, generated via agents, has caused incidents (SEVs), highlighting that agents are not yet perfect and personal oversight is crucial.

In conclusion, Agrawal urges leaders to embrace building as a core part of their role. By dedicating even an hour or two a day and leveraging AI agents for overnight execution, leaders can stay deeply calibrated with technological advancements, effectively communicate vision through working prototypes, and model the future of high-leverage development within their organizations.

• Tags:
• AI Agents
• Software Development
• Prompt Engineering
• Feature
• MLOps
• AI Code Review
• LLM Development
• Feature Flags
• Manager Productivity
• Prototype Development
• Leadership in AI