Engineering metrics

Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish

Better AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish

Nicholas Arcolano of Jellyfish uses data from about 50,000 engineers across 447 companies to test what really drives AI coding output. A natural experiment forcing users onto Grok barely moved merged PRs. The analysis breaks throughput into tool, skill use, token consumption, company and team, and individual engineer, finding the individual and organization matter far more than the model.

The Software Factory: From Bug Report to Production Code — Davis Palmie, Factory

The Software Factory: From Bug Report to Production Code — Davis Palmie, Factory

Davis Palmie of Factory explains the software factory, a system of agents that turns bug reports and user feedback into production code. He covers leadership concerns about cost, model choice, engineering roles, and org structure, argues that coding was never the bottleneck, and lays out principles for a factory: model agnosticism, sovereign deployment, and SDLC-wide integration. He also describes eight pillars of agent readiness, starting with narrow use cases like incident triage, and closes with why outcome metrics should replace token counts.

No, That's Not a Software Factory — Ryan Cooke, WorkOS

No, That's Not a Software Factory — Ryan Cooke, WorkOS

Ryan Cooke of WorkOS argues that software factories should be judged by customer outcomes rather than PR counts or AI-generated code volume. He describes how WorkOS embedded its product engineering process into its factory through TARS, Horizon, a PM agent that drafts Hilltop docs, and an internal MCP gateway connected to Snowflake, along with bug triage, owned sandboxes, a memory layer and outcome-focused metrics.

How to measure AI developer productivity in 2025 | Nicole Forsgren

How to measure AI developer productivity in 2025 | Nicole Forsgren

Nicole Forsgren, creator of the DORA and SPACE frameworks, explains why traditional productivity metrics are failing in the age of AI. She details how AI impacts developer flow state, shifts the focus from writing to reviewing code, and provides a new framework for measuring and improving Developer Experience (DevEx) by aligning engineering efforts with strategic business goals.