What Does an AI Engineer Do Now? - Aparna Dhinakaran, Arize AI
Aparna Dhinakaran of Arize AI argues that the AI engineer's job is shifting from writing features to building the loop that builds them: evals that review non-deterministic work, telemetry that agents consume, and always-on agents that act on it. She uses Arize's Signal agent as a case study, covering triggers, sandboxes, durable state, model and harness independence, and evaluating the evaluators.
Aparna Dhinakaran starts from a question the field keeps re-answering: what does an AI engineer actually do? Her answer is that the job keeps changing as the abstraction layer rises. Andrej Karpathy’s recent funnel makes the gap concrete. Billions of people have touched an LLM, around 20 million use it professionally, and a few thousand frontier users run swarms of always-on agents. The path from casual use to running a team or factory of agents is the same climb engineers have made for decades, only faster.
From writing features to building the loop
The central shift is from building a feature to building the loop that produces features. Previously the engineer wrote the feature, read the diff, and did code review. Now the engineer sets up agents that do the review, writes evals to assess non-deterministic output, and has agents read observability data to find patterns. The loop runs build, eval, ship, monitor, and improve. The AI engineer is responsible for the system that runs each of those steps, not only for performing them.
Signal: an always-on agent over traces
Arize built Signal to run this loop. It reads traces arriving at a configurable volume and cadence, identifies issues in the telemetry, and, when it has repo access, opens a fix or a PR. Signal has to keep memory: what it has already flagged, what looked suspicious but was not captured well enough, and what has since grown into a critical issue as more data arrived.
This work exposed a change in who consumes telemetry. Observability data was built for humans, then for humans assisted by agents. The goal now is to design telemetry so agents are the primary consumers, and that telemetry becomes the trigger that wakes agents up.
Triggers, runs, and sandboxes
An always-on agent can be triggered in several ways:
- A cadence, such as pulling the last set of traces every three or six hours
- Events, such as a Slack message reporting a user complaint
- Metric thresholds
Once triggered, each run assembles its context inside a sandbox: the relevant traces, third-party telemetry such as APM logs and metrics, repo access, and the keys needed to operate. The sandbox is the only path in and out, mediated by a single proxy, and everything that happens inside it is traced. Every output, including generated issues and PRs, is written to memory.
What persists and what gets torn down
After each run, findings persist. Issues the agent detected go into memory, so the next run does not start from zero and has a history to build on. Traces of every tool call and any PR opened are kept, and skills the agent learns during a run can be written back to the repo. The sandbox, credentials, and repo access are torn down.
Durable and model-agnostic design
Because state lives outside the sandbox, a failed sandbox or harness does not force a restart from scratch. A new sandbox and harness can resume from the step that failed. Signal is also agnostic to the model and harness. It runs on Claude Code, Cursor, and Codex, with the goal of supporting the major open-source harnesses as well. Teams can choose among models within a provider family. This flexibility doubles as an evaluation tool and a cost control: as cheaper models are swapped in, the question is whether the quality of PRs and issue detection holds.
Writing evals and evaluating the evaluators
This is where an AI engineer should spend much of their time. Simple evals, such as a dataset of prompts checked for regressions, are the starting point. The harder and more important work is aligning evals with human judgment. Practically, that means looking at traces alongside agents, identifying recurring error modes, and building golden-truth datasets. Before shipping a new evaluator, teams should confirm that the LLM judge agrees with human labels. Once validated, the same evaluator runs offline during testing and online over production telemetry. Arize provides a skill that helps build evaluators and align them with human judgment. The question teams ask most is ‘How do I trust the evaluator?’, and the answer is to evaluate the evaluators.
Tracing the factory itself
Tracing has to cover the coding agents that build the system, such as Claude Code, Cursor, and Codex, so the factory of agents is itself observable. Online evals run over production telemetry, and that data feeds back into the agents that read it and surface problems. The resulting system collects data, evaluates it, runs agents over the results to find problems and gaps, and improves the product.
Closing the gap between labs and businesses
There is a large gap between what happens in frontier labs and what happens in businesses. Part of it is that business data is far messier than the constrained, testable data inside a lab. AI engineers are the role meant to close that gap. They do it by tracing everything, including the agents they build; by running evals that are validated against human judgment; and by building systems in which agents improve products from the telemetry they collect, rather than shipping isolated features.