The End of the Junior Data Engineer?

Matthew Glickman, co-founder and CEO of Genesis Computing, explains that the impetus for his company came from a recurring pattern he observed while at Snowflake: enterprises would create impressive AI demos using models like GPT-4 but consistently fail to move them into production. This struggle highlights the “last 10%” problem in enterprise AI.

The “Last 10%” Problem: Reliability Over Accuracy

The final stage of deploying an AI project is deceptively difficult. The challenge isn’t necessarily accuracy but achieving consistent reliability and predictability. Unlike deterministic software, the probabilistic nature of Large Language Models (LLMs) means they can fail unexpectedly, even after successful demos.

This issue is particularly acute for “sexy” use cases like AI-powered dashboards for CEOs or customer-facing chatbots. The risk profile is highly asymmetric; a single failure, like an agent giving away free flights, can be a disaster. The real opportunity lies in internal, enterprise use cases where the user is a professional who can handle exceptions and values getting 60-70% of their time back. The goal is to build systems that can confidently assess their own output, escalating to a human only when necessary, rather than requiring human review of every result.

Why Data Engineers are the Ideal Persona

Genesis Computing focuses specifically on the data engineer persona for several key reasons:

• Willingness to Automate: Data engineers are often overwhelmed with a backlog of tasks and are eager to automate the repetitive, less glamorous parts of their job, such as fixing broken pipelines or performing data cleansing.
• Focus on High-Impact Work: They prefer to focus on business-impacting tasks, like building data feeds for a CMO’s campaign to reduce customer churn, rather than the low-level mechanics of data infrastructure.
• Enabling More AI: Glickman challenges the notion that a perfect data platform is a prerequisite for AI. Instead, he argues, “You should be using AI to get your data in good shape so you can do more AI.”

The tool is positioned not as a replacement but as a force multiplier that gives data engineers “superpowers,” akin to hiring a team of junior data engineers who can work 24/7.

How Genesis AI Agents Work

The workflow begins with a business requirement, much like assigning a task to a junior engineer.

1. Onboarding: The Genesis system, personified by an agent named Eve, is installed directly within the customer’s data environment (e.g., Snowflake, Databricks, AWS), giving it access to data sources and catalogs.
2. Blueprints: The system uses pre-built “blueprints,” which are like learned skills or playbooks that define how to use a company’s specific toolset (e.g., dbt, Spark) to accomplish a task.
3. Knowledge Acquisition: Over time, the system acquires institutional knowledge by observing code, documentation, and human interventions. If a human corrects the system on how “revenue” is defined, that knowledge is captured and reused. This knowledge becomes a secure, proprietary asset for the customer, as the system runs in their environment, not as a multi-tenant SaaS.

Killer Use Case: Legacy Migration and Rationalization

A powerful application for these AI agents is migrating and modernizing legacy data systems.

• Beyond Code Translation: Migrating systems built on technologies like SAP, Informatica, or even Perl and COBOL is not about a line-for-line code translation. The real goal is to understand the original business logic and re-implement it using modern best practices.
• Logic Rationalization: Agents can analyze entire codebases and discover that the same business logic (e.g., customer classification) has been implemented in multiple, inconsistent ways across different systems. The project then becomes one of “rationalization”—creating a single, correct implementation.
• Documentation-First Approach: The agent first reads the old code and documentation to produce a new, human-readable document outlining the business logic. This document is reviewed and approved by business users (like a CFO) before any new code is generated, ensuring the documentation and the final implementation are always in sync.

Under the Hood: The Need for Frontier Models

Genesis relies on top-tier “frontier” commercial models from providers like OpenAI, Anthropic, and Google.

• Continuous Improvement: For the specific, complex use case of data engineering, each new release of a frontier model brings a significant leap in capability.
• Model Personalities: Different models exhibit distinct behaviors. Claude models are often more “chatty” and explanatory, while newer GPT models can be less verbose and more cost-effective. Initial tests show Gemini to be highly “self-critical” and less tolerant of tools that don’t behave exactly as documented.
• Open-Weight Models: While customers are not yet demanding open-weight models for performance reasons, their existence puts downward pressure on the price and upward pressure on the performance of the leading commercial models.

Human-in-the-Loop and the Build vs. Buy Dilemma

The system is designed to be human-in-the-loop, but with a crucial distinction: the system escalates to the human when it lacks confidence or needs clarification. The goal is to save time, not create review tasks.

Glickman also addresses the “build vs. buy” disease common among engineering teams. He advises that in the AI era, companies should only build what is a core differentiator for their business. For everything else, it is more efficient to buy from experts who live and breathe that specific problem space, especially given the rapid pace of change in AI.

• Tags:
• AI Agents
• LLMs
• Feature
• Automation
• Data Engineering
• Legacy Systems
• Genesis Computing
• Matthew Glickman