Beyond Chatbots: How to build Agentic AI systems with Google Gemini // Philipp Schmid

The field of AI is at a turning point, moving from predictive models that find patterns to generative, agentic systems that take action. While early models focused on text completion, the journey has progressed through instruction tuning and conversational AI to the current frontier: AI agents. These systems operate over longer inference times, taking a user-defined goal and a set of tools to execute complex, multi-step tasks.

Defining AI Agents vs. Workflows

An AI agent is fundamentally a system that uses an LLM to decide the control flow of an application. The critical distinction from a simple workflow is this decision-making capability.

• Workflow: A predefined, static sequence of steps (e.g., user query -> LLM call -> aggregate -> final output). The path is fixed.
• Agent: A dynamic system where the LLM acts as a reasoning engine or “brain” to decide the next step. It might call a tool, ask the user for clarification, or generate a direct response.

An agent is composed of three core components:

1. Reasoning: The core LLM that powers the decision-making loop.
2. Tools: The “hands” of the agent, allowing it to interact with external systems like APIs to retrieve data or perform actions.
3. Context and Memory: Includes short-term memory (the current conversation) and long-term memory (information and preferences from prior interactions).

Core Agentic Patterns

Several common structures, or “agentic patterns,” serve as building blocks for creating robust agentic systems. These patterns are not rigid rules but composable guides for structuring agent logic.

1. Reflection Pattern

This pattern involves a loop of generation and critique. The LLM generates a response (e.g., a piece of code or text), and then a subsequent prompt asks it—or another process—to reflect on and critique that output. Based on the feedback, the LLM refines its response. This is highly effective for tasks like code generation, where an initial script can be written, executed, and then refined based on the error messages returned.

2. Tool Use Pattern

The most fundamental pattern, where the model is given access to a set of tools defined by a structured schema (like a JSON definition). The LLM decides whether to generate a text response or a structured JSON output that specifies which tool to call and with what arguments. The system then executes this function and feeds the result back to the model to inform the next step.

3. Orchestrator (Planner) Pattern

For more complex tasks, the orchestrator pattern is used. Here, a primary LLM first generates a step-by-step plan before any actions are executed. It breaks down a high-level goal into smaller, manageable tasks. Each task can then be delegated to a different, specialized agent or model. This allows for more structured problem-solving and the use of specialized models for specific sub-tasks.

4. Multi-Tool Use Pattern

This pattern facilitates a “handoff” between different specialized agents while maintaining a single, unified context. For example, a user request to “book a flight to Toronto and a hotel” might first be handled by a flight-booking agent. Once the flight is booked, the entire conversation context is seamlessly handed off to a hotel-booking agent, which already knows the arrival/departure dates and destination, creating a much smoother user experience.

The Unique Challenge of Evaluating Agents

Evaluating agents is significantly more complex than evaluating traditional LLMs.

• LLM Evaluation: Typically involves comparing a model’s output to a ground truth answer. It’s verifiable, cost-effective to calculate on a per-token basis, and often measured by capability (e.g., accuracy on a benchmark like MMLU).
• Agent Evaluation: Involves assessing actions in a real environment (e.g., sending emails, querying databases).
  • Cost: Can be unbounded and hard to predict, as it depends on the number of tool calls and iterations in the agent loop.
  • Task-Specificity: Evaluations must be custom-built for the agent’s specific domain and goals.
  • Focus on Reliability: The key metric shifts from one-off capability to reliability. An agent must perform its task correctly and consistently across many runs. A system that is 95% successful on one run is less valuable than one that is 80% successful across all runs, as you want a consistent and trustworthy user experience.

Building Agents with Google Gemini

Developers can start building agents with Gemini using a variety of tools:

• AI Studio: A web-based platform for experimenting with Gemini models. It provides a user-friendly interface for prompting, testing tools (like built-in Google Search), and getting an API key.
• Gemini SDK: The SDK simplifies development by handling the function-calling loop automatically. A developer can define a tool as a standard Python function, and the SDK will convert its docstring and parameters into the required JSON schema, manage the API calls, and parse the model’s response.
• Open Source Ecosystem: Gemini is deeply integrated with popular frameworks to ensure developers are not limited in their choices. Key integrations include:
  • LangChain & LangGraph: For building composable, graph-based multi-agent systems.
  • LlamaIndex: For building agents that can reason over large volumes of documents.
  • CrewAI: For creating role-based, collaborative agent systems.

For practical examples, Google has open-sourced a Deep Research Agent built with LangGraph and the Gemini CLI, which serve as excellent references for understanding system prompts and tool implementation.

• Tags:
• AI Agents
• LLM
• Gemini
• Feature
• MLOps
• Model Evaluation
• Function Calling
• Agentic Patterns