Gemini Tool Use Workflow

Welcome to understanding how Gemini handles tool use in its API! In previous lesson, you learned how to define tool schemas that describe your functions for Gemini. Now, you'll discover how to include these tools in your Gemini API requests and, most importantly, how to interpret Gemini's responses when it decides to call a tool.

In this lesson, you'll learn how to configure your requests to enable tool use, understand the different types of responses Gemini can provide, and extract the specific information you need to execute the tools Gemini requests. By the end, you'll be able to recognize when Gemini wants to use a tool and gather all the details needed to make that happen.

It is important to understand that Gemini does not execute your code. When Gemini "calls a tool," it is actually sending a structured request back to you. Your Python application is responsible for the heavy lifting:

1. Parsing the request from Gemini.
2. Executing the actual Python function locally in your environment.
3. Sending the function's output back to Gemini to continue the conversation.

This design ensures you maintain full control over your security and environment, as the model never has direct access to your local system.

Integrating tools into your agent involves a specific lifecycle. To build a functional tool-calling agent, you must follow these steps:

• Define: Describe your tools using function schemas.
• Provide: Pass these tools to Gemini in your API request.
• Detect: Inspect the response structure to see if Gemini wants to call a tool.
• Extract: Pull the tool name and arguments from the response.
• Execute: Run your local code with the provided arguments.
• Return: Send the results back to Gemini to complete the interaction.

Let's dive into the details of how this works in practice.

Gemini can automatically detect and use any tools you provide in your request. However, you can guide Gemini's behavior by including instructions in the system message or prompt, which helps ensure more consistent and appropriate tool usage.

Here's an example of how you can provide tool usage guidance in Gemini's system instruction:

from google import genai
from google.genai import types

# System instruction with optional tool usage guidance
system_instruction = (
    "You are a helpful math assistant. "
    "When performing calculations, use the available tools for accuracy."
)

When you send a request to Gemini, you can include this instruction as part of the system_instruction parameter (or as the first message in the conversation, depending on the API version). This guidance helps Gemini understand your preferences for when and how to use tools, but it is not required for tool functionality. Gemini can still recognize and use tools based solely on their availability in the request.