Parallelizing Agent Tools

Welcome back! In the previous lesson, you successfully enabled your agent system to handle multiple concurrent conversations using Ruby threads. Each conversation runs in its own thread, allowing your system to manage many users simultaneously. However, there is still a critical bottleneck within each individual conversation: when the model requests multiple tools in a single turn, those tools execute one after another.

In this lesson, we will remove that bottleneck by parallelizing tool execution within a single agent turn. You will learn how to use Ruby threads to execute multiple function_call items returned by client.responses.create concurrently, dramatically improving your system's efficiency when the model needs to perform several calculations or operations at once.

Let's examine why sequential tool execution creates a bottleneck within a single agent turn. Currently, when the model requests multiple tools in one response, our agent processes them one at a time. This sequential approach works, but it is inefficient.

Here is what happens with the current sequential approach: when the model is asked to find the square roots of three different numbers, it may request three separate square_root function calls in a single response. With sequential execution, the agent calls the first square_root function, waits for it to complete, then calls the second, waits again, and finally calls the third.

If each calculation takes 100 milliseconds, the total time is 300 milliseconds, even though these three calculations are completely independent and could happen simultaneously.

This becomes especially problematic when tools involve slow operations. Imagine you had a tool that performs a complex computation taking 2 seconds to complete. If the model requests that tool three times in one turn, sequential execution would take 6 seconds total. But since these are three independent operations, they could all happen at the same time, reducing the total wait to just 2 seconds. The solution is to execute independent function calls concurrently using Ruby threads, allowing multiple tools to run in parallel during a single agent turn. Let's see how to implement this.

Our tool functions — like sum_numbers and square_root — are regular synchronous Ruby methods. We do not need to change them at all. Instead, we change how the agent calls these tools inside the run method of the Agent class.

The Responses API returns a response object whose output is an array of items. Each tool request from the model arrives as an item with type == "function_call". To find all tool requests in a response, we filter response.output by that type:

function_calls = response.output.select do |item|
  item.type.to_s == "function_call"
end

If function_calls is non-empty, the model wants to call tools. We then separate regular tool calls from the special handoff call — since they must be handled differently:

if function_calls.any?
  tool_calls = []
  handoff_call = nil

  function_calls.each do |function_call|
    if function_call.name == "handoff"
      handoff_call = function_call
    else
      tool_calls << function_call
    end
  end

Regular tool calls — like square_root or sum_numbers — are collected in the tool_calls array. Any handoff request is stored separately in handoff_call. This separation is essential: regular tools are independent operations that can run concurrently, while a handoff transfers control to another agent and requires special handling. Once we have made this separation, we can execute all regular tools in parallel using threads.

With all regular function calls collected in the tool_calls array, we can execute them concurrently. We create one thread per function call, let them all run simultaneously, and then wait for every thread to finish:

tool_threads = tool_calls.map do |function_call|
  Thread.new do
    call_tool(function_call)
  end
end

For each function_call object in tool_calls, we create a new thread with Thread.new { call_tool(function_call) }. Each thread immediately starts executing its call_tool method in parallel with all the others — the map operation returns an array of thread objects stored in tool_threads.

Inside call_tool, the function is executed and the result is wrapped in a function_call_output hash — the format the Responses API expects when feeding tool results back into the conversation:

def call_tool(function_call)
  tool_name  = function_call.name
  call_id    = function_call.call_id
  tool_input = JSON.parse(function_call.arguments || "{}")

  result = @tools.fetch(tool_name).call(**tool_input.transform_keys(&:to_sym))

  {
    type:   "function_call_output",
    call_id: call_id,
    output: JSON.generate(result: result)
  }
end

After spawning all threads, we gather their results with map(&:value):

function_outputs.concat(tool_threads.map(&:value))

The value method blocks until the thread finishes and returns its result. By mapping value over all threads, we wait for every tool to complete and collect all function_call_output hashes into a single array, preserving order. The total wait time is determined by the slowest individual tool rather than the sum of all tools.

The run method handles any handoff request before regular tool threads are started. Because a handoff can redirect the entire conversation to another agent, it must be resolved first:

if handoff_call
  success, result = call_handoff(handoff_call, messages)
  return result if success

  messages << function_call_message(handoff_call)
end

If the model included a handoff function call in its response, call_handoff is invoked immediately. It returns two values: a boolean success and a result. If the handoff succeeds, the run method returns the target agent's response right away, ending this agent's involvement in the conversation.

If the handoff fails — for example, because the named target agent does not exist — success is false and result is a function_call_output hash containing an error message. In that case, the code appends the function_call_message for the failed handoff to the conversation history, and the error output is later merged into function_outputs so the model can see what went wrong:

function_outputs = []
function_outputs << result if handoff_call && !success
function_outputs.concat(tool_threads.map(&:value))

messages.concat(function_outputs)

This design means regular tools always run concurrently, while a handoff is either resolved immediately — returning control to the caller — or converted into a plain function_call_output error message that feeds back into the next model turn.

Let's look at the complete run method to see how all the pieces fit together:

def run(input_messages)
  messages = input_messages.map(&:dup)
  turn = 0

  while turn < @max_turns
    turn += 1

    response = create_response(messages)

    function_calls = response.output.select do |item|
      item.type.to_s == "function_call"
    end

    if function_calls.any?
      tool_calls   = []
      handoff_call = nil

      function_calls.each do |function_call|
        if function_call.name == "handoff"
          handoff_call = function_call
        else
          tool_calls << function_call
        end
      end

      if handoff_call
        success, result = call_handoff(handoff_call, messages)
        return result if success

        messages << function_call_message(handoff_call)
      end

      tool_calls.each do |function_call|
        messages << function_call_message(function_call)
      end

      # Execute independent tool calls concurrently.
      tool_threads = tool_calls.map do |function_call|
        Thread.new do
          call_tool(function_call)
        end
      end

      function_outputs = []
      function_outputs << result if handoff_call && !success
      function_outputs.concat(tool_threads.map(&:value))

      messages.concat(function_outputs)
    else
      messages << text_message("assistant", response.output_text)
      return [messages, response.output_text]
    end
  end

  raise "Max turns reached"
end

The loop runs until the model stops calling tools or max_turns is reached. Each iteration follows these steps:

1. Creates a response by calling create_response(messages), which sends the full conversation history to the Responses API.
2. Collects function calls by filtering response.output for items whose type is "function_call".
3. Separates the handoff from regular tool calls, storing them in handoff_call and tool_calls respectively.
4. Attempts the handoff if present. A successful handoff returns immediately; a failed handoff appends its function_call_message to history and stores the error output in result.
5. Appends function_call_message items for every regular tool call. These records tell the Responses API which functions were invoked with which arguments and call IDs.
6. Spawns threads — one per regular tool call — and runs all of them concurrently.
7. Builds function_outputs: the failed-handoff error (if any) followed by all function_call_output hashes from the concurrent threads.
8. Appends function_outputs to the conversation so the model can read every tool result in the next turn.

When function_calls is empty, the model is done calling tools. The assistant's final text is extracted via response.output_text, appended to messages, and returned together with the full conversation history.

When we run our agent with a request that triggers multiple tool calls, the output demonstrates how tools execute simultaneously:

🔧 [math_assistant] Tool called: square_root({"a"=>144})
🔧 [math_assistant] Tool called: square_root({"a"=>256})
🔧 [math_assistant] Tool called: square_root({"a"=>625})

Final response:
The principal square roots are:
- √144 = 12
- √256 = 16
- √625 = 25

Notice how all three square_root tool calls appear in rapid succession. This happens because they are executing in separate threads concurrently. The log lines may even interleave or appear in slightly different orders on different runs, depending on thread scheduling. The key observation is that all three tool calls start essentially at the same time — rather than waiting for each to complete before starting the next.

If each square_root calculation took 100 milliseconds, sequential execution would require 300 milliseconds total, but concurrent execution completes in just over 100 milliseconds — the time of the slowest single operation.

To observe this timing effect yourself, add a short sleep inside one of the existing tool functions in functions.rb:

def square_root(a:)
  raise ArgumentError, "a must be non-negative" if a < 0

  sleep(1) # simulate a slow operation
  Math.sqrt(a)
end

With this change, asking the agent to compute three square roots concurrently will still take only about 1 second instead of 3 seconds, because all three threads sleep simultaneously. Remove the sleep call when you are done experimenting.

You have successfully parallelized tool execution within your Ruby agent system by using threads to run multiple function calls concurrently. You learned how to:

• Filter response.output for function_call items to detect tool requests from the model
• Separate regular tool calls from the handoff call inside Agent#run
• Spawn a thread for each independent tool call using Thread.new
• Synchronize all threads with map(&:value) to collect function_call_output results in order
• Append both function_call_message and function_call_output records to the conversation so the Responses API loop can continue correctly

The performance improvement is significant: instead of executing tools one at a time, your agent now runs multiple tools simultaneously, reducing the total time to roughly the duration of the slowest single tool.

To practice these concepts, try the following exercises:

1. Multiple Tool Types: Modify main.rb to ask a question that requires different tool types in one turn, such as "Calculate 5 + 3 and find the square root of 64." Observe how both sum_numbers and square_root execute concurrently in the logs.

2. Error Handling: Request a calculation that will cause an error, like "Divide 10 by 0 and also calculate 5 times 3." Verify that the error in divide_numbers (division by zero) does not prevent multiply_numbers from completing successfully, and that the model receives both results.

3. Sequential vs Concurrent Timing: Add sleep(1) inside one of your tool functions to simulate a slow operation. Request that tool three times in one question and observe how concurrent execution takes about 1 second total instead of 3 seconds.

4. Complex Calculations: Ask the agent to perform a calculation that requires many steps, such as "Find the square roots of 144, 256, 625, and 900, then sum all the results." Observe how all four square_root calls execute concurrently, followed by the sum_numbers call in a second turn.

In the next lesson, we will explore how to combine conversation-level parallelism with tool-level parallelism, enabling your system to handle multiple users each making multi-tool requests simultaneously.