Implementing Concurrent Tool Execution

Welcome back! In the previous lesson, you successfully enabled your agent system to handle multiple concurrent conversations using Ruby threads. Each conversation runs in its own thread, allowing your system to manage many users simultaneously. However, there is still a critical bottleneck within each individual conversation: when Claude requests multiple tools in a single turn, those tools execute one after another.

In this lesson, we will remove that bottleneck by parallelizing tool execution within a single agent turn. You will learn how to use Ruby threads to execute multiple tool calls concurrently, dramatically improving your system's efficiency when Claude needs to perform several calculations or operations at once.

Let's examine why sequential tool execution creates a bottleneck within a single agent turn. Currently, when Claude requests multiple tools in one response, our agent processes them one at a time. This sequential approach works, but it is inefficient.

Here is what happens with the current sequential approach. When Claude asks to find the square roots of three different numbers, it might request three separate square_root tool calls in a single response. With sequential execution, the agent calls the first square_root function, waits for it to complete, then calls the second, waits again, and finally calls the third.

If each calculation takes 100 milliseconds, the total time is 300 milliseconds — even though these three calculations are completely independent and could happen simultaneously.

This becomes especially problematic when tools involve external operations. Imagine you have a tool that makes an HTTP request to a weather API, taking 2 seconds to complete. If Claude requests weather data for three different cities in one turn, sequential execution would take 6 seconds total. But since these are three independent network requests, they could all happen at the same time, reducing the total wait to just 2 seconds. The solution is to execute independent tool calls concurrently using Ruby threads, allowing multiple tools to run in parallel during a single agent turn. Let's see how to implement this.

Our tool functions, like sum_numbers and multiply_numbers, are regular synchronous Ruby methods. We do not need to change them at all. Instead, we will change how the agent calls these tools. Rather than executing each tool sequentially, we will spawn a separate Ruby thread for each tool call and let them all run concurrently.

Let's look at how this works in the run method. When Claude's response includes tool usage, we iterate through all the tool use requests and separate them into regular tools and handoffs:

if response.stop_reason.to_s == "tool_use"
  tool_uses = []
  handoff_use = nil

  response.content.each do |content_item|
    next unless content_item.type.to_s == "tool_use"

    if content_item.name == "handoff"
      handoff_use = content_item
    else
      tool_uses << content_item
    end
  end

The key insight here is that we are collecting all the regular tool calls into the tool_uses array before executing any of them. This allows us to process them all at once instead of one at a time. We handle handoffs separately because they transfer control to another agent — we need to know if a handoff succeeds before continuing. Regular tools, however, are independent operations that can run concurrently. Once we have separated the tool uses, we can execute them in parallel using threads.

With all regular tool calls collected in the tool_uses array, we can now execute them concurrently using Ruby threads. We will create one thread for each tool call and then wait for all threads to complete:

# Execute all tool calls concurrently using threads
tool_threads = tool_uses.map do |tu|
  Thread.new { call_tool(tu) }
end
tool_results = tool_threads.map(&:value)

This is where the magic happens. For each tool use object in tool_uses, we create a new thread with Thread.new { call_tool(tu) }. Each thread immediately starts executing its call_tool method in parallel with all the others. The map operation returns an array of thread objects, which we store in tool_threads.

Then we use tool_threads.map(&:value) to wait for all threads to complete and collect their results. The value method on a thread blocks until that thread finishes executing and returns the result. By mapping value over all threads, we wait for every tool to complete and gather all the results into the tool_results array, preserving the original order.

This means if Claude requests three tool calls, all three execute simultaneously, and the total time is determined by the slowest tool rather than the sum of all tools. Now let's see how handoffs integrate with this concurrent execution.

After executing all regular tools concurrently, we need to handle any handoff requests. handoffs transfer control to another agent, so they require special treatment:

if handoff_use
  success, handoff_result = call_handoff(handoff_use, messages)
  return handoff_result if success
  tool_results << handoff_result
end

messages << { role: "user", content: tool_results }

If Claude requested a handoff (stored in handoff_use), we call call_handoff with the handoff request and the current conversation messages. This method attempts to transfer control to another specialized agent. It returns two values: a boolean success indicating whether the handoff worked, and the handoff_result containing either the response from the target agent or an error message.

If the handoff succeeds (success is true), we immediately return the result from the target agent, ending this agent's involvement in the conversation. If the handoff fails — for example, if the target agent doesn't exist — we add the error message to tool_results so Claude can see what went wrong and adjust its approach.

Finally, we append all tool results to the messages array as a user message, allowing Claude to process the outcomes and continue the conversation. Notice that handoffs are processed after all regular tools complete, ensuring Claude has all the information it needs before potentially transferring control.

Let's examine the complete flow of how our agent handles tool execution, including the interaction between concurrent tool calls and handoffs:

if response.stop_reason.to_s == "tool_use"
  tool_uses = []
  handoff_use = nil

  response.content.each do |content_item|
    next unless content_item.type.to_s == "tool_use"

    if content_item.name == "handoff"
      handoff_use = content_item
    else
      tool_uses << content_item
    end
  end

  # Execute all tool calls concurrently using threads
  tool_threads = tool_uses.map do |tu|
    Thread.new { call_tool(tu) }
  end
  tool_results = tool_threads.map(&:value)

  if handoff_use
    success, handoff_result = call_handoff(handoff_use, messages)
    return handoff_result if success
    tool_results << handoff_result
  end

  messages << { role: "user", content: tool_results }

When Claude's response includes tool usage, we first iterate through all the tool requests and separate them into two categories: regular tools go into tool_uses, and any handoff request is stored in handoff_use. This separation is crucial because regular tools can run in parallel, but handoffs transfer control.

For all regular tools, we create a thread for each one with Thread.new { call_tool(tu) }. These threads start executing immediately and run concurrently. We then collect all the thread objects and call map(&:value) to wait for every thread to finish and gather their results. This is a blocking operation — we will not proceed until all tools have completed — but the tools themselves run in parallel, so the total wait time is minimized.

After all regular tools complete, we check if there was a handoff request. If so, we attempt the handoff using call_handoff, which includes special logic to clean up the messages before passing them to the target agent. Specifically, it removes the last assistant message (the one containing the tool requests) if it exists, ensuring the target agent receives a clean conversation history.

If the handoff succeeds, we return immediately with the target agent's response. If it fails, we add the error to our tool_results. Finally, we append all results to the conversation, allowing Claude to see the outcomes and continue. This flow ensures maximum parallelism for independent operations while maintaining correct control flow for handoffs.

When we run our agent with a request that triggers multiple tool calls, the output demonstrates how tools execute simultaneously:

🔧 [math_assistant] Tool called: square_root({:a=>144})
🔧 [math_assistant] Tool called: square_root({:a=>256})
🔧 [math_assistant] Tool called: square_root({:a=>625})

Final response:
Here are the square roots:

| Number | Square Root |
|--------|-------------|
| 144    | **12**      |
| 256    | **16**      |
| 625    | **25**      |

All three are perfect squares, meaning their square roots are whole numbers! 🎯

Notice how all three square_root tool calls appear in rapid succession. This happens because they are executing in separate threads concurrently. The log lines may even interleave or appear in slightly different orders on different runs, depending on thread scheduling. The key observation is that all three tool calls start essentially at the same time — rather than waiting for each to complete before starting the next.

The final response shows that Claude successfully processed all three results and presented them in a clean, organized format. The agent maintained the same quality of results while executing the tools much faster than sequential execution would have allowed. If each square_root calculation took 100 milliseconds, sequential execution would require 300 milliseconds total, but concurrent execution completes in just over 100 milliseconds — the time of the slowest single operation.

This performance improvement becomes even more dramatic with tools that involve network requests or database queries. Imagine a tool that fetches data from an external API, taking 2 seconds per call. If Claude requests this tool three times in one turn, sequential execution would take 6 seconds, but concurrent execution would take only 2 seconds. The efficiency gains scale with the number of independent tools requested in a single turn.

You have successfully parallelized tool execution within your Ruby agent system by using threads to run multiple tool calls concurrently. You learned how to separate tool uses from handoffs, spawn a thread for each independent tool call using Thread.new, and synchronize all threads using map(&:value) to collect results.

The performance improvement is significant: instead of executing tools one at a time, your agent now runs multiple tools simultaneously, reducing the total time to roughly the duration of the slowest single tool.

To practice these concepts, try the following exercises:

1. Multiple Tool Types: Modify main.rb to ask a question that requires different tool types in one turn, such as "Calculate (5 + 3) and then find the square root of 64." Observe how both sum_numbers and square_root execute concurrently in the logs.

2. Error Handling: Request a calculation that will cause an error, like "Divide 10 by 0 and also calculate 5 times 3." Verify that the error in divide_numbers (division by zero) does not prevent multiply_numbers from completing successfully, and that Claude receives both results.

3. Sequential vs Concurrent Timing: Add sleep(1) inside one of your tool functions to simulate a slow operation (like an API call). Request that tool three times in one question and observe how concurrent execution takes about 1 second total instead of 3 seconds.

4. Complex Calculations: Ask Claude to perform a complex calculation that requires many steps, such as "Find the square roots of 144, 256, 625, and 900, then sum all the results." Observe how all four square_root calls execute concurrently, followed by the sum_numbers call in a second turn.

In the next lesson, we will explore how to combine conversation-level parallelism with tool-level parallelism, enabling your system to handle multiple users each making multi-tool requests simultaneously.