Parallelizing OpenAI Agent Systems

Welcome to the first lesson of Parallelizing OpenAI Agentic Systems in Ruby! In this lesson, you'll learn how to use Ruby's threading capabilities to run multiple concurrent conversations with gpt-5. You've already seen the Agent class that can handle conversations, use tools, and hand off control to other specialized agents.

To keep the core concurrency concept easier to retain, this lesson is split into three checkpoints:

1. Agent prerequisites: the small parts of Agent you need before adding threads.
2. Thread orchestration: the main delta — creating Thread objects, collecting Thread#value, and keeping client state isolated.
3. Concurrency limitations: what still runs sequentially, including per-turn tool loops and handoffs.

Focus most of your attention on the thread orchestration pattern. The Agent internals are included only as prerequisites so you can see why concurrent runs are safe.

Let's start by understanding why we need concurrency in the first place. When you make a regular API call to OpenAI's Responses API, your program stops and waits for the response. This is called blocking behavior. If you need to have three separate conversations with gpt-5, your program handles them one at a time: start conversation 1, wait for all responses, finish conversation 1, then start conversation 2, and so on.

This sequential approach wastes time. While your program waits for gpt-5 to respond to conversation 1, it could be starting conversation 2 or 3. Network calls and API processing take time, but your CPU sits idle during these waits.

Concurrent execution using Ruby threads solves this problem. When you create a thread for an API call, your program can continue creating more threads and starting other conversations while waiting for responses. Think of it like a restaurant: a sequential waiter takes one order, goes to the kitchen, waits for the food, delivers it, and only then takes the next order. A concurrent approach is like having multiple orders in flight — while one meal is being prepared, other orders are being taken and other meals are being delivered.

This concurrent approach is particularly effective for I/O-bound operations like API calls, network requests, and database queries, where most of the time is spent waiting rather than computing. For CPU-heavy tasks that require intense computation, threads won't provide the same benefits since Ruby threads (depending on your Ruby implementation) may not run truly in parallel for CPU-bound work. However, for network I/O like API calls to gpt-5, threads allow you to overlap the waiting time.

For agent systems, this means a single agent can manage multiple conversations simultaneously by launching each conversation in its own thread. This becomes especially powerful when you need to process multiple independent user requests or run several agents in parallel. Let's see how the provided code implements this pattern.

Before we dive into concurrent execution, let's understand the key parts of the Agent class that make concurrent conversations possible.

Unlike a typical system prompt, the Agent class uses a developer prompt that prepends a base instruction set explaining the agent's agentic behavior:

BASE_DEVELOPER_PROMPT = (
  "You are an autonomous agent that can take multiple tool-calling steps when helpful. " \
  "The user only sees your response when you stop using tools, not your tool usage or reasoning steps. " \
  "When you provide your answer without calling tools, make it complete and standalone.\n" \
  "Additional instructions:\n"
)

Your custom system_prompt is appended to this base, and the result is injected as the first message with role: "developer" in every request.

The OpenAI Responses API expects a specific message format. The Agent class provides a text_message helper to build these structured input messages:

def text_message(role, text)
  {
    role: role,
    content: [
      { type: "input_text", text: text }
    ]
  }
end

Each message is a hash containing a role and a content array of typed content blocks. The "input_text" type is used for regular text exchanges — whether from the "developer", "user", or "assistant".

The build_request_args method assembles the full payload sent to the API:

def build_request_args(messages)
  all_tools = @tool_schemas.dup
  all_tools << @handoff_schema unless @handoffs.empty?

  args = {
    model: @model,
    input: [
      text_message("developer", @developer_prompt),
      *messages
    ],
    reasoning: { effort: @reasoning_effort },
    store: false
  }

  args[:tools] = all_tools unless all_tools.empty?
  args
end

Notice that the developer prompt is always prepended as the first element of the input array, followed by the conversation history.

A crucial detail is where the OpenAI::Client is instantiated. Rather than creating it once during initialization and sharing it across all threads, a fresh client is created inside create_response for every API call:

def create_response(messages)
  client = OpenAI::Client.new
  client.responses.create(**build_request_args(messages))
end

This means each API call — and each concurrent thread — uses its own isolated HTTP client instance, avoiding any shared state between threads.

After receiving a response, the run method inspects response.output to decide what to do next:

function_calls = response.output.select do |item|
  item.type.to_s == "function_call"
end

If any function_call items are found in the output, the agent processes them and loops. Tool results are returned as function_call_output messages:

{
  type: "function_call_output",
  call_id: call_id,
  output: JSON.generate(result: result)
}

If no function_call items are found, the agent returns the final text response via response.output_text and exits the loop.

Now let's look at how to set up multiple concurrent conversations. The provided main.rb shows the pattern. First, we define multiple prompts that we want to process:

prompts = [
  "Calculate the compound interest for $1000 at 5% for 10 years.",
  "What is the derivative of x^3 + 2x^2 - 5?",
  "Solve for x: 3x + 15 = 45"
]

Next, we create a single agent instance that will handle all conversations:

agent = Agent.new(
  name: "math_expert",
  system_prompt: "You are a concise math assistant."
)

This agent doesn't have any tools or tool_schemas configured in this example — it will rely purely on gpt-5's built-in mathematical reasoning capabilities and answer directly without calling any tools. If you wanted to provide specific tools (like calculator functions), you would pass them via the tools: and tool_schemas: parameters as supported by the Agent class.

With our agent and prompts ready, we can now create threads to run multiple conversations concurrently:

threads = prompts.map do |prompt|
  Thread.new do
    agent.run([text_message("user", prompt)])
  end
end

This code creates one thread for each prompt using Thread.new. Inside each thread's block, we call agent.run() with a single user message built by the text_message helper:

def text_message(role, text)
  {
    role: role,
    content: [
      { type: "input_text", text: text }
    ]
  }
end

The main.rb file defines its own top-level version of this helper so message creation stays consistent with the Agent class internals. The map operation returns an array of thread objects, and importantly, the threads have already started executing at this point.

The key insight here is that each thread runs its own conversation independently. While one thread waits for gpt-5's response to the compound interest question, another thread can be waiting for the x^3 + 2x^2 - 5 derivative question, and a third can be waiting for the 3x + 15 = 45 equation solving question. Ruby's thread scheduler handles switching between threads efficiently.

To collect the results, we wait for all threads to complete:

puts "🚀 Processing #{prompts.length} requests concurrently..."

results = threads.map(&:value)

The Thread#value method blocks until the thread completes and returns the value returned by the thread's block (in this case, the result of agent.run()). By calling threads.map(&:value), we wait for all threads to finish and collect all their results in order. This is the synchronization point where the main thread waits for all conversations to complete.

Finally, we display the results:

results.each_with_index do |(_, text), i|
  puts "\nResult #{i + 1}:"
  puts text
end

Note that agent.run() returns a tuple of [messages, text], so we destructure it and display only the final text response.

Running this concurrent agent system is straightforward — it's just a regular Ruby script:

ruby src/main.rb

When Ruby executes this script, it:

1. Loads the required classes and creates the agent
2. Creates and immediately starts all threads (in prompts.map { Thread.new { ... } })
3. Continues to the threads.map(&:value) line, which blocks waiting for all threads
4. Once all threads complete, displays the results and exits

The concurrency happens automatically between steps 2 and 3. While the main thread is blocked at threads.map(&:value), Ruby's thread scheduler switches between the worker threads, allowing them to make progress on their OpenAI Responses API calls.

Each thread independently calls agent.run(), which internally calls create_response, which instantiates a fresh OpenAI::Client for that specific request. This means the three concurrent conversations each use their own separate HTTP client — no connection sharing occurs between threads.

This is different from a purely sequential approach where you might have:

results = prompts.map do |prompt|
  agent.run([text_message("user", prompt)])
end

In the sequential version, each conversation would complete fully before the next one started, wasting time during network I/O waits.

When you run the concurrent version, you'll see output like this:

🚀 Processing 3 requests concurrently...

Result 1:
Assuming annual compounding:
- Amount after 10 years: A = 1000(1.05)^10 ≈ 1000 × 1.628894626 ≈ $1,628.89
- Compound interest earned: A − P ≈ 1628.89 − 1000 = $628.89

Result 2:
3x^2 + 4x

Result 3:
x = 10

⏱️ Finished in 4.66 seconds

The key observation is that all three conversations happen concurrently, even though the output is displayed sequentially (because we wait for all threads to complete before displaying results). The timing line confirms the total elapsed time — because the API wait times overlap, the three conversations complete in roughly the time it would take for just one, rather than three times as long.

If the agent were configured with tools, you might see interleaved tool call logs from different threads:

# Example with tools configured
agent = Agent.new(
  name: "math_expert",
  system_prompt: "You are a math assistant.",
  tools: { "multiply" => ->(a:, b:) { a * b } },
  tool_schemas: [{ "type" => "function", "name" => "multiply", ... }]
)

The output would show tool calls from different conversations intermixed:

🔧 [math_expert] Tool called: multiply({"a"=>5, "b"=>16})
🔧 [math_expert] Tool called: multiply({"a"=>3, "b"=>3})
🔧 [math_expert] Tool called: multiply({"a"=>4, "b"=>4})

This interleaving demonstrates that the conversations are truly running concurrently — tool calls from different threads are being processed as each thread makes progress.

Now that we've seen concurrent execution in action, let's reflect on what we've achieved and what limitations remain.

By wrapping multiple agent.run() calls in separate Ruby threads, we overlap the I/O wait time across independent conversations. Each thread calls create_response, which fires off an HTTP request to the OpenAI Responses API and then blocks while awaiting the response. While one thread is blocked waiting, Ruby's scheduler gives CPU time to other threads that are ready to make progress. For three math problems processed concurrently, the total elapsed time is roughly that of the single slowest request, not the sum of all three.

Inside one call to agent.run(), operations are sequential. When gpt-5 requests multiple tool uses in a single response turn, the agent processes them one at a time in the loop:

function_calls.each do |function_call|
  messages << function_call_message(function_call)
  function_outputs << call_tool(function_call)
end

If gpt-5 returns five function_call items in one turn, they execute sequentially within that thread, even though other conversations are running in parallel threads.

If an agent transfers control to another agent via a handoff, the current thread blocks completely while the target agent runs its full conversation:

success, result = call_handoff(function_call, messages)
return result if success

Handoffs are synchronous within a thread — they do not launch a new thread.

The provided code deliberately sidesteps the most common concurrency pitfall around shared HTTP client state. Rather than creating one OpenAI::Client at initialization and sharing it across all threads, a new client is created inside create_response on every call:

def create_response(messages)
  client = OpenAI::Client.new
  client.responses.create(**build_request_args(messages))
end

This means each thread operates with a fully independent client instance, including its own connection pool and internal state. You don't need to worry about threads stepping on each other's HTTP connections or corrupting shared client state.

The degree of true parallelism depends on your Ruby implementation. MRI (the standard Ruby) uses the Global VM Lock (GVL), which prevents true parallel execution of Ruby code. However, for I/O-bound operations like API calls, the GVL is released during network waits, so you still get meaningful concurrency benefits. Alternative Ruby implementations like JRuby or TruffleRuby can provide true parallel thread execution even for CPU-bound work.

In the next lessons, we'll explore patterns for parallelizing tool execution within a single conversation and coordinating multiple specialized agents concurrently.

You've successfully learned how to use Ruby threads to run multiple concurrent conversations with gpt-5 through the OpenAI Responses API. The key pattern is:

1. Create multiple threads using Thread.new { agent.run([text_message("user", prompt)]) }
2. Each thread runs an independent conversation with its own fresh OpenAI::Client
3. Collect results with threads.map(&:value) to wait for completion

This threading approach allows you to overlap I/O wait time across multiple conversations, dramatically improving throughput when processing multiple independent requests.

In the upcoming practice, you will apply these concepts by building a system that processes a collection of diverse prompts in parallel using the provided Agent implementation and text_message helper.