Executing Agent Tools in Parallel

By Michael Samuel Naeem · July 2, 2026 · 5 min read

Sequential tool execution is the silent latency killer in agents. The model asks for three independent lookups — user profile, order status, shipping address — and your orchestrator runs them one at a time, adding 300ms each, while the user stares at a spinner. Parallel execution cuts that to one round-trip when the tools are truly independent. But "run everything concurrently" creates race conditions on shared state and compliance nightmares on ordered writes. The engineering is in knowing which tools can overlap and building an executor that handles partial failures gracefully.

Dependency analysis

Before executing, classify the tool calls:

@dataclass
class ToolCall:
    id: str
    name: str
    args: dict
    depends_on: list[str] = field(default_factory=list)

def plan_execution(calls: list[ToolCall]) -> list[list[ToolCall]]:
    """Returns batches — each batch can run in parallel, batches run sequentially."""
    batches = []
    remaining = list(calls)
    completed_ids = set()

    while remaining:
        ready = [c for c in remaining if all(d in completed_ids for d in c.depends_on)]
        if not ready:
            raise CyclicDependency("tool calls have circular dependencies")
        batches.append(ready)
        completed_ids.update(c.id for c in ready)
        remaining = [c for c in remaining if c.id not in completed_ids]

    return batches

Most LLM tool calls in a single response are independent — the model asked for three things it needs. Run them as one batch.

The parallel executor

async def execute_batch(calls: list[ToolCall], state: SessionState) -> dict[str, ToolResult]:
    async def run_one(call: ToolCall) -> tuple[str, ToolResult]:
        try:
            result = await asyncio.wait_for(
                execute_tool(call.name, call.args, state),
                timeout=TOOL_TIMEOUT,
            )
            return call.id, ToolResult(success=True, data=result)
        except asyncio.TimeoutError:
            return call.id, ToolResult(success=False, error="timeout")
        except Exception as e:
            return call.id, ToolResult(success=False, error=str(e))

    results = await asyncio.gather(*[run_one(c) for c in calls])
    return dict(results)

asyncio.gather with per-tool timeouts gives you concurrent execution with isolated failure. One timeout doesn't kill the batch.

Encouraging parallel calls from the model

Models default to sequential tool use unless prompted otherwise:

When you need multiple independent pieces of information, call all
required tools in a single response rather than one at a time. Only
call tools sequentially when one result is needed as input to the next.

Also structure your tool definitions to make independence obvious — separate get_user_profile, get_order_status, and get_shipping_info rather than one mega-tool that does all three sequentially internally.

When parallel is wrong

Write operations on the same entity. Never parallelize two updates to the same order, user, or record. The model might call update_status("shipped") and update_status("cancelled") in the same turn.

Compliance-ordered operations. Audit logs requiring "check balance, then deduct" must stay sequential even if they're technically independent reads followed by a write.

Rate-limited APIs. Parallelizing ten calls against an API with 5 req/s limit gets you throttled. Add per-service concurrency limits:

stripe_semaphore = asyncio.Semaphore(3)

async def call_stripe(tool_call):
    async with stripe_semaphore:
        return await execute_tool(tool_call)

Context-dependent tools. If tool B needs tool A's output (look up user ID, then fetch their orders), enforce the dependency in your execution planner, not by hoping the model ordered them correctly.

Partial failure handling

Return structured results so the model can reason about incomplete data:

def format_tool_results(results: dict[str, ToolResult]) -> str:
    parts = []
    for call_id, result in results.items():
        if result.success:
            parts.append(f"[{call_id}] Success: {result.data}")
        else:
            parts.append(f"[{call_id}] Failed ({result.error}): proceed without this data or retry")
    return "\n".join(parts)

The model often produces a useful partial answer when given explicit failure context — better than a generic "tool error" that triggers a full retry loop.

Latency impact

In a typical support agent scenario:

That's nearly 2× improvement from execution alone, before any model optimization. Track p50 and p95 end-to-end latency in your agent traces to verify the gain in production.

Treat production rollout as a measured change: ship with observability, validate rollback, and review metrics 24 hours after deploy — patterns that look obvious in docs fail when skipped under release pressure.

Common production mistakes

Teams get parallel tool execution wrong in predictable ways:

• Skipping failure-mode rehearsal — run a game day or fault injection exercise before peak traffic, not after the first outage.
• Missing correlation context — every error path should carry request, trace, or tenant identifiers so incidents are debuggable.
• Optimizing for demo, not steady state — load tests, cache warm-up, and cold-start paths matter more than local dev latency.
• Undocumented trade-offs — if you chose speed over strict correctness (or vice versa), write that down for the next engineer.

Agent systems using parallel tool execution loop infinitely when tool errors are swallowed, subagent budgets have no hard cap, and human-in-the-loop gates are bypassed under latency pressure.

Debugging and triage workflow

When parallel tool execution misbehaves in production, work top-down instead of guessing:

1. Confirm scope — one tenant, region, or deployment stage? Narrow blast radius before deep diving.
2. Check recent changes — deploys, flag flips, config pushes, and schema migrations in the last 24 hours.
3. Compare golden signals — latency, error rate, saturation, and traffic for the affected surface vs. baseline.
4. Reproduce minimally — smallest input or scenario that triggers the failure; capture traces/logs with correlation IDs.
5. Fix forward or rollback — if rollback is faster than root-cause during incident, rollback first, postmortem second.
6. Add a guard — alert, integration test, or circuit breaker so the same class of failure is caught earlier next time.

Document the timeline during triage. Future you (and on-call) will need timestamps, not just conclusions.

Resources

• OpenAI parallel function calling
• Anthropic tool use documentation
• Python asyncio.gather documentation
• Temporal parallel activity execution
• Agent tool selection and routing