Routing Between Many Agent Tools

By Michael Samuel Naeem · July 5, 2026 · 5 min read

Tool selection accuracy falls off a cliff around tool fifteen. I've watched an agent with thirty tools call delete_user when the user asked for account settings — not because the model is careless, but because thirty tool descriptions in context creates a needle-in-a-haystack problem that gets worse with every tool you add. The fix isn't better prompting; it's a routing layer that presents the model with five relevant tools instead of fifty. Tool routing is the scaling mechanism nobody builds until their agent starts calling the wrong API.

The two-stage pattern

User query → Router (50 tools → 5-8 candidates) → LLM (picks 1-2 tools) → Execute

Stage 1 is fast and cheap. Stage 2 is the model doing what it's good at — choosing among a small, relevant set.

Tool groups with intent routing

Organize tools into groups and classify intent first:

TOOL_GROUPS = {
    "account": ["get_profile", "update_profile", "change_password", "delete_account"],
    "billing": ["get_invoices", "create_refund", "update_payment_method"],
    "orders": ["list_orders", "get_order", "cancel_order", "track_shipment"],
    "support": ["create_ticket", "search_kb", "escalate"],
}

async def route_tools(user_message: str) -> list[str]:
    intent = await classify_intent(user_message)  # cheap model call
    primary = TOOL_GROUPS.get(intent, [])
    # Always include search as fallback
    return primary + ["search_kb"]

The classifier is a 50-token call to a small model: "Classify this message: account, billing, orders, or support." It costs fractions of a cent and runs in 100ms.

Embedding-based routing for dynamic catalogs

When tools change frequently or groups aren't obvious:

async def route_by_embedding(query: str, top_k: int = 8) -> list[ToolDef]:
    query_embedding = await embed(query)
    scores = [
        (tool, cosine_similarity(query_embedding, tool.embedding))
        for tool in all_tools
    ]
    return [tool for tool, score in sorted(scores, key=lambda x: -x[1])[:top_k]]

Pre-compute tool description embeddings offline. At runtime, embed the query and return top-K. Include tool name, description, and one example use case in the embedded text for better matching.

Tool description quality matters more than count

Before adding routing, fix your tool descriptions. The routing layer can't compensate for:

# Bad — model can't distinguish tools
{"name": "process", "description": "Process a request"}

# Good — specific, disambiguated
{"name": "create_refund", "description": "Issue a refund for a completed order. Requires order_id and amount. Use only after verifying eligibility with get_order."}

Add negative scope to descriptions: "Use for X, NOT for Y." This helps both the router and the model.

Dynamic tool loading per workflow phase

Combine routing with graph workflows — different phases expose different tools:

The model never sees action tools during intake. This is routing by architecture, not by algorithm, and it's the most reliable approach.

Sub-agents as extreme routing

When tool groups have different security boundaries, route to sub-agents instead of tool groups:

• Research sub-agent: 8 read tools
• Executor sub-agent: 5 write tools
• Orchestrator: 2 meta-tools (delegate, synthesize)

Each sub-agent's routing problem is trivial because its catalog is already narrow.

Measuring routing quality

Track these metrics:

• Tool selection accuracy: did the model pick the right tool from the routed set?
• Routing recall: was the correct tool in the routed candidate set?
• Routing precision: what fraction of routed tools were actually used?

Low recall means your router is too aggressive — widen top-K or fix group boundaries. Low selection accuracy with high recall means descriptions or prompts need work, not routing.

Log routing decisions in your agent traces for debugging misfires.

Embedding-based tool retrieval

When tool catalogs exceed 20–30 entries, flat routing groups aren't enough. Embed tool descriptions and retrieve top-K by query similarity:

async def route_tools(query: str, k: int = 5) -> list[Tool]:
    query_vec = embed(query)
    candidates = vector_index.search(query_vec, top_k=k * 2)
    return [t for t in candidates if t.risk_tier <= session.max_risk_tier][:k]

Combine embedding retrieval with hard filters:

• Permission filter — user role excludes delete_account
• Environment filter — staging tools hidden in production
• State filter — create_refund only if order exists in session context

Embedding recall without filters causes dangerous tool exposure — a "delete" tool retrieved because the user said "remove from my cart."

Fallback and escalation paths

Routing should degrade gracefully:

Never loop indefinitely re-routing the same failed tool. Escalate to human after N failures on write operations.

Production checklist

• Tool descriptions include parameters, returns, and negative scope
• Router logs: query hash, candidates, selected tools, latency
• Weekly review of unused tools — remove from catalog to reduce noise
• Integration tests per workflow phase with expected tool sets
• Write tools require explicit user confirmation or HITL approval

Pair with tool use error recovery when routed tools fail mid-execution.

Production checklist

• Tool descriptions include negative scope ("NOT for...")
• Router logs query hash, candidates, and selection latency
• Write tools gated by workflow phase, not just routing
• Weekly review of unused tools in catalog
• Embedding retrieval filtered by permission before similarity rank

Most production routing incidents trace to stale tool descriptions, not embedding model quality — schedule a monthly catalog review with the teams that own each tool.

Resources

• OpenAI function calling best practices
• Anthropic tool use guide
• LangChain tool retrieval
• Berkeley Function Calling Leaderboard
• Parallel tool execution patterns