AI agents have moved from chat to action. They now refund orders, open pull requests, update CRM records and send invoices. That changes the engineering problem. A bad chat answer is embarrassing. A duplicated refund is a finance ticket. Once an agent writes to real systems, each tool call needs the reliability engineering we already apply to payments and distributed jobs: idempotency, careful retries, rate limit awareness, deduplication, timeouts, human approval and tracing. Here is how to apply each one, with real provider behaviour and TypeScript you can adapt.

Why agent tool calls fail differently

A tool call has more ways to fail than a normal API request. The model can pick the wrong tool or fill arguments badly. The network can drop the response after the provider has done the work. The provider can throttle you. The agent loop can time out and replay the step.

The dangerous case is the ambiguous one: you sent a write and do not know whether it landed. Retrying might do it twice. Stopping might leave the task half done. Most patterns below exist to make that case safe.

Models also like to retry. Given a vague error, a model will often call the tool again with slightly different arguments. Your code, not the model, should decide what is retryable.

Make every write idempotent

An idempotent operation has the same effect however many times it runs. Reads usually are. Writes need help.

Stripe's idempotent requests are the reference design. The client sends an Idempotency-Key header. Stripe saves the status code and body of the first request for that key, success or failure, and returns the same result for later requests with that key, including 500 errors. Keys can be up to 255 characters, Stripe suggests V4 UUIDs, and keys can be pruned once they are at least 24 hours old. If a reused key arrives with different parameters, Stripe returns an error. Results are only saved once an endpoint starts executing, so requests that fail validation or conflict with a concurrent request can be retried. All POST requests accept keys.

For agents, what matters is where the key comes from. A fresh UUID per attempt defeats the purpose. Derive the key from the logical action: run ID, tool name and the tool call ID the model produced. If the loop replays a step after a crash, it sends the same key.

Many APIs have no idempotency support. For those, record the intent in your database before calling, check for an existing record first, and prefer upsert-by-external-ID endpoints over blind creates. Keep personal data out of keys, as Stripe also advises.

Retry without making things worse

Exponential backoff with jitter

Retries fix transient failures, and cause outages when done badly. If a thousand agents retry after exactly one second, they all hit the recovering service together.

Marc Brooker's Exponential Backoff And Jitter on the AWS Architecture Blog compares strategies. "Full Jitter" sleeps a random time between zero and min(cap, base * 2 ^ attempt). In the article's simulations, Full Jitter and "Decorrelated Jitter" both clearly beat "Equal Jitter", with Full Jitter doing less work for slightly more time. It is simple, so it makes a good default.

Retry only what is safe:

Network errors and timeouts, when the call is idempotent or carries a key.
429, 503 and other 5xx responses, within a small attempt budget.
Not 400, 401, 404 or 422, and not 403 unless it is a rate limit. Return these to the model as structured errors so it can fix its input or stop.

This wrapper combines a stable key, a per-attempt timeout, Full Jitter and server hints:

import { createHash } from "node:crypto";
 
type ToolContext = { runId: string; toolCallId: string; userId: string };
 
export class ToolCallError extends Error {
  constructor(message: string, readonly retryable: boolean, readonly status?: number) {
    super(message);
  }
}
 
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
const fullJitter = (attempt: number, baseMs: number, capMs: number) =>
  Math.random() * Math.min(capMs, baseMs * 2 ** attempt);
 
// Same key for every retry and replay of one logical step.
export const idempotencyKey = (ctx: ToolContext, tool: string) =>
  createHash("sha256").update(`${ctx.runId}:${tool}:${ctx.toolCallId}`).digest("hex");
 
// Retry-After is delay-seconds or an HTTP-date.
function serverDelayMs(res: Response): number | undefined {
  const retryAfter = res.headers.get("retry-after");
  if (retryAfter) {
    const secs = Number(retryAfter);
    if (!Number.isNaN(secs)) return secs * 1000;
    const at = Date.parse(retryAfter);
    if (!Number.isNaN(at)) return Math.max(0, at - Date.now());
  }
  const reset = res.headers.get("x-ratelimit-reset"); // UTC epoch seconds
  if (reset && res.headers.get("x-ratelimit-remaining") === "0") {
    return Math.max(0, Number(reset) * 1000 - Date.now());
  }
  return undefined;
}
 
export async function callTool(
  ctx: ToolContext,
  tool: string,
  request: (signal: AbortSignal, key: string) => Promise<Response>,
  { maxAttempts = 5, baseMs = 250, capMs = 20_000, timeoutMs = 15_000 } = {},
): Promise<unknown> {
  const key = idempotencyKey(ctx, tool);
 
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const last = attempt === maxAttempts - 1;
    let res: Response;
    try {
      res = await request(AbortSignal.timeout(timeoutMs), key);
    } catch (err) {
      // Outcome unknown. Retrying is safe only because the key is stable.
      if (last) throw new ToolCallError(`${tool} failed: ${String(err)}`, true);
      await sleep(fullJitter(attempt, baseMs, capMs));
      continue;
    }
 
    if (res.ok) return res.json();
 
    const hint = serverDelayMs(res);
    const rateLimited = res.status === 429 || (res.status === 403 && hint !== undefined);
    const retryable = rateLimited || res.status >= 500;
 
    if (!retryable || last) {
      throw new ToolCallError(`${tool} returned ${res.status}: ${await res.text()}`, retryable, res.status);
    }
    await sleep(hint ?? fullJitter(attempt, baseMs, capMs));
  }
  throw new ToolCallError(`${tool} exhausted retries`, true);
}

The request function passes the key through:

const refund = await callTool(ctx, "stripe_create_refund", (signal, key) =>
  fetch("https://api.stripe.com/v1/refunds", {
    method: "POST",
    signal,
    headers: {
      Authorization: `Bearer ${process.env.STRIPE_SECRET_KEY}`,
      "Idempotency-Key": key,
    },
    body: new URLSearchParams({ payment_intent: paymentIntentId }),
  }),
);

In production, add a total time budget per call and a circuit breaker for providers that keep failing.

Honour Retry-After and provider rate limits

Backoff is a guess. A server hint is information. RFC 9110 allows Retry-After to be an HTTP-date or a number of seconds, so handle both. Limits also differ widely between providers.

GitHub. The REST API rate limits allow 5,000 requests per hour for authenticated users and 60 unauthenticated. Secondary limits apply on top, including no more than 100 concurrent requests, 900 points per minute for REST endpoints, and 80 content-generating requests per minute or 500 per hour. A coding agent posting comments will meet that last one first. Exceeding a limit returns 403 or 429. GitHub says to wait for retry-after if present, otherwise to wait until x-ratelimit-reset when remaining is zero, otherwise to wait at least a minute, backing off exponentially if errors continue.

Shopify. Shopify uses a leaky bucket: requests fill a bucket that drains at a fixed rate, so you can burst while there is room. The REST Admin API bucket holds 40 requests per app per store and leaks 2 per second, or 400 and 20 on Shopify Plus. The X-Shopify-Shop-Api-Call-Limit header reports usage like 32/40, and throttled calls get 429 with Retry-After. The GraphQL Admin API uses query cost instead: 100 points per second on standard plans, 200 on Advanced and 1,000 on Plus, with no single query above 1,000 points.

Read these headers on every response, not just failures, and slow down before the bucket fills. Remember that limits belong to the credential. Ten agent runs for one user share one GitHub budget, so key your limiter per connected account.

Treat webhooks as at-least-once delivery

Agents also react to events, and events arrive more than once.

Stripe's webhook docs are clear. In live mode Stripe retries delivery for up to three days with exponential backoff. Order is not guaranteed. The same event can arrive more than once, so Stripe recommends logging processed event IDs and skipping repeats. Occasionally two separate Event objects are sent for one change, which you can catch using the data.object ID plus the event type. Handlers should verify the Stripe-Signature header against the raw body and return 2xx quickly, before any complex logic.

So: verify, record atomically, acknowledge, process later.

import Stripe from "stripe";
import type { Request, Response } from "express";
 
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);
 
declare const db: { insertIfAbsent(row: { eventId: string; type: string; objectId: string }): Promise<boolean> };
declare const queue: { enqueue(job: { eventId: string }): Promise<void> };
 
// Mount with express.raw({ type: "application/json" }) so the body stays raw.
export async function stripeWebhook(req: Request, res: Response) {
  let event: Stripe.Event;
  try {
    event = stripe.webhooks.constructEvent(
      req.body,
      req.headers["stripe-signature"] as string,
      process.env.STRIPE_WEBHOOK_SECRET!,
    );
  } catch {
    return res.status(400).send("Invalid signature");
  }
 
  const objectId = (event.data.object as { id?: string }).id ?? "";
  // Unique constraint on event_id makes this atomic. False means a repeat.
  const isNew = await db.insertIfAbsent({ eventId: event.id, type: event.type, objectId });
  if (isNew) await queue.enqueue({ eventId: event.id });
 
  return res.sendStatus(200); // acknowledge repeats too, so retries stop
}

Use a unique constraint, not a read then a write, or two concurrent deliveries both pass. Make the worker idempotent as well, and have it check whether it already acted on that object and event type. Since order is not guaranteed, fetch the object's current state from the API instead of trusting event sequence.

Plan for timeouts and partial failure

Every call needs a timeout, or a hung connection holds the run open indefinitely. Set one per attempt, one per call and one per run.

A timeout does not mean failure. It means unknown. Resolve it by retrying with the same key, or by looking the resource up before trying again.

Multi-step tasks add partial failure. Say an agent creates a Shopify discount, then the customer email fails. You can resume from the saved step, compensate by disabling the discount, or escalate to a human with a summary. Pick one per workflow and document it. Then tell the model the truth: a result like {"status": "unknown", "retry_safe": true} leads to better decisions than a generic error string.

Design tools the model uses safely

Schemas the model calls correctly

Many reliability bugs start with bad arguments. Anthropic's tool definition guidance calls detailed descriptions "by far the most important factor in tool performance" and suggests at least three to four sentences per tool: what it does, when to use it and when not to, what each parameter means, and its limits. It also recommends service-prefixed names such as github_list_prs, fewer but more capable tools, and lean responses with stable identifiers. input_examples helps with complex inputs, and strict: true turns on schema validation.

On top of that:

Use enums wherever the set of values is known.
Take money in minor units with a separate currency field.
Require IDs, not names. "Refund order 1042" beats "refund the Smith order".
Separate read tools from write tools so writes can be gated.
Validate every argument in code anyway.

Approval gates for irreversible actions

Payments, deletions, bulk sends and permission changes should not run on model judgement alone. Classify each tool as read, reversible write or irreversible write, and gate the last group.

A good gate pauses the run, stores the exact proposed arguments, shows a person a plain summary, and resumes only on approval. Execute exactly what was approved. Any change is a new proposal. Record who approved and when, and expire stale approvals. Gates also make rollouts safer: launch a new write tool fully gated, study the proposals, then relax the gate for low-risk cases.

Observe every call and evaluate tool use

Trace each tool call as a span tied to the agent run and the end user whose credentials it used.

The OpenTelemetry GenAI semantic conventions define an execute tool span named execute_tool {gen_ai.tool.name} with kind INTERNAL. gen_ai.operation.name and gen_ai.tool.name are required, error.type is required when the call fails, and gen_ai.tool.call.id is recommended. Arguments and results are opt-in, a sensible default for payloads full of personal data. The spec also says a span should cover the whole logical operation including automatic retries. It is still marked as in development, so pin a version. Add your own attributes for tenant, connected account, idempotency key and approval status.

Tracing shows what happened. Evals show whether the agent is improving. Build scenarios with expected behaviour: which tool, which arguments, which tools must not be called. Include failure paths such as a 429, a timeout, a duplicate webhook and an action that should hit a gate. Score selection, arguments and recovery separately, and rerun on every prompt, schema or model change.

Production-readiness checklist

None of these patterns are new. What is new is that a model decides when to call the API, which makes the guarantees around it matter more. Ambiguous writes, duplicate webhooks and throttled bursts will all happen, so design for them from the first tool.

If you would rather not rebuild this layer for every app your agent touches, that is what we are building at Hoook. Hoook is in beta and handles managed auth and prebuilt tools across 3,000+ apps, paces calls to each app's rate limits, retries with backoff and logs every call. That leaves you free to focus on the parts only you can decide: which actions need a human, and what good tool use looks like in your evals.

Reliable AI Agent Tool Calls: Retries, Idempotency and Approval Gates