AI rate limits: give retries one owner and a deadline

Give model retries one owner and an overall deadline. If the SDK retries inside a worker retry loop, a single logical request can produce many physical provider calls. AWS recommends a single retry point for suitable service operations, and the Claude SDK exposes controls to disable its own retries when the application takes responsibility ^[1][2].

How many requests does one failure actually send?#

Start by drawing the call path: queue delivery, worker attempt, application loop, SDK request. Record which layers can repeat work and whether their counters include the original attempt. The Python SDK documentation says selected failures are retried twice by default ^[4]. A worker counter that increments only when the SDK returns does not count those internal attempts.

Scenario: a support-draft worker allows three outer attempts, while its SDK allows the original call plus two retries. If every inner attempt fails, the arrangement can issue nine provider requests for one logical draft. That is arithmetic for an illustrative configuration, not a measured incident. Adding a queue redelivery creates another opportunity to repeat the same work.

The extra calls can matter even when they never produce an answer. They occupy connections, extend the time a user waits and add traffic while the provider is limiting it. AWS’s retry analysis explains why independent layers amplify overload ^[1]. Longer sleeps spread the calls out. They still leave several layers able to send the next request.

Make logical request ID, worker attempt and provider attempt separate fields. Record the deadline, provider request ID when available, failure class, requested delay and final disposition. A log that says only “retrying” hides whether the SDK already sent another request.

Pick the owner before picking the backoff#

For a simple synchronous feature, allowing the SDK to handle its documented retries may be enough. In that design, the worker should not add another automatic model-call loop. For a queued feature that needs deadline-aware scheduling, let the application own admission and disable SDK retries explicitly ^[2][4]. Both are defensible; running them independently is the configuration to account for.

# Integration fragment, reviewed against the SDK docs; not executed here.
from anthropic import AsyncAnthropic

# The application now owns retries, classification and admission.
client = AsyncAnthropic(max_retries=0)

This setting transfers responsibility; it does not implement a retry policy. The application must decide which failures can be retried, stop after its attempt limit and preserve cancellation. Use the SDK’s typed errors and structured response information where available. Do not classify a failure by searching a human-readable log for a convenient phrase ^[2][4].

Give the support-draft operation an explicit result contract. It can return a completed draft, defer work to an approved queue or report that generation could not finish. It does not send the draft to a customer. That separation means a model retry cannot itself duplicate the customer-facing side effect. Our AI tool-permissions article addresses the authorization boundary around later actions.

Which rate-limit responses should be retried?#

Do not treat every rate-limit response as an instruction to sleep briefly and repeat. Anthropic distinguishes ordinary rate limits from usage-tier spend caps. Its documentation describes a spend-cap response with a structured error code and no Retry-After header; retrying does not restore access ^[3]. Classify that condition before entering the delay policy.

When a transient response supplies a documented delay, honor it. For the Claude API, Retry-After expresses seconds to wait ^[3]. A local maximum backoff is a cap on your own exponential delay, not permission to shorten the provider’s instruction. If the required wait does not fit the operation’s deadline, return a deferred or failed result instead of retrying early.

Our conservative example declines an unparseable delay and lets an operator or a later queue policy decide what happens. It does not assume that a missing header proves a spend cap: other retryable failures can lack that header. Error type and structured details establish classification; the header contributes scheduling information.

A small policy that can refuse the next attempt#

The helper below decides whether another attempt can be admitted. It does not call an API, sleep, acquire a shared permit or classify errors. Its inputs make those boundaries visible. Pass the number of physical attempts already used, an overall monotonic deadline and a random value for jitter. The small defaults are illustrative policy values to tune against your own latency contract.

import math


def retry_delay(*, attempts_used, max_attempts, now, deadline,
                retry_after=None, jitter_unit=0.5,
                base=0.5, cap=8.0, min_attempt_time=0.25):
    """Return a delay, or None to refuse another attempt.

    attempts_used includes the original request. All times are seconds.
    now and deadline must use the same monotonic clock.
    Call only after the failure has been classified as retryable.
    """
    if not (1 <= attempts_used < max_attempts):
        return None
    numbers = (now, deadline, jitter_unit, base, cap, min_attempt_time)
    if not all(math.isfinite(x) for x in numbers):
        raise ValueError("policy values must be finite")
    if not (0 <= jitter_unit <= 1 and base > 0 and cap > 0
            and min_attempt_time > 0):
        raise ValueError("invalid retry policy")
    if retry_after is None:
        server_delay = 0.0
    else:
        try:
            server_delay = float(retry_after)
        except (TypeError, ValueError, OverflowError):
            return None
        if not math.isfinite(server_delay) or server_delay < 0:
            return None
    # A capped local backoff never shortens a provider delay.
    ceiling = min(base, cap)
    for _ in range(attempts_used - 1):
        ceiling = min(ceiling * 2, cap)
        if ceiling == cap:
            break
    delay = server_delay + jitter_unit * ceiling
    if now + delay + min_attempt_time >= deadline:
        return None
    return delay

The provider delay is added to a jittered local backoff, so this helper never shortens a valid instruction. A very long hint simply fails the deadline check. The minimum attempt time reserves some room after sleeping; it is a planning allowance, not a promise that the next inference will complete in that period.

The monotonic clock matters because the budget describes elapsed waiting time. Use the same clock when producing now and deadline. A wall-clock timestamp from another service is not interchangeable with a local event-loop deadline. At the caller, keep the same operation deadline across queue admission, waiting and the eventual request.

Admission is not enforcement of an in-flight request. Python supplies timeout context managers and an absolute event-loop deadline through asyncio.timeout_at ^[5]. Place the operation inside that boundary and retain the SDK’s own request timeout as appropriate. Cancellation still depends on the awaited work cooperating; do not detach the provider call and let it continue after the caller has stopped waiting.

# Python 3.11+ integration sketch; no provider request was run.
import asyncio

async def make_one_attempt(client, request, deadline):
    # deadline was computed earlier with this loop's time().
    async with asyncio.timeout_at(deadline):
        return await client.messages.create(**request)

# The same deadline must cover any admission wait and backoff sleep.
# An outer loop must also enforce its physical-attempt limit.

What do the refusal cases establish?#

Test refusal before testing a happy retry. For the support-draft case, a provider delay longer than the remaining deadline must not cause an earlier request. Exhausted attempts must not gain a fresh budget because a wrapper was called again. Invalid or nonfinite delay values must not enter a sleep operation. These are application policy assertions, separate from provider availability.

• Inject a valid provider delay and confirm the chosen wait is at least that long.
• Exhaust the physical-attempt allowance and confirm the helper returns no next delay.
• Place the deadline before the proposed wait plus attempt allowance; confirm admission is refused.
• Supply malformed, negative and nonfinite hints; confirm they do not produce a retry.
• Cancel while waiting, then inspect the caller and worker for orphaned operations.
• With several replicas, confirm a shared admission mechanism limits aggregate retries as intended.

The local verification for this article executes deterministic tests of the original helper only. It does not establish SDK integration, remote cancellation, shared admission or performance under a provider outage. Those remain acceptance tests for the application environment. Record those untested boundaries beside the helper’s test result.

Finally, separate inference recovery from action recovery. A transport timeout does not prove a remote operation had no effect; AWS’s guidance ties safe retries to idempotency ^[6]. If generated output can lead to an external change, persist the approved action and its identity before performing it. Our transactional outbox guide develops that delivery problem beyond the model call.

Questions for the retry owner#

Should I always set max_retries to zero?

No. Disable SDK retries when another layer deliberately owns retry scheduling and has the required classification, budget and timeout behavior ^[2][4]. For a simpler application, SDK-owned retries can be appropriate. The important part is accounting for all physical requests and avoiding an unexamined second loop.

Can I cap a provider delay to keep the page responsive?

Do not shorten a documented minimum wait and then send another request early ^[3]. If the wait cannot fit the user-facing deadline, stop, return a clear failure or defer the operation through an approved asynchronous contract. Responsiveness comes from ending the wait, not ignoring the throttle.

Does backoff solve a usage-tier spend cap?

No. Anthropic documents a usage-tier spend-cap response that continues failing until access resumes ^[3]. Detect the structured condition and stop automatic retries. Do not infer this class solely from a missing delay header; inspect the provider’s error details.

Can I replay a model stream after receiving partial output?

Treat that as a separate recovery decision. The caller may already have consumed text or initiated later work, so replay can duplicate or contradict visible output. Keep external actions idempotent and record their identities ^[6]. A retry before any visible output has a different contract.