Lambda SQS retries: verify the partial-batch contract

When healthy SQS messages keep returning after a neighboring record fails, verify the partial-batch contract at both ends. The Lambda event source mapping must enable ReportBatchItemFailures, and the handler must return the received messageId values that failed. The response body alone does not prove the deployed mapping will interpret it as intended. ^[1]

Our worked case is a fictional invoice-import queue. A batch contains a valid invoice event, a malformed event and another valid event. The business processor can reject the malformed record without invalidating the independent imports. The goal is to retry that failed record while preserving reliable evidence about every accepted import. No AWS account execution or customer workload is implied.

This guide concentrates on acknowledgement boundaries. It does not implement invoice persistence, choose an idempotency store or claim that partial responses deliver exactly-once processing. Those decisions remain part of the application. The smaller question is already consequential: which messages does this invocation tell Lambda it could not finish?

Why are healthy messages coming back?#

Lambda polls SQS and invokes the function with a batch. Messages remain in the queue but are hidden during the visibility timeout. By default, an error processing the batch makes the whole batch available again after that timeout. A good record can therefore return because of its neighbor, even when its own business operation already completed. ^[2]

Partial responses let the function report individual failures so that successful messages need not be retried as part of that failed batch. They reduce retry amplification. They do not eliminate the possibility of duplicate delivery or the crash window between a business write and acknowledgement. The code still needs an outcome it can recognize on a later delivery. ^[1][2]

Trace the fictional batch using stable labels. Event A imports successfully. Event B fails validation. Event C imports successfully. A normal partial response reports the SQS identifier for B. If instead the function raises an exception after processing C, the invocation is a complete batch failure even if it built a correct failure list earlier. The service receives the invocation outcome, not your local variable. ^[1]

That last distinction explains why a log line saying “record failed” is weak evidence. It shows that application code noticed a problem. It does not establish that the final response included the right identifier, that the function returned normally, or that the deployed mapping enabled partial reporting. Capture those facts separately in a staging rehearsal.

Does the deployed mapping understand your response?#

Read FunctionResponseTypes from the exact deployed event source mapping.

Find the mapping attached to the exact function and queue under investigation. A function can have more than one event source. A correct template for another environment, alias or queue does not explain this invocation. Read the mapping identifier, target function, source ARN and response types together. Keep environment identifiers in the restricted run record rather than a public issue.

# Read-only; substitute the mapping UUID from your deployment record.
aws lambda get-event-source-mapping \
  --uuid '<mapping-uuid>' \
  --query '{UUID:UUID,Function:FunctionArn,Source:EventSourceArn,ResponseTypes:FunctionResponseTypes,State:State}'

The required response type is ReportBatchItemFailures. Apply a configuration correction through the team's normal deployment process, then read the mapping again. Do not treat a local infrastructure edit as proof of deployed behavior. Keep the function revision or version used by the rehearsal in the same record, so a later code rollout cannot be confused with the tested one. ^[1]

Check timeout settings separately. AWS recommends queue visibility of at least six times the function timeout, plus the batching window when configured. This allows room for processing and throttling retries; it does not make a slow downstream service reliable. Use the documented relationship as a configuration check and measure actual processing behavior in your own workload. ^[3]

Write a response whose identifiers can be checked#

The following original helper isolates a standard-queue response contract. It accepts a business callback so a test can control outcomes. It records failures by the SQS message identifier and defers remaining records when the invocation approaches its time budget. The reserve is illustrative and must cover the application's own response and cleanup work.

import logging

log = logging.getLogger(__name__)

def process_standard(records, process_one, remaining_ms, reserve_ms=2000):
    failures = []
    for index, record in enumerate(records):
        if remaining_ms() <= reserve_ms:
            failures.extend(
                {"itemIdentifier": item["messageId"]}
                for item in records[index:]
            )
            break
        try:
            process_one(record)
        except Exception:
            log.exception("Record failed", extra={"message_id": record["messageId"]})
            failures.append({"itemIdentifier": record["messageId"]})
    return {"batchItemFailures": failures}

This helper expects actual SQS-shaped records with valid identifiers. It does not sanitize a malformed synthetic event into a successful acknowledgement. In a Lambda adapter, pass context.get_remaining_time_in_millis as the clock callback and the application's idempotent record processor as the business callback. Exercise that adapter as well as the helper before rollout.

The broad exception handler is a deliberate per-record policy for this example. It keeps failed records eligible for retry and logs the exception without logging the invoice body. In a real application, classify permanent validation failures and retryable dependency failures, and decide how each reaches a review or dead-letter path. Do not silently drop a permanently invalid record just because another retry seems unlikely to help.

For a shared dependency outage, continuing through the entire batch may only repeat the same failed call. A circuit breaker or an early-defer policy can report all remaining records without attempting each one. That is different from returning an empty failure list. The latter claims success, while deferring the remaining identifiers preserves work for recovery.

records = [{"messageId": key} for key in ("a", "b", "c")]
seen = []

def process_one(record):
    seen.append(record["messageId"])
    if record["messageId"] == "b":
        raise ValueError("illustrative invalid invoice")

result = process_standard(records, process_one, lambda: 10000)
assert seen == ["a", "b", "c"]
assert result == {"batchItemFailures": [{"itemIdentifier": "b"}]}

The assertions establish helper behavior on a synthetic event. They cannot establish AWS polling, deletion or visibility behavior. Add negative fixtures for an empty identifier, the wrong identifier field and a business identifier accidentally used in place of messageId. Keep those malformed-response experiments out of production queues. Use the Developer Guide as the service contract. ^[1]

What changes for FIFO and deadline failures?#

The helper above continues after a failure, which suits independent records in the illustrative standard queue. AWS's FIFO guidance is different: stop after the first failure and report failed and unprocessed messages. Processing later messages after their predecessor failed can violate the order the application was relying on. Do not reuse the standard helper unchanged. ^[1][4]

A deadline check before a record is useful but limited. It cannot interrupt a blocking call that ignores the remaining budget. Set appropriate downstream timeouts and leave enough room to build the response. If the runtime times out before returning, the intended partial response never reaches Lambda. Rehearse that branch, including records completed before the timeout. ^[1]

Maintained libraries reduce the amount of custom code needed for queue-specific behavior. Powertools provides batch processors and distinguishes SQS from stream checkpoint semantics. Adopting it still requires a deployed response-type setting, a correct record callback and tests for the application's failure policy. A library name in a dependency file is not evidence that the adapter uses it correctly. ^[5]

How do you prove recovery without trusting a green invocation?#

Count accepted, failed and deferred records separately from successful Lambda invocations.

After partial reporting is enabled, an invocation can return successfully while reporting record failures. AWS therefore recommends code-level tracking for overall application outcomes. Track useful business counts and failed-record counts alongside queue age and dead-letter behavior. A falling Lambda error count alone can describe a response-shape change rather than improved business processing. ^[4]

Rehearse a malformed record between successful records, then verify the queue and the durable business results.

In a dedicated staging queue, send known event identifiers and record their durable business outcomes. Confirm the deployed mapping before the rehearsal. Observe which failed messages return after visibility and which accepted operations remain single durable outcomes under redelivery. Do not infer deletion from a single approximate queue-depth sample; use the event identities and processing record to interpret it.

Rehearse a crash after a successful business operation but before the handler returns. The subsequent delivery must recognize that completed operation. A unique business event key or other durable idempotency design belongs at that boundary. The SQS receipt handle describes a delivery attempt; it is not a substitute for the business action's identity. ^[2][4]

Keep poison-message handling explicit. A source queue redrive policy and a monitored dead-letter queue bound repeated failed processing. Choose the receive-count policy from expected failure and recovery behavior, then give the dead-letter queue an owner and a replay procedure. A message moved out of the source queue is contained work, not completed work. ^[3][4]

The release record should join the mapping UUID, function revision, received record identifiers, final response and durable business outcomes. If any part is missing, describe the test as partial. A synthetic helper test can be valuable without being called a queue integration test. A staging queue rehearsal can prove service behavior without being called a production incident.

For the fictional imports, success means the valid invoices have durable outcomes, the malformed event remains observable on its failure path, and a replay cannot create another invoice for the same business event. The partial-batch response helps preserve that separation. Its job is to report unfinished messages accurately, not to certify the entire workflow.

Distinguish deferred work from failed work#

The deadline branch in the helper reports unattempted records as failures to the service because they remain unfinished. The application's metrics can still distinguish those deferred records from records whose business callback raised an error. That distinction explains whether the remedy is a smaller batch, a faster dependency or repair of an invalid event, without changing the acknowledgement contract.

Keep the failure record useful for replay. Preserve the message identity, business event identity, function revision and error classification in an appropriately restricted log or processing ledger. Avoid copying the entire payload into every exception record. A later redrive should be able to identify the original business operation and its completed state, rather than creating a new operation merely because delivery occurred again.

Finally, test how the response is serialized by the actual Lambda adapter. A helper returning a dictionary is insufficient if another wrapper converts it to an API Gateway response or discards it. This is an SQS-triggered handler, so HTTP status codes and an HTTP response body wrapper are not its acknowledgement protocol. Keep the adapter test next to the synthetic batch fixture, and reserve the service-level claim for a staging queue rehearsal. ^[1]

Frequently asked questions#

Does returning batchItemFailures enable the feature?

No. The event source mapping must also declare ReportBatchItemFailures. Inspect the deployed mapping rather than relying only on an infrastructure template. The response and the mapping form one service contract; a handler unit test cannot establish both halves. ^[1]

Which identifier belongs in itemIdentifier?

Use the messageId from the received SQS record. Do not substitute the receipt handle, a business event identifier or a generated UUID. Business idempotency may use a different stable key, but the partial-batch response must identify messages in the batch Lambda delivered. ^[1]

Can the same handler be used unchanged for FIFO?

Do not assume so. AWS recommends stopping after the first failure and reporting failed and unprocessed messages to preserve ordering. A processor that continues after a failure in a standard queue needs a deliberate FIFO policy before reuse. ^[1][4]

Are successful records now processed exactly once?

No. Lambda SQS delivery can still produce duplicates, and partial responses do not atomically couple an external side effect with acknowledgement. Make each business operation safe to repeat and verify its durable result independently of the invocation response. ^[2][5]