Python Requests timeout: bound the worker lifetime

A Python Requests timeout limits connection or read inactivity; it does not bound the lifetime of the whole download. If a read-only import must stop holding a worker after a deadline, put that HTTP operation in a process the parent can terminate and wait for. Keep the result temporary until the parent accepts it. Returning a timeout to the caller and stopping the underlying work are separate requirements. ^[1][5]

What does timeout fail to bound?#

Requests' connect and read timeouts are not wall-clock deadlines. A server that continues to send small amounts of data can keep a response progressing without completing it. A caller can therefore exceed its intended operation budget while the configured read-inactivity timeout behaves exactly as designed. Omitting the timeout entirely leaves another avoidable failure path. ^[1][2]

A common repair checks a monotonic clock between chunks. That helps only when execution returns to the loop. If a read or header operation remains blocked, the check has no opportunity to run. A deadline check cannot interrupt a socket read that has not returned. urllib3 also documents that DNS resolution does not obey the socket timeout. ^[4]

Likewise, a shorter wait on a future limits the caller's wait; it does not establish that a running function was cancelled. Python's futures documentation distinguishes pending work from already-running work. Treating those states as equivalent can leave the original HTTP operation occupying resources after the application reports failure. ^[6]

Use names that preserve this distinction. read_timeout describes an inactivity rule. operation_budget describes how long the owner is prepared to keep the operation alive. result_accepted describes whether the downloaded artifact became application data. An implementation should show where each rule is enforced rather than use a generic timeout variable for all of them.

The worked case: a read-only catalog import#

Scenario: a background job fetches a JSON catalog from a fixed, trusted partner endpoint. The application should either accept a complete catalog or record a failed import. A partial file must never replace the last accepted result. The worker also needs a bounded response to a partner that sends headers or a body indefinitely slowly. This is a fictional design exercise, not a production incident.

Assume the parent already selected an approved URL and has an application-level byte limit and schema contract. It launches a child dedicated to the HTTP operation. The child can write only a temporary artifact selected for this attempt; the parent decides whether the attempt succeeded. The partner endpoint is read-only, so terminating a download does not abandon an intended remote mutation.

That last assumption is important. Killing a client does not undo a request already received by a server. Do not adapt this example into a payment, message-send or provisioning operation and infer that a timeout means nothing happened. Such a workflow needs reconciliation and an operation identity understood by the destination. Our durable action identity guide addresses that separate problem.

The following implementation uses the standard library for process ownership and Requests inside the child. Install dependencies through your normal project environment. The endpoint is an input for this trusted job, not a public arbitrary-URL fetch service. If an untrusted caller can select the URL, URL policy and network isolation become additional design requirements outside this example.

How does the child produce an incomplete result safely?#

The child writes a candidate artifact and exits successfully only after receiving a successful HTTP response, completing its body and checking the JSON shape. The parent passes a temporary output path. No child code replaces the application's accepted catalog. A failed or terminated child therefore leaves an attempt artifact that can be discarded.

This illustrative child expects a JSON array and limits the decoded bytes it writes. Its numbers are policy examples, not library recommendations or performance measurements. The inactivity timeout still has a purpose inside the larger process boundary: an unresponsive connection should fail sooner when possible. ^[1][2]

# fetch_catalog.py
import json
import sys
from pathlib import Path
import requests

url, output_path = sys.argv[1:]
byte_limit = 2_000_000
written = 0

with requests.get(
    url, stream=True, timeout=(2.0, 1.0),
    allow_redirects=False,
) as response:
    response.raise_for_status()
    if response.status_code != 200:
        raise ValueError("expected a complete catalog response")
    with Path(output_path).open("wb") as output:
        for chunk in response.iter_content(chunk_size=16_384):
            if not chunk:
                continue
            written += len(chunk)
            if written > byte_limit:
                raise ValueError("catalog exceeds byte limit")
            output.write(chunk)

catalog = json.loads(Path(output_path).read_bytes())
if not isinstance(catalog, list):
    raise ValueError("catalog must be a JSON array")

Disabling redirects makes an unexpected redirect a failed attempt rather than silently changing the download destination. The explicit status check also rejects a redirect that raise_for_status alone would not reject. A real partner contract may permit a different success status or controlled redirects; encode that policy deliberately rather than copying the example unchanged. ^[2][3]

The response context manager ensures normal exit closes the response. The byte limit prevents an accepted artifact from exceeding this job's size policy, but it is not a complete memory or decompression sandbox. Parsing still consumes memory, and process creation still consumes resources. For hostile inputs, add appropriate resource isolation and a parser suitable for the contract. ^[1]

Where does the parent enforce ownership?#

The parent calls subprocess.run with its operation budget. Python documents that a timeout there kills the child and waits for it. That is materially different from merely ceasing to await a running thread. Process creation can itself take time that the timeout cannot interrupt, so this design is an ownership boundary rather than a hard real-time guarantee. ^[5][6]

The parent below measures the budget from before launch and refuses to accept a result that arrives after its deadline. It keeps the temporary file beside the destination so acceptance can use a same-filesystem replacement. The final rename, schema checks and ordinary filesystem behavior still require your platform's own operational assumptions; this is not a durable transaction protocol for remote storage.

# import_catalog.py
import json
import os
import subprocess
import sys
import tempfile
import time
from pathlib import Path


def import_catalog(url, destination, budget_s=4.0):
    destination = Path(destination)
    deadline = time.monotonic() + budget_s
    child = Path(__file__).with_name("fetch_catalog.py")
    with tempfile.TemporaryDirectory(
        dir=destination.parent, prefix=".catalog-attempt-"
    ) as scratch:
        candidate = Path(scratch) / "catalog.json"
        remaining = deadline - time.monotonic()
        if remaining <= 0:
            raise TimeoutError("import budget expired before launch")
        try:
            subprocess.run(
                [sys.executable, str(child), url, str(candidate)],
                check=True,
                timeout=remaining,
                stdout=subprocess.DEVNULL,
                stderr=subprocess.DEVNULL,
            )
        except subprocess.TimeoutExpired as exc:
            raise TimeoutError("catalog worker exceeded budget") from exc
        if time.monotonic() >= deadline:
            raise TimeoutError("catalog finished after acceptance deadline")
        data = json.loads(candidate.read_bytes())
        if not isinstance(data, list):
            raise ValueError("candidate does not satisfy catalog contract")
        if time.monotonic() >= deadline:
            raise TimeoutError("catalog validation exceeded budget")
        os.replace(candidate, destination)
        return len(data)

The destination directory must already exist and be writable by this job. Run the parent in the same Python environment that supplies Requests to the child. The child should not create grandchildren; run owns this direct process, not an arbitrary process tree. If your implementation invokes other programs, define and test ownership for them separately. ^[5]

A child failure becomes a failed import, not an automatic retry. In this minimal example, diagnostic output is discarded to keep uncontrolled child output out of memory and logs. An operational implementation should record a bounded failure class and attempt identifier, and provide a controlled diagnostic path. Never put credentials or an entire partner payload into an exception message.

What should an adversarial verification exercise include?#

Verify completed work, stopped work and rejected artifacts separately. The happy path alone cannot establish this contract. Use a loopback test server or an isolated fixture whose behaviors you control, and keep those results separate from any claim about a real partner. The following is our proposed exercise, not a benchmark or a vendor certification.

• Return a small valid catalog promptly. The parent should accept the complete array and report its item count.
• Keep a body progressing slowly enough to avoid read inactivity. The parent should terminate the attempt and preserve the previous accepted catalog.
• Delay the response headers. The same ownership rule should apply even before a body-reading loop starts.
• Return invalid JSON, a JSON object, an oversized body or an unexpected status. Each should fail without replacing the destination.
• Repeat a failed attempt. Confirm temporary directories are cleaned and process count does not grow with each timeout.

Record elapsed time with a monotonic clock around the entire parent operation. Requests' Response.elapsed measures the interval through response headers and does not cover body consumption, so it is the wrong measure for this acceptance test. Also record whether the child has exited; latency alone cannot prove cleanup. ^[3]

Treat a timeout threshold as a policy to test under realistic operating conditions. Process startup, scheduling, termination and filesystem work can add overhead. If the surrounding service has a strict latency objective, leave margin and measure it on the deployed platform. A statement that every operating system will return at precisely the requested instant would exceed what this implementation and the documented API establish. ^[5]

When is a separate process the wrong trade-off?#

Per-download processes add startup and interpreter costs and give up convenient in-process connection reuse. They can be a reasonable containment choice for infrequent imports in a synchronous codebase. They are a poor default for every high-throughput HTTP call. In that setting, evaluate a client and runtime with cancellation behavior you can test, a bounded worker pool, or an architecture that isolates upstream work differently.

A proxy timeout may improve the caller's experience without stopping the application worker. Conversely, a child process can exit while the remote server keeps processing a request it already received. Record who stops the local process and who reconciles any remote work. The question is which resource and result your component owns, and what evidence establishes that it released them.

Budget retries at the owner of the whole operation. Starting a new operation budget for each retry defeats the parent's intended bound. This article deliberately includes no retry loop: a read-only import can report failure, preserve its previous artifact and let a separately governed job policy decide whether another attempt is appropriate.

For code that already uses thread executors, our asyncio timeout analysis explains why stopping a wait is different from stopping a thread. The subprocess approach here provides a different ownership mechanism with different costs. Choose the process boundary when its startup and cleanup costs fit the import workload.

Who owns cleanup after an unsuccessful import?#

The parent owns the temporary directory and the decision to replace the destination. That ownership should remain true when the child raises an exception, exceeds its byte policy or is killed by the process timeout. A partial candidate is useful only as tightly controlled diagnostic material; it should not become an alternate catalog that another component discovers by filename.

Also test the parent itself failing. If it crashes before accepting the candidate, the previous destination should remain the accepted artifact, while abandoned attempt directories need a bounded cleanup policy. If it crashes after replacement but before recording job success, a retry may download again. The read-only nature of this example makes that easier to reconcile, but a production import still needs an explicit job-state policy.

An atomic filesystem replacement is not the same promise as durable storage after power loss, nor does it coordinate a file with a separate database transaction. If consumers require those stronger guarantees, design that commit boundary explicitly. The short example isolates HTTP work and candidate acceptance; it does not claim to implement a complete catalog publication system.

Keep the service's concurrency limit around the parent operation. A process per attempt is still unbounded resource consumption if the scheduler admits unlimited attempts. When the partner slows down, stop accepting more work than the host can contain. A timeout can release an owned attempt, but it cannot choose your admission limit for you.

Which timeout questions remain?#

Is a Requests timeout useless?

No. Connect and read-inactivity limits are valuable failure controls and should be explicit. They simply enforce a different rule from a total operation budget. Keep them inside the chosen ownership boundary rather than presenting them as proof that all HTTP work ends on time. ^[1][2]

Can a future timeout stop the HTTP operation?

It can stop the caller's wait, but it does not prove cancellation of an already-running function. Inspect the executor's documented behavior and verify the worker exits. Pending-task cancellation and running-task termination are different capabilities in the futures API. ^[6]

Does terminating the child undo remote work?

No. The process boundary controls the local worker. It cannot retract a request already received by the destination. Use this example for the stated read-only operation; mutating remote requests need destination-aware idempotency and reconciliation before a retry is safe.

Is the subprocess budget an exact real-time deadline?

No. Python documents limitations around process creation, and termination and cleanup have operating-system costs. Measure the entire parent operation and reserve margin. The important guarantee to test is ownership and result rejection, without inventing a universal latency bound. ^[5]