Redis cache refresh: reject an expired lock’s write
Prevent an expired Redis refresh worker from overwriting a newer cache value. Check ownership and publish atomically, then test late writers and design limits.
Quick answer
Redis cache refresh needs an ownership check at publication, not only at release. If a loader outlives its lease, another loader can publish a newer value before the old one returns. Compare the lease token and write the cache in the same short script, then reject a late result that no longer owns the lease. [1][4]
On this page
Redis cache refresh needs an ownership check at publication, not only at release. If a loader outlives its lease, another loader can publish a newer value before the old one returns. Compare the lease token and write the cache in the same short script, then reject a late result that no longer owns the lease. [1][4]
Scenario: a catalogue service rebuilds a cached product summary. An old loader pauses after reading its snapshot. Its lease expires, a new loader acquires the key and publishes a newer summary, and the old loader resumes. A token-safe lease release can preserve the new loader's lock while still allowing an unconditional cache write to replace the newer summary.
This illustrative race is narrower than the whole cache-stampede problem. It asks whether an expired refresh worker can still publish. The cache is a disposable read model; it must not authorize payments, grant access or replace the authoritative product record. Those choices need a stronger consistency design.
How can a refresh make the cache older?#
The expiry bounds ownership of the lock key, not execution of the loader. Redis cannot stop an application process merely because its lease has expired. The process can continue computing and later issue another write. Redis's lock guidance specifically discusses the danger of an old client interfering with a newer owner. [1]
Follow the case as an interleaving. Worker OLD acquires a random ownership token and begins a database read. It is delayed after obtaining an earlier snapshot. The lock expires. Worker CURRENT acquires a different token, reads a later snapshot and fills the cache. OLD then returns and performs its ordinary cache write. No unsafe lock deletion is required for the cache to regress.
This is why a safe release script is necessary but insufficient. Comparing the token before deleting the lease protects the coordination key. It does not automatically protect the data key written earlier in the application's control flow. Read the code at both boundaries: publication and release.
Review the cache write even when the lease-release code already checks its token. It generates a unique token, gives the lease an expiry and releases only if the token matches. Those are useful properties. The missing question is whether the same ownership evidence is required when the candidate value becomes visible.
Keep a cache-refresh timeline in the incident record. Record when a candidate read began, whether publication was accepted and which source revision the candidate represents if the source supplies one. A high cache hit rate alone says little about whether a late writer replaced a newer snapshot.
Put ownership and publication in one operation#
Use a short Redis script that reads the lease token, compares it and writes the candidate only while that token still owns the lease. Redis executes scripts atomically with respect to other server activity. Keep source loading outside the script; slow database or HTTP work does not belong inside the server's execution window. [3][4]
The following illustrative Lua script accepts a token, serialized candidate and cache TTL. The TTL is validated before mutation. The keys must be controlled by the application, use the expected string types and share a hash slot if you use Redis Cluster. Script atomicity is not a general rollback facility for arbitrary runtime errors. [3][4][5]
-- KEYS[1]: refresh lease; KEYS[2]: cache value
-- ARGV[1]: ownership token; ARGV[2]: serialized candidate
-- ARGV[3]: positive integer cache TTL in milliseconds
local ttl = tonumber(ARGV[3])
if not ttl or ttl <= 0 or ttl ~= math.floor(ttl) then
return redis.error_reply('invalid cache TTL')
end
if redis.call('GET', KEYS[1]) ~= ARGV[1] then
return 0
end
redis.call('SET', KEYS[2], ARGV[2], 'PX', ARGV[3])
redis.call('DEL', KEYS[1])
return 1Acquire the lease with a new unpredictable token using SET with NX and an expiry. NX makes acquisition conditional on absence. The token must be unique for each acquisition, including a retry by the same worker. Reusing a token erases the distinction between the old and current ownership periods. [1][2]
For the example, use catalog:{product-42}:lease and catalog:{product-42}:value. The shared hash tag is deliberate: Redis Cluster requires a multi-key operation to address keys in the same slot. A key prefix alone does not establish that relationship. Avoid putting every product into one shared tag, which would concentrate unrelated products into the same slot. [5]
Treat the script result as part of the application contract. An accepted publication allows the caller to use its candidate under the cache's normal freshness policy. A rejected publication means the caller must discard that candidate, re-read the cache or return a bounded failure. Returning the rejected candidate to the waiting request would defeat part of the protection even though Redis itself stayed correct.
If the loader fails before publication, release its lease with the same compare-and-delete discipline. An unconditional DEL is unsafe after expiry because it can remove a later owner's lease. The Redis lock guide gives a compatible Lua pattern and documents a newer conditional-delete alternative; choose the form supported by your deployed version. [1]
What should the callers that lost the lease do?#
They need an explicit waiting and fallback policy. A short polling wait can re-read the value until an absolute deadline. A stale-value policy can serve an older approved snapshot while a refresh runs. Neither choice should cause every waiting caller to start its own unrestricted origin request when its timer expires.
That timeout fallback is a second stampede path. The lease protects the first loader, but callers that all abandon the wait together can recreate the original fan-out. Bound origin concurrency independently of the per-key lease. Use a queue, admission limit or temporary failure according to the endpoint's serving requirements.
Stale serving also needs a content decision. A catalogue description may tolerate a bounded stale interval. Stock availability or an authorization result may not. Define the allowable staleness beside the data type, and retain a hard stale deadline. A failed refresh should not move that deadline forward merely to keep responses successful.
Redis's cache-aside example demonstrates single-flight loading and emphasizes choosing a TTL and handling missing records deliberately. Its lock protects coordination; the narrower script here adds a publication condition for a loader that has lost ownership. This is a proposed refinement for the stated race, not a claim that every cache implementation requires this exact script. [6]
After acquiring the lease, check whether another path has already filled the cache before doing expensive work. This double-check can avoid unnecessary loads during a handoff. It does not replace publication admission, because the worker can still pause after that check. Keep the two mechanisms separate in the code review.
Where does this protection stop?#
It protects a candidate write against the current lease state on the Redis server executing the script. It does not establish that the candidate is the latest authoritative value. An accepted worker can read an old database snapshot while still holding a valid lease. Ownership and freshness are different properties.
Consider an authoritative update that deletes the cache while a loader is running. If that invalidation does not also change the admission condition, the loader can repopulate the cache with its older snapshot. Solving that case may require source revisions, an invalidation generation or versioned cache keys. Choose a mechanism that survives the failure model of your actual source and cache.
Similarly, the script does not fence writes to another system. A token check in Redis cannot revoke a database update the process already started. If refresh work performs side effects rather than building a disposable value, move the correctness condition to the system that accepts those effects. The durable action identity discussion describes a related need to make retry admission durable.
Failover is another boundary. Redis's lock documentation explains why asynchronous replication can matter to mutual exclusion. Do not advertise a single-primary cache lease as a strict distributed transaction or exactly-once guarantee. Decide whether occasional duplicate loads or stale cache content are acceptable under your recovery model. [1]
| Mechanism | What it controls | What still needs a decision |
|---|---|---|
| Conditional lease acquisition | Which refresh starts as owner | Loader deadline and duplicate work after expiry |
| Compare-and-publish script | Whether the current lease admits this cache write | Source revision and invalidation races |
| Token-safe release | Whether this worker may remove the lease | Cache freshness and external side effects |
| Bounded waiting policy | Caller delay and origin admission | Which responses may use stale data |
The implementation adds: another coordination operation, a script to maintain and a rejected-candidate branch in the caller. A low-traffic endpoint with cheap origin reads may not need distributed coalescing at all. A local single-flight mechanism may be enough if all relevant callers share one process. Select the machinery from the actual concurrency boundary.
Verify the paused-writer boundary#
Test the race deliberately rather than hoping a load test happens to pause a worker at the right moment. In an isolated Redis instance, let OLD acquire a short lease and keep its candidate outside Redis. Wait for the lease to expire, then let CURRENT acquire a different token and publish its value through the script.
Resume OLD and submit its candidate. The script must reject it and the cached value must remain CURRENT. Add an unconditional cache write as a negative control: it should reproduce the regression. That confirms the fixture reaches the dangerous interleaving instead of passing because the old worker never attempts publication.
Also check a normal publication while ownership is valid, a publication after expiry without a successor and a wrong-token attempt while a current lease exists. Invalid TTL input should fail before any cache mutation. Preserve both the returned result and the stored value in the test output; checking only one can miss a caller or script error.
Keep the caller branch under test too. After rejected publication, it must not return OLD as if it were accepted. Re-read the currently stored candidate, serve an explicitly allowed stale response or report that the refresh could not complete. Which choice is appropriate depends on the endpoint, but silently ignoring rejection is not a complete implementation.
These checks establish behavior in the tested single-instance setup. They do not establish failover safety, multi-region correctness or consistency with a real database. Keep those limits beside the result. A deterministic race fixture is valuable precisely because its claim is narrow enough to verify.
A rejected write is also an operational signal. Count rejected refresh publications separately from loader errors and ordinary misses. A rising rejection count can indicate a lease that is routinely shorter than the load, process pauses or competing refresh paths. Investigate that cause before extending the lease, because a longer lease also increases how long other callers wait after a worker disappears.
Frequently asked questions#
- Is increasing the lease TTL enough?
- It reduces the chance that an ordinary load exceeds the lease, but it does not prevent a paused process from returning late. Use a TTL appropriate to measured loading behavior and still decide how a candidate is admitted after ownership has changed. [1]
- Does compare-and-publish make the cache strongly consistent?
- No. It prevents this lease-loss publication race on the Redis instance executing the script. It does not prove that the loader read the latest source revision, coordinate invalidation automatically or fence effects in another system. State those limits in the cache contract.
- Can the Lua script query the database?
- Keep the loader outside Redis. A Redis script executes server-side Redis operations and blocks other server activity during its execution. Load the candidate in the application, then use a short script only to decide whether that candidate may become visible. [3][4]
- Should a rejected worker delete the lease anyway?
- No. A different token may now own it. Use token-safe release on failure and treat publication rejection as lost ownership. The worker should discard its candidate and follow the endpoint's bounded reread, stale-response or failure policy instead. [1]
