Leases and locks, and why a lock is a promise about time
A lock without an expiry is a permanent outage waiting for a single crash.
What you will be able to do
- Explain a lease and the failure mode it introduces
- Use fencing tokens to stop an expired holder from writing
- Choose between pessimistic locking and conditional writes
- Recognise when a distributed lock is unnecessary
The expired holder
Worker A takes a lease for ten seconds. It then stalls — a long garbage-collection pause, a frozen VM, a saturated network. The lease expires and worker B takes it. Worker A wakes up believing it still owns the work and writes. Now two writers acted on the same resource with full confidence.
The standard fix is a fencing token: a monotonically increasing number issued with each lease. Every write to the protected resource includes the token, and the resource rejects any token lower than the highest it has seen. Worker A's write arrives with token 41 after B has written with 42, and it is refused.
A: acquire lock("job-7") -> token 41, lease 10s
A: GC pause 14s
B: acquire lock("job-7") -> token 42
B: write(resource, token=42) accepted
A: write(resource, token=41) REJECTED, 41 < 42The cheaper alternative: conditional writes
| Approach | How it works | Best for |
|---|---|---|
| Conditional update | UPDATE ... WHERE version = 7 | Single-row state transitions |
| Unique constraint | Insert fails if the key exists | Deduplication, one-time claims |
| Compare-and-set | Set value if current value matches | Counters, config flips |
| Lease + fencing | Token older writes are rejected | Long jobs with external side effects |
| Partition ownership | One consumer owns a key range | Stream processing |
The first three need no coordination service. A database with a version column can serialize state changes for a single row, and the row is the lock. In a design, reaching for Redis or ZooKeeper before checking whether a conditional write suffices reads as inexperience.
Lock scope and ordering
- Hold for milliseconds. A lock held across an HTTP call to a third party is held for an unbounded time.
- Acquire in a fixed order. Two workflows locking A then B and B then A is a deadlock by construction.
- Always set a TTL. A lock with no expiry survives exactly until the first crash.
- Do not queue behind a lock. Reject with retry-after instead of building a queue of waiting requests that will time out together.
Scheduled jobs, the classic case
A nightly job runs on five instances; you want one to actually run it. Options in increasing cost: let every instance try a conditional insert of a job-run row keyed by date (cheap, correct for one execution), use the scheduler's built-in single-flight, or hold a lease with a fencing token. The conditional insert is usually enough and needs no new infrastructure.
Worth remembering
- A lease trades correctness for liveness: the holder may still be alive after its lease expires.
- Fencing tokens make storage reject writes from a stale holder, which is the only real fix.
- Conditional writes or compare-and-set solve most lock needs without a lock service.
- Hold locks for milliseconds, never across a user request or a network call you don't control.