Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Distributed primitives

The redis-tower-primitives crate builds locks, leader election, expirable semaphores, countdown latches, delayed queues, block-allocated IDs, and shared rate limits on the generic RedisExecutor interface. It works with standalone, multiplexed, pooled, resilient, cluster, Sentinel, and universal clients that implement that interface. Every script is a public, line-documented constant and runs through the EVALSHA-first Script helper.

Distributed lock

DistributedLock::new(lock_key, fencing_key, ttl) requires its TTL. Acquisition atomically checks the lock, increments the fencing counter, and performs SET NX PX. A lease can release or extend only while its random owner token still matches.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::DistributedLock;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let lock = DistributedLock::new(
    "{job:17}:lock",
    "{job:17}:fence",
    Duration::from_secs(15),
)?;

if let Some(lease) = lock.acquire(&mut client).await? {
    let fencing_token = lease.fencing_token();
    // The protected service must reject tokens lower than the greatest token
    // it has already accepted.
    write_with_fence(fencing_token).await?;
    lease.release(&mut client).await?;
}
Ok(())
}
async fn write_with_fence(_token: u64) -> Result<(), Box<dyn std::error::Error>> { Ok(()) }

Lock acquisition touches two keys. In Redis Cluster they must share a hash tag, as {job:17}:lock and {job:17}:fence do.

Renewal is opt-in. LockLease::spawn_renewal consumes the lease and starts a task only at that explicit call. Its owned handle cancels and aborts renewal on drop. shutdown().await stops cleanly and returns the lease for release.

Lock failure mode

The TTL bounds ownership; it does not prove that a paused or partitioned owner has stopped. Such an owner can resume after a higher-token replacement begins work. The protected resource must enforce fencing tokens. A lost acquisition response is indeterminate, and Redis failover can roll back an unreplicated fencing increment if deployment durability permits it.

Leader election

LeaderElection::new(key, ttl, renewal_interval) requires both timing choices. No task starts during construction. A successful campaign(executor) call atomically claims the single election key and returns a Campaign containing the owned Leadership renewal handle and a separate LeadershipEvents receiver.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::{LeaderElection, LeadershipEvent};

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let candidate = MultiplexedClient::connect("127.0.0.1:6379").await?;
let election = LeaderElection::new(
    "{scheduler}:leader",
    Duration::from_secs(15),
    Duration::from_secs(5),
)?;

if let Some(campaign) = election.campaign(candidate).await? {
    let (leadership, mut events) = campaign.into_parts();
    assert_eq!(events.recv().await, Some(LeadershipEvent::Elected));
    run_one_scheduler_term().await?;
    let _outcome = leadership.abdicate().await?;
    assert_eq!(events.recv().await, Some(LeadershipEvent::Demoted));
}
Ok(())
}
async fn run_one_scheduler_term() -> Result<(), Box<dyn std::error::Error>> { Ok(()) }

LeadershipEvent::RenewalFailed means the process must stop leader-only work immediately. LeadershipEvent::Demoted reports expiration, replacement, or completed abdication. abdicate().await confirms cleanup. Dropping leadership requests the same compare-and-delete operation in the detached owned task, and the TTL bounds the key if that best-effort cleanup cannot run.

Leader-election failure mode

A TTL lease cannot prove that a paused or partitioned process has stopped. Redis failover can also lose an unreplicated campaign or renewal. Applications must make leader work idempotent or protect downstream writes with another stale-owner mechanism. A lost campaign response is indeterminate, so the caller must not infer election without a successful return and Elected event.

Expirable semaphore

ExpirableSemaphore::new(key, permit_limit, ttl) stores random permit tokens in one sorted set. Acquisition uses Redis TIME, removes expired holders, and checks capacity atomically. A SemaphorePermit supports token-checked renew and release; drop leaves recovery to the required TTL.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::ExpirableSemaphore;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let semaphore = ExpirableSemaphore::new(
    "{worker-pool}:permits",
    16,
    Duration::from_secs(30),
)?;

if let Some(permit) = semaphore.try_acquire(&mut client).await? {
    run_bounded_work().await?;
    permit.release(&mut client).await?;
}
Ok(())
}
async fn run_bounded_work() -> Result<(), Box<dyn std::error::Error>> { Ok(()) }

All users of a semaphore key must agree on its limit and TTL. There is no automatic permit-renewal task; the holder chooses whether and when to call renew.

Semaphore failure mode

Expiry recovers capacity after a crash, but a paused holder can resume after a replacement acquires the freed permit. The protected operation must tolerate stale overlap. Lost acquire, renew, and release responses are indeterminate and must not be blindly retried when duplicate bounded work would be unsafe.

Countdown latch

CountDownLatch::new(key, initial_count, ttl) requires a positive count and an absolute TTL. initialize never resets an existing latch. count_down uses DECR without allowing the count below zero. wait requires a poll interval and timeout, starts no task, and reports count-zero release, expiry, and timeout as distinct outcomes. Its timeout bounds polling sleeps but does not cancel an in-flight Redis request.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::{CountDownLatch, LatchWaitOutcome};

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let latch = CountDownLatch::new("{deploy:42}:ready", 3, Duration::from_secs(60))?;
latch.initialize(&mut client).await?;

match latch
    .wait(
        &mut client,
        Duration::from_millis(100),
        Duration::from_secs(30),
    )
    .await?
{
    LatchWaitOutcome::Released => start_traffic().await?,
    LatchWaitOutcome::Expired | LatchWaitOutcome::TimedOut { .. } => abort_deploy().await?,
}
Ok(())
}
async fn start_traffic() -> Result<(), Box<dyn std::error::Error>> { Ok(()) }
async fn abort_deploy() -> Result<(), Box<dyn std::error::Error>> { Ok(()) }

Latch failure mode

The TTL is an abandonment bound, not a successful release signal. Waiters receive Expired for a missing key and must decide how to recover. A lost countdown response is indeterminate because the decrement may already have committed; retrying can release the latch earlier than intended.

Delayed queue

DelayedQueue::new(key, retention) requires the post-deadline interval during which a message remains claimable. enqueue accepts binary payloads and a caller-selected delay, then uses Redis TIME to calculate its sorted-set score. Random member prefixes preserve duplicate payloads. claim_due atomically removes a bounded batch in deadline order and starts no poll task.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::DelayedQueue;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let queue = DelayedQueue::new(
    "{mail}:delayed",
    Duration::from_secs(60),
)?;
queue
    .enqueue(&mut client, b"message-42", Duration::from_secs(5))
    .await?;

let claim = queue.claim_due(&mut client, 32).await?;
for payload in claim.payloads() {
    deliver(payload).await?;
}
record_expired(claim.expired());
Ok(())
}
async fn deliver(_payload: &[u8]) -> Result<(), Box<dyn std::error::Error>> { Ok(()) }
fn record_expired(_count: u64) {}

The queue touches one key and is cluster-safe. Every participant sharing that key must use the same retention. The key TTL tracks the latest scheduled deadline plus retention; a claim also reports messages it actively pruned for exceeding retention. Redis may expire the whole key without a later claim, so that reported count is diagnostic rather than a complete loss counter.

Delayed-queue failure mode

Claims are deliberately at-most-once: selected messages are removed before the script returns. A lost claim response can therefore lose the batch. A lost enqueue response is indeterminate and retrying can create a duplicate. The caller owns polling, processing, retries, and any dead-letter policy; use Redis Streams or another acknowledged queue when at-least-once delivery is required.

Block-allocated ID generator

IdGenerator::new(key, block_size) reserves a contiguous positive range with exactly one INCRBY. IdBlock then yields those IDs locally without more Redis I/O or a background task.

use redis_tower::MultiplexedClient;
use redis_tower_primitives::IdGenerator;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let generator = IdGenerator::new("{orders}:id", 1_000)?;
let mut block = generator.allocate(&mut client).await?;

while let Some(id) = block.next() {
    create_order(id).await?;
}
Ok(())
}
async fn create_order(_id: u64) -> Result<(), Box<dyn std::error::Error>> { Ok(()) }

The counter intentionally has no TTL. The key must be reserved for compatible ID generators and kept for at least as long as issued IDs can matter.

ID-generator failure mode

A lost INCRBY response makes one block indeterminate. Allocating another block is uniqueness-safe but leaves a gap; callers must never infer the unknown range. Deleting, expiring, or restoring an older counter can reissue IDs, and Redis failover can roll back an increment when durability permits it. Configure persistence and replication for the required uniqueness boundary.

GCRA rate limiter

GcraRateLimiter::new(key, quota, window) requires both quota and window. Its single Lua call reads Redis TIME, prunes obsolete virtual arrivals from one sorted set, and atomically admits or denies a request. A quota of 100 over one second sustains 100 requests per second and permits an initial burst of 100.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::GcraRateLimiter;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let limiter = GcraRateLimiter::new("{tenant:7}:rate", 100, Duration::from_secs(1))?;
let decision = limiter.check(&mut client).await?;
if !decision.is_allowed() {
    tokio::time::sleep(decision.retry_after()).await;
}
Ok(())
}

The script touches one key and is cluster-safe. A shared Redis limiter enforces cross-process quota, while tower-resilience-ratelimiter can put its RateLimiterLayer in backpressure mode for process-local admission pressure; production services commonly use both.

Rate-limit failure mode

Redis failures are surfaced without a fail-open/fail-closed policy. Applications must choose that policy at their boundary. A connection loss is indeterminate: the request may have consumed a cell before the response disappeared, so an automatic retry can double-consume quota. Failover can lose recent admissions when Redis durability permits data loss.