Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

redis-tower documentation

redis-tower is a Redis client for Rust where every connection is a Tower Service. Commands have concrete request and response types, concurrent work can be auto-pipelined, and operational policy composes through Tower layers.

If you are evaluating or adopting the project, start here:

  • Migrating from redis-rs maps the most common redis-rs idioms to redis-tower.
  • Migrating from Fred covers client construction, command execution, pipelines, transactions, topology, pub/sub, reconnection, and shutdown.
  • Production tuning explains client selection, pipeline and pool sizing, timeouts, backpressure, reconnects, observability, and graceful shutdown.
  • Cloud and rotating credentials covers AWS ElastiCache IAM, Microsoft Entra ID, reconnect-time refresh, proactive reauthentication, and token-handling boundaries.
  • Serverless and scale-to-zero covers deferred connection, cold-start lifecycle semantics, invocation deadlines, and shutdown.
  • Pool health probing design records why active probing and idle reaping are pool-native, explicitly spawned tasks.
  • Client-side caching covers cloneable cached clients, tracking modes, invalidation races, failure behavior, and cache metrics.
  • Distributed primitives covers fenced locks, leader election, expirable semaphores, countdown latches, delayed queues, block-allocated IDs, Redis-time GCRA rate limiting, cluster keys, and failure behavior.
  • The command cookbook maps typed command categories to response shapes, binary data, pipelines and transactions, stream ownership, raw/module replies, dedicated sessions, and topology entry points.
  • The command capability ledger separates command-name availability, typed semantics, wire support, and routing/session evidence at a pinned Redis documentation revision.
  • The client comparison records pinned, primary-source contracts for Rust and cross-language Redis clients.
  • The test conformance report maps generated test inventory, topology and server matrices, and destructive fault coverage to their source and CI evidence.
  • Differential testing records the MCP-derived semantic corpus against a pinned redis-rs oracle and its normalization rules.
  • Binary data and typed arguments explains byte ownership, exact Cluster routing, response types, and the per-family compatibility inventory.
  • RESP codec support and validation defines accepted and rejected wire forms, the fragmentation oracle, reviewed fuzz seeds, and mutation triage.
  • The release process records the manual prepare/publish split, workspace publish order, validation gates, and post-publish verification.

For installation, a quick API overview, and examples across the main modules, see the repository README. The typed API reference on docs.rs provides crate-level tours, module guides, and compiled examples alongside every public type and method.

Local preview

Install mdBook, then run:

mdbook serve --open

For the same non-interactive checks used in CI:

mdbook build
python3 scripts/check_docs_links.py

The generated site is written under target/mdbook and is never checked in.

Migrating from redis-rs

This guide maps redis-rs 1.7 to the current redis-tower API. Both clients provide typed conveniences, raw commands, multiplexing, pipelines, transactions, Cluster, Sentinel, TLS, pub/sub, and RESP3. The migration is primarily about choosing explicit connection and response contracts, not replacing an “untyped” client with a typed one.

The migration in one table

redis-rs 1.7redis-tower
MultiplexedConnectionMultiplexedClient
ConnectionManagerFactory-backed MultiplexedClient, or serialized ResilientRedisClient
AsyncCommands / AsyncTypedCommands methodsCommand values such as Get and Set, passed to execute
Caller-selected FromRedisValue responseCommand-defined response type; RedisValueExt converts at the edge
CmdRawCommand, returning Frame, or TypedRawCommand<T>
Pipeline / atomic pipelinePipeline / Transaction
Async pub/sub connectionDedicated PubSubConnection
Async Cluster connectionMultiplexedClusterClient
Sentinel clientMultiplexedSentinelClient
RedisError + ErrorKindRedisError + classification helpers

Dependencies

The versions below are the released APIs used by this guide. Choose only the redis-tower topology crates the application needs:

[dependencies]
# Before
redis = { version = "=1.7.0", features = ["tokio-comp", "cluster-async", "connection-manager"] }

# After
redis-tower = "0.1.4"
redis-tower-cluster = "0.1.4"  # only for Redis Cluster
redis-tower-sentinel = "0.1.4" # only for Sentinel
bytes = "1"
tokio-stream = "0.1"           # only for stream APIs such as pub/sub

For TLS, enable tls-rustls (recommended) or tls-native-tls on the facade and on each topology crate that opens TLS connections; neither backend is enabled by the facade’s default feature set. Redis Stack command groups can be selected individually or through the facade’s default commands-stack feature.

TlsConfig::with_root_ca_pem replaces the default trust roots with every CA in the supplied PEM bundle for either backend. Rustls loads native roots once per process, caching the result or loading error; an empty native store falls back to the webpki bundle, while loading failures with no usable roots return the loader diagnostics. Restart the process to pick up changes to native roots. danger_accept_invalid_hostnames(true) works on both backends and matches redis-rs’s hostname-only bypass: the certificate must still chain to a trusted root, but its names need not match the connected host or IP. That is the usual answer for a cluster whose nodes report IPs the certificate does not name. danger_accept_invalid_certs(true) disables all certificate verification. Both are dangerous settings. Caller-built ClientConfig / TlsConnector configurations retain their own verification and trust policy.

The standalone registry compatibility harness is compiled in CI against these exact crates.io versions. It is an independent workspace, so local path crates cannot mask a published-API mismatch. The larger live oracle is documented in Differential testing.

Connect and share

// redis-rs
let redis_rs = redis::Client::open("redis://127.0.0.1:6379/0")?;
let mut connection = redis_rs.get_multiplexed_async_connection().await?;

// redis-tower
use redis_tower::MultiplexedClient;
let client = MultiplexedClient::connect_url("redis://127.0.0.1:6379/0").await?;

Both handles can be cloned and shared across Tokio tasks. Do not add an Arc<Mutex<_>> around MultiplexedClient; its worker already serializes and auto-pipelines concurrent requests.

Choose a different redis-tower connection when the ownership contract differs:

  • RedisConnection exclusively owns one socket.
  • RedisClient is a simple serialized shared connection.
  • ConnectionPool<RedisConnection> owns multiple independent sockets.
  • A factory-backed MultiplexedClient reconnects without giving up concurrent multiplexing.
  • ResilientRedisClient is a simpler reconnecting, mutex-serialized client.

Protocol, URLs, and authentication

The defaults differ. A normal redis-rs 1.7 URL selects RESP2; add ?protocol=resp3 to request RESP3. redis-tower’s default ProtocolVersion::Auto tries HELLO 3 and falls back to RESP2 when RESP3 is not available. If wire shape matters during a migration, pin both clients instead of comparing different protocols:

// redis-rs RESP2
let redis_rs = redis::Client::open("redis://127.0.0.1:6379/?protocol=resp2")?;

// redis-tower RESP2. The released URL parser does not consume `protocol=`;
// select the protocol with ConnectionConfig.
use redis_tower::{ConnectionConfig, MultiplexedClient, ProtocolVersion};
let config = ConnectionConfig::new().with_protocol(ProtocolVersion::Resp2);
let tower = MultiplexedClient::connect_url_with_connection_config(
    "redis://127.0.0.1:6379/",
    &config,
).await?;

// Use ProtocolVersion::Resp3 to force RESP3 without fallback.

redis://user:password@host/db and rediss://... carry ACL credentials, database selection, and TLS. Percent-encode URL-special bytes in usernames and passwords. redis-tower authenticates and selects the database as part of setup; a reconnecting URL factory repeats those steps on every new socket.

redis-tower’s Unix URL parser accepts authentication, database, and protocol query keys:

redis+unix:///run/redis.sock?user=app&pass=secret&db=1&protocol=resp3

For rotating tokens, replace static URL credentials with redis-tower’s CredentialProvider. CredentialConnectionFactory::from_url preserves the URL transport, database, and protocol while taking authentication from the provider. Its with_setup_timeout covers provider lookup through final session setup. SharedCredentialProvider gives type-erased hosts one cloneable cache owner for ordinary, Cluster, and dedicated connections.

redis-rs’s streaming provider is not a signal to inject AUTH into every socket state. redis-tower owners expose proactive reauthentication only where they serialize it safely. Pub/Sub reconnects and replays confirmed subscriptions; MONITOR terminates and reopens with an explicit observation gap; transactions and blocking operations move to fresh credentials only at a new operation boundary. See Cloud and rotating credentials.

Typed commands and response shapes

redis-rs command traits select a conversion from the type requested by the caller. redis-tower command values have one public response type:

// redis-rs
use redis::AsyncCommands;
let _: () = connection.set("key", "value").await?;
let value: Option<String> = connection.get("key").await?;

// redis-tower
use redis_tower::commands::{Get, Set};
client.execute(Set::new("key", "value")).await?;
let value: Option<bytes::Bytes> = client.execute(Get::new("key")).await?;

String-shaped Redis data stays binary-safe as Bytes. Convert only at an application boundary:

use redis_tower::RedisValueExt;
let value: String = client.execute(Get::new("key")).await?.parse_into()?;

SET is one place where choosing a return conversion is not enough. Redis uses OK, null, and the previous bulk value differently depending on NX, XX, and GET. The base redis-tower Set keeps its original Option<Bytes> response so existing code continues to compile, but that compatibility shape maps both OK and a rejected conditional write to None. Use the additive outcome mode when the decision matters:

use redis_tower::commands::{Set, SetPreviousValue, SetStatus};

let outcome = client
    .execute(Set::new("lease", "owner-a").nx().get().with_outcome())
    .await?;

match (outcome.status, outcome.previous) {
    (SetStatus::Applied, SetPreviousValue::Missing) => {
        println!("lease acquired from an absent key");
    }
    (SetStatus::NotApplied, SetPreviousValue::Value(owner)) => {
        println!("lease already held by {owner:?}");
    }
    other => println!("other SET outcome: {other:?}"),
}

Call with_outcome() last. Without GET, the status remains unambiguous but the previous value is SetPreviousValue::NotRequested; with GET, missing, empty, and arbitrary binary previous values remain distinct.

Do not assume identical public values just because both clients decoded the same RESP frame. RESP2 may represent a map as a flat array; RESP3 has map, set, boolean, and verbatim-string types. redis-rs Value, redis-tower Frame, and a typed command response deliberately expose different layers. Compare the documented command result, normalizing only semantics Redis declares unordered or protocol-dependent.

Binary arguments

The published 0.1.4 facade depends on redis-tower-commands 0.1.3, where typed builders take keys and values as binary-safe CommandArg inputs. Byte slices, arrays, Vec<u8>, and Bytes pass through unchanged, so opaque data does not need the raw-command escape hatch:

use redis_tower::commands::{Get, Set};

let key = b"binary:\xff".as_slice();
let payload = vec![0x00, 0xfe, 0xff];
client.execute(Set::new(key, payload)).await?;
let value: Option<bytes::Bytes> = client.execute(Get::new(key)).await?;

RawCommand::arg remains binary-safe for commands without a typed builder. Do not force opaque data through UTF-8. The per-family inventory and ownership rules are tracked in Binary data and typed arguments.

Raw commands

// redis-rs
let value: redis::Value = redis::cmd("MYCOMMAND")
    .arg("key")
    .query_async(&mut connection)
    .await?;

// redis-tower
use redis_tower::commands::RawCommand;
let frame = client
    .execute(RawCommand::new("MYCOMMAND").arg("key"))
    .await?;

RawCommand returns a Frame; validate its shape at the boundary. A typed builder is preferable when available because it also carries key-routing, blocking, idempotency, and response-decoding metadata.

Pipelines and transactions

Concurrent MultiplexedClient calls are automatically batched. Use an explicit Pipeline when one task needs an ordered batch on one connection:

use redis_tower::{Pipeline, RedisConnection};
use redis_tower::commands::{Get, Incr};

let mut connection = RedisConnection::connect("127.0.0.1:6379").await?;
let mut replies = Pipeline::new()
    .push(Incr::new("counter"))
    .push(Get::new("counter"))
    .execute(&mut connection)
    .await?;
let count: i64 = replies.take(0)?;
let observed: Option<bytes::Bytes> = replies.take(1)?;

A pipeline is not atomic. Replace redis-rs’s atomic pipeline with Transaction; its result explicitly distinguishes Committed from Aborted after a WATCH conflict:

use redis_tower::{Transaction, TransactionResult};

match Transaction::new()
    .watch(["counter"])
    .push(Incr::new("counter"))
    .execute(&mut connection)
    .await?
{
    TransactionResult::Committed(mut replies) => {
        let count: i64 = replies.take(0)?;
        let _ = count;
    }
    TransactionResult::Aborted => { /* rebuild and retry if appropriate */ }
}

For read-compute-write WATCH loops, keep the entire loop on a dedicated RedisConnection (or another connection instance that the application owns exclusively for the full loop).

Dedicated and stateful sessions

Do not move every operation onto the default multiplexed client. These APIs need an exclusive or purpose-built session:

  • Run BLPOP, blocking XREAD, and other blocking commands on a dedicated RedisConnection or a pool with enough independent connections. One blocking request would otherwise stop the shared pipeline worker.
  • Construct PubSubConnection from a fresh RedisConnection; it owns subscription state, and reconnect_with can restore confirmed subscriptions after the application supplies a replacement connection.
  • Construct MonitorStream from a fresh RedisConnection; entering MONITOR permanently changes that socket’s mode until it is closed.
  • Keep manual connection-state sequences and read-compute-write WATCH loops on one exclusive connection.
use redis_tower::{PubSubConnection, RedisConnection};
use tokio_stream::StreamExt;

let connection = RedisConnection::connect_url("redis://127.0.0.1:6379").await?;
let mut pubsub = PubSubConnection::from_connection(connection)?;
pubsub.subscribe(&["events"]).await?;
while let Some(message) = pubsub.next().await {
    let message = message?;
    let _ = message;
}

Cluster and Sentinel

use redis_tower_cluster::MultiplexedClusterClient;
let cluster = MultiplexedClusterClient::connect_url(
    "rediss://default:secret@seed.example:6379",
).await?;

use redis_tower_sentinel::MultiplexedSentinelClient;
let sentinel = MultiplexedSentinelClient::connect(
    &["127.0.0.1:26379"],
    "mymaster",
).await?;

Cluster handles MOVED/ASK routing and topology refresh. Sentinel separates Sentinel-hop and Redis-node credentials/TLS in its builder and verifies the discovered role. Cluster pub/sub makes ownership explicit: fixed-node regular subscriptions and slot-following sharded subscriptions are different APIs.

Reconnection, retry, and unknown execution

redis-rs ConnectionManager reconnects in the background: the command that observes the dropped connection errors, while later commands wait for the new connection. redis-tower separates three policies:

  1. a connection factory recreates and reconfigures the socket;
  2. reconnect backoff controls when another socket is attempted;
  3. retry or offline-queue policy decides whether an application command is eligible to be sent again.

A write whose request reached Redis but whose reply was lost has unknown execution. redis-tower does not blindly replay a non-idempotent command such as INCR. Applications that retry such work need an idempotency key or a transaction/script that makes duplication safe.

Tower middleware

redis-tower clients implement tower::Service, so timeout, tracing, metrics, circuit-breaker, concurrency, and application-specific policy can compose at a stable request boundary. The built-in layers are documented on the redis-tower crate. Connection setup timeout, per-command deadline, Redis’s own blocking timeout, and an end-to-end request deadline are separate controls; preserve that distinction during migration.

Lessons from an MCP integration

A downstream MCP integration motivated the public differential corpus. The reusable lessons are backed by that repository-local case ledger and executable tests:

  • keep redis-rs as an independent oracle instead of sharing redis-tower’s serializers or response conversion;
  • define an application response boundary instead of leaking either client’s raw protocol enum through a public schema;
  • distinguish top-level server errors from nested per-entry errors;
  • exercise RESP2 and RESP3 rather than assuming one public response shape;
  • test Cluster routing, blocking sessions, binary/null/error values, cancellation, and lost replies as separate contracts.

The corpus pins redis-rs 1.7.0 and records the intentionally narrow normalizations used for each case.

Migrating from Fred

This guide maps common fred 10.1 idioms to redis-tower. Fred and redis-tower are both asynchronous Redis clients with automatic pipelining, cluster and Sentinel support, TLS, pub/sub, transactions, and reconnection building blocks. The important difference is API shape: Fred exposes command interface traits with caller-selected response conversion, while redis-tower sends typed command values through clients that implement Tower Service.

The Fred snippets are intentionally versioned to 10.1. Fred has changed its builder and client names across major releases, so first translate older Fred code to the equivalent 10.1 concept, then apply the mapping here. The Fred crate docs and Fred examples are the source of truth for Fred-specific behavior.

The migration in one table

Fredredis-tower
Builder + Client::init()A client connect / connect_url constructor
Client for concurrent commandsMultiplexedClient
Interface methods such as get and setCommand values such as Get and Set, passed to execute
Caller-selected FromValue responseCommand-defined response type; RedisValueExt converts at the edge
client.pipeline()Automatic batching on MultiplexedClient; Pipeline for an explicit batch on an exclusive connection
Pool / ExclusivePoolConnectionPool<S>; use RedisConnection when one operation needs exclusive ownership
ReconnectPolicy on a builderFactory-backed MultiplexedClient or ResilientRedisClient with ReconnectConfig
Offline command buffering during reconnectOpt-in OfflineQueueConfig on ResilientRedisClient, for idempotent typed commands only
SubscriberClientDedicated PubSubConnection stream
client.multi()Transaction / transaction_with_retries
custom(cmd!(...))RawCommand
Client::quit() + connection taskMultiplexedClient::shutdown(); dropping an exclusive RedisConnection closes it

Dependencies and imports

A Fred application often starts with a broad prelude and opts into interface features:

[dependencies]
fred = { version = "10", features = ["i-all", "enable-rustls"] }

The redis-tower facade always includes the core typed commands. Its default commands-stack feature adds every Redis Stack command group; the example below disables defaults so Stack groups and TLS can be selected explicitly. Cluster, Sentinel, sync, modules, and a runtime-selected universal client are separate crates:

[dependencies]
redis-tower = { version = "0.1", default-features = false, features = ["tls-rustls"] }
redis-tower-cluster = { version = "0.1", features = ["tls-rustls"] }
redis-tower-sentinel = { version = "0.1", features = ["tls-rustls"] }
bytes = "1"
tokio-stream = "0.1"

# Add only when application code must choose a topology at runtime.
redis-tower-client = "0.1"

Add redis-tower features such as commands-json, commands-search, or commands-stack only when the application uses those Redis Stack command groups. This keeps compile time and the public command surface intentional.

Instead of importing Fred interface traits through fred::prelude::*, import the command builders you use (or the complete command prelude):

use redis_tower::{MultiplexedClient, RedisValueExt};
use redis_tower::commands::{Get, Set};
// For broad command-heavy modules: use redis_tower::commands::*;

Connect and share a client

Fred separates construction from initialization:

use fred::prelude::*;

let config = Config::from_url("redis://default:secret@127.0.0.1:6379/0")?;
let client = Builder::from_config(config).build()?;
let connection_task = client.init().await?;

redis-tower constructors return a connected, usable client:

use redis_tower::MultiplexedClient;

let client = MultiplexedClient::connect_url(
    "redis://default:secret@127.0.0.1:6379/0",
).await?;

MultiplexedClient is the closest default replacement for Fred’s cheaply cloneable Client: clones share one background worker and one auto-pipelined connection, and execute(&self, ...) is safe from many Tokio tasks. Do not put it behind Arc<Mutex<_>>.

There are two lifecycle differences to account for:

  1. MultiplexedClient::connect_url owns its worker internally; there is no separate connection task to retain or await.
  2. A plain connect client does not automatically reconnect. Choose a factory-backed client before replacing Fred code that relies on its reconnect policy. See Reconnection.

Execute typed commands

Fred places each Redis command on an interface trait and converts the response to the type requested by the caller:

use fred::prelude::*;

client.set("key", "value", None, None, false).await?;
let value: Option<String> = client.get("key").await?;

In redis-tower, the command type determines the response type:

use redis_tower::commands::{Get, Set};

client.execute(Set::new("key", "value")).await?;
let value: Option<bytes::Bytes> = client.execute(Get::new("key")).await?;

Binary-safe string responses use bytes::Bytes. Convert only where the application requires another representation:

use redis_tower::RedisValueExt;

let value: String = client.execute(Get::new("key")).await?.parse_into()?;

This moves a class of response-shape mistakes from runtime conversion into the command’s implementation. Collection commands similarly return a concrete shape such as Vec<Bytes>, Vec<(Bytes, Bytes)>, or a command-specific response struct.

Command options

Fred commonly represents command options as additional method arguments. redis-tower puts them on a builder, which makes call sites self-describing:

use redis_tower::commands::SetStatus;

let outcome = client
    .execute(Set::new("lease", "owner-a").nx().ex(30).with_outcome())
    .await?;
if outcome.status == SetStatus::NotApplied {
    println!("the lease already exists");
}
client.execute(Get::new("lease")).await?;

Check the command type’s rustdoc when translating a long Fred call: the builder methods serialize in Redis grammar order even when they are chained in another order. Call with_outcome() last when conditional SET code needs to distinguish an applied write from a failed NX or XX condition; the base Set keeps its original Option<Bytes> response for compatibility.

Commands without a typed builder

Fred’s custom interface maps to RawCommand:

use redis_tower::commands::RawCommand;

let frame = client
    .execute(RawCommand::new("MYCOMMAND").arg("key").arg("value"))
    .await?;

RawCommand returns a protocol Frame; match and validate the expected shape at the boundary. Prefer a typed builder when one exists because it also carries cluster key metadata, blocking behavior, idempotency, and response parsing.

Automatic and explicit pipelining

Fred automatically pipelines concurrent commands and also exposes an explicit client.pipeline() buffer. MultiplexedClient automatically batches commands that arrive concurrently, so ordinary application code needs no explicit pipeline:

let a = client.execute(Get::new("a"));
let b = client.execute(Get::new("b"));
let (a, b) = tokio::try_join!(a, b)?;

For a deliberate ordered batch on one exclusive connection, replace Fred’s explicit pipeline with Pipeline:

use redis_tower::{Pipeline, RedisConnection};
use redis_tower::commands::{Get, Incr};

let mut connection = RedisConnection::connect("127.0.0.1:6379").await?;
let mut results = Pipeline::new()
    .push(Incr::new("counter"))
    .push(Get::new("counter"))
    .execute(&mut connection)
    .await?;

let incremented: i64 = results.take(0)?;
let observed: Option<bytes::Bytes> = results.take(1)?;

An explicit Pipeline works with RedisConnection, RedisClient, and other PipelineExecutor implementations. It is distinct from a transaction: Redis can execute other clients’ commands between entries, and an error in one entry does not roll back the others.

Transactions

Replace Fred’s client.multi() / exec block with Transaction. Do not send raw MULTI and EXEC builders as independent calls through a multiplexed client; other tasks could interleave commands on the shared connection.

use redis_tower::{MultiplexedClient, Transaction, TransactionResult};
use redis_tower::commands::{Get, Incr};

let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
match Transaction::new()
    .watch(["counter"])
    .push(Incr::new("counter"))
    .push(Get::new("counter"))
    .execute(&mut client)
    .await?
{
    TransactionResult::Committed(mut results) => {
        let value: i64 = results.take(0)?;
        let observed: Option<bytes::Bytes> = results.take(1)?;
        let _ = (value, observed);
    }
    TransactionResult::Aborted => {
        // A watched key changed; rebuild this already-known transaction.
    }
}

The standard MultiplexedClient sends the complete WATCH/MULTI/EXEC sequence as one contiguous worker request. For read-compute-write logic and a bounded retry loop, call transaction_with_retries on a dedicated RedisConnection or an exclusive connection checked out from an appropriately sized pool.

Pooling and blocking commands

Fred offers shared Pool, exclusive ExclusivePool, and optional dynamic pooling. In redis-tower, first ask whether a pool is needed:

  • Keep a single MultiplexedClient for high-concurrency, non-blocking command traffic. One connection can sustain many concurrent callers through automatic pipelining.
  • Use ConnectionPool<RedisConnection> when commands block (BLPOP, blocking XREAD, and similar), reply parsing is expensive enough to cause head-of-line blocking, or the application genuinely needs multiple independent connections.
  • Use a dedicated RedisConnection for pub/sub, MONITOR, or an operation that must own its session state.
use redis_tower::{ConnectionPool, RedisConnection};
use redis_tower::pool::{DispatchStrategy, PoolConfig};

let pool = ConnectionPool::connect_with_config(
    PoolConfig::default()
        .name("blocking")
        .size(8)
        .dispatch(DispatchStrategy::LeastConnections),
    || RedisConnection::connect("127.0.0.1:6379"),
).await?;

Unlike Fred’s ExclusivePool::acquire, normal ConnectionPool::execute handles reservation and release for one command. Keep session-scoped sequences on a connection type that explicitly guarantees exclusive ownership.

Reconnection

A Fred builder can attach a ReconnectPolicy to its client. The closest high-throughput redis-tower replacement is a factory-backed MultiplexedClient. The factory recreates the connection and replays URL authentication and database selection; the reconnect config controls bounded exponential backoff and jitter.

use std::time::Duration;
use redis_tower::{AutoPipelineConfig, MultiplexedClient};
use redis_tower::auto_pipeline::AutoPipelineReconnectConfig;
use redis_tower::reconnect::{ReconnectConfig, UrlConnectionFactory};

let factory = UrlConnectionFactory::new(
    "redis://default:secret@127.0.0.1:6379/0",
);
let reconnect = ReconnectConfig::default()
    .base_delay(Duration::from_millis(50))
    .max_delay(Duration::from_secs(2))
    .connect_timeout(Duration::from_secs(3));

let client = MultiplexedClient::from_factory(
    factory,
    AutoPipelineConfig::default(),
    AutoPipelineReconnectConfig::new(reconnect),
).await?;

For rotating cloud credentials, use CredentialConnectionFactory::from_url instead. It keeps the URL’s transport, database, and RESP protocol settings while replacing static URL credentials with the provider, and its setup timeout covers credential lookup, transport setup, authentication, database selection, and protocol negotiation. Share one SharedCredentialProvider across ordinary and dedicated connections so they use the same refresh cache.

Credential rotation is session-aware. Ordinary connections can reauthenticate in place when they are idle. Pub/Sub must reconnect and replay confirmed subscriptions, MONITOR must terminate and reopen with an observable gap, and blocking or transactional work should rotate at a fresh operation boundary. See the cloud authentication guide for complete examples and owner-lifecycle rules.

ResilientRedisClient is a simpler alternative with built-in reconnection, but it serializes all commands through one mutex. It is appropriate for modest traffic or control-plane work; prefer the factory-backed multiplexed form for high concurrency.

Reconnection and command replay are separate decisions. A lost connection can leave the caller unsure whether Redis executed the command. Opt into client.retry(RetryPolicy::default()) only when the typed command is marked idempotent; redis-tower’s retry wrapper refuses to replay non-idempotent writes.

Queueing idempotent work while offline

The default ResilientRedisClient starts a reconnect campaign after a connection error but does not hold later work for Redis to return. If the Fred application intentionally buffered commands during outages, opt into a bounded offline queue:

use redis_tower::{OfflineQueueConfig, ResilientRedisClient};

let client = ResilientRedisClient::connect_url_with_offline_queue(
    "redis://default:secret@127.0.0.1:6379/0",
    OfflineQueueConfig::new(256).with_max_replay_attempts(3),
).await?;

Every clone shares this queue. It admits only typed commands whose Command::idempotent() implementation returns true, preserves queue-ticket replay order, returns RedisError::QueueFull instead of exceeding its capacity, and returns RedisError::ReconnectFailed to admitted work when the reconnect or per-command replay budget is exhausted. Concurrently first-polled futures can be ticketed in either scheduler order. Non-idempotent commands and raw Service<Frame> calls fail while offline rather than risk duplicating a side effect. Canceling an in-wire command quarantines its socket before the next ticket runs. A capacity of zero is a valid fail-fast configuration.

Use offline_queue_depth() and is_reconnecting() for overload and readiness signals. A tower::buffer::Buffer in front of ReconnectService is an option for frame-level stacks, but a frame has lost the typed command’s idempotency metadata. The buffer therefore cannot make a safe replay decision for you.

Cluster and Sentinel

Fred selects centralized, clustered, or Sentinel deployment through its configuration and returns the same general client type. redis-tower uses topology-specific crates so their configuration and routing contracts remain explicit:

use redis_tower_cluster::MultiplexedClusterClient;

let cluster = MultiplexedClusterClient::builder("127.0.0.1:7000")
    .max_redirects(5)
    .connect()
    .await?;
use redis_tower_sentinel::MultiplexedSentinelClient;

let sentinel = MultiplexedSentinelClient::builder(
    &["127.0.0.1:26379", "127.0.0.1:26380"],
    "mymaster",
)
.connect_with_reconnect()
.await?;

For application code that must select a topology from configuration at runtime, use redis_tower_client::UniversalClient. Its URL schemes are redis://, redis+cluster://, and redis+sentinel://; all variants expose the same typed execute method. As with Fred’s URLs, credentials ride in the userinfo (redis+cluster://user:pass@host:7000, percent-decoded), the rediss variants enable TLS, and sentinel URLs take separate sentinel credentials as query parameters – Fred’s ?sentinelUsername=U&sentinelPassword=P becomes ?sentinel_username=U&sentinel_password=P:

redis+sentinel://app:node-pass@s1:26379,s2:26379/mymaster?sentinel_password=sent-pass

Read the feature matrix before assuming exact parity. RESP3 negotiation is available on standalone and cluster clients, while Sentinel support remains in progress. Some Fred features such as dynamic pools or alternate async runtimes do not have direct redis-tower equivalents.

Pub/sub

Fred’s optional SubscriberClient manages subscription state and offers broadcast receivers or callbacks. redis-tower makes the dedicated Redis connection and message stream explicit:

use redis_tower::{PubSubConnection, RedisConnection};
use tokio_stream::StreamExt;

let connection = RedisConnection::connect_url("redis://127.0.0.1:6379").await?;
let mut subscriber = PubSubConnection::from_connection(connection)?;
subscriber.subscribe(&["events"]).await?;

while let Some(message) = subscriber.next().await {
    let message = message?;
    println!("{}: {:?}", message.channel, message.payload);
}

PubSubConnection tracks confirmed channel, pattern, and shard-channel subscriptions. After a connection failure, call reconnect_with(&factory) to open a new dedicated connection and replay them. This recovery step is explicit; do not assume that merely polling the stream reconnects it.

Errors, health, and middleware

Replace Fred ErrorKind matching with redis-tower’s classification helpers:

Intentredis-tower
Connection failederror.is_connection_error()
Retry may be usefulerror.is_retryable()
Redis Cluster redirecterror.is_moved() / error.is_ask()
Wrong data typeerror.is_wrongtype()
Pool saturatedmatch RedisError::PoolAcquisitionTimeout
Pipeline queue shed loadmatch RedisError::QueueFull
Reconnect lifecyclesubscribe to ConnectionEventBus before constructing a reconnecting client

Fred callbacks and client configuration cover much of its operational policy. With redis-tower, compose the frame service with TracingLayer, MetricsLayer, CommandTimeoutLayer, and the Redis-aware circuit breaker, or wrap typed execution with RetryLayer. The production tuning page shows how those pieces fit together.

Connection lifecycle streams are bounded observations, not flow control. A slow subscriber receives ConnectionEventRecvError::Lagged { skipped }; event consumption never delays reconnect or failover. Use events to drive metrics, logs, and diagnostics, not as a durable audit log or a prerequisite for recovery.

Graceful shutdown

Fred applications normally call quit() and then await or otherwise account for the connection task. For a redis-tower multiplexed client, stop producers, drop their clones, and consume the final owner with shutdown():

// Stop accepting application work and wait for spawned tasks first.
client.shutdown().await;

If another clone is still alive, shutdown() returns without stopping the shared worker. Treat ownership during shutdown as an application-level invariant. For pools call pool.close().await to reject new work and drain accepted commands, then drop the remaining clones and connections.

Migration checklist

  1. Inventory Fred features and deployment modes; add only the matching redis-tower crates and feature flags.
  2. Replace the Fred client with MultiplexedClient, unless the workload is blocking or needs exclusive session state.
  3. Convert interface calls to typed command builders and make Bytes-to-domain conversions explicit.
  4. Decide whether concurrent automatic batching is enough before translating every explicit Fred pipeline.
  5. Rebuild transactions with Transaction, not independent raw MULTI/EXEC calls.
  6. Choose an explicit reconnect path and confirm AUTH, SELECT, TLS, and credentials are replayed by its factory; decide whether offline work should fail fast or use the bounded idempotent-only queue.
  7. Put blocking commands, pub/sub, and MONITOR on dedicated connections.
  8. Add bounded timeouts, queue or pool backpressure, metrics, readiness checks, and graceful shutdown before switching production traffic.
  9. Run the old and new clients against the same Redis version in staging and compare response conversions, latency distributions, error rates, and reconnect/failover behavior.

Cloud and rotating credentials

redis-tower separates two credential-rotation moments:

  1. Every fresh socket asks a CredentialProvider for current credentials. A CredentialConnectionFactory, Cluster builder, or Sentinel builder carries that provider through initial connect, reconnect, pool growth, topology discovery, and failover.
  2. A StreamingCredentialProvider emits replacements for sockets that remain open. An owned reauthentication handle sends AUTH to every current data socket and stops when the handle is dropped or shut down.

Connection setup always opens in RESP2, authenticates, and only then negotiates the requested protocol. If Redis rejects setup credentials with NOAUTH or WRONGPASS, the provider is force-refreshed and AUTH is attempted once more. This bounded retry applies only to connection setup. redis-tower never replays a user command after an authentication error.

Use CredentialConnectionFactory::from_url when the deployment URL also owns transport, database, or protocol configuration. The provider replaces only the URL’s username/password; TCP versus TLS versus Unix socket, db, and protocol remain part of every initial or replacement connection. Bound the whole sequence—not just the socket dial—with with_setup_timeout:

let factory = CredentialConnectionFactory::from_url(
    "rediss://cache.example.net:6380/2?protocol=resp3",
    provider.clone(),
)
.with_connection_config(connection_config)
.with_setup_timeout(Duration::from_secs(5));

ConnectionConfig::connect_timeout covers the TCP or Unix connect operation. The factory setup timeout additionally covers provider lookup, TLS, the one allowed forced refresh, SELECT, and HELLO. A timeout drops the partial socket and returns RedisError::ConnectTimeout.

Shared provider ownership

Concrete AWS and Entra providers are cheap clones backed by one internal cache. Type-erased hosts can use SharedCredentialProvider to retain the same property after converting a provider to Arc<dyn CredentialProvider> or Arc<dyn StreamingCredentialProvider>:

use std::sync::Arc;
use redis_tower::{
    CredentialConnectionFactory, CredentialProvider, SharedCredentialProvider,
};

let erased: Arc<dyn CredentialProvider> = build_provider();
let shared = SharedCredentialProvider::from_arc(erased);

let ordinary = CredentialConnectionFactory::from_url(redis_url, shared.clone());
let dedicated = CredentialConnectionFactory::from_url(redis_url, shared.clone());
let cluster = MultiplexedClusterClient::builder(cluster_seed)
    .credentials(shared)
    .connect()
    .await?;

Every clone delegates to the same allocation, so concurrent ordinary, dedicated, and Cluster connection attempts share the provider’s cache and single-flight behavior.

Established sockets and dedicated modes

Credential lookup during connection setup and credential changes after setup are different contracts:

OwnerOn provider updateRejected replacement AUTH
Standalone MultiplexedClientUse spawn_credential_reauthentication to serialize AUTH through its workerThe worker stays installed with its previous Redis identity. The task logs a redacted warning and consumes later updates. For fail-closed rotation, stop admission, shut down the client, and rebuild it through the provider factory before resuming traffic.
Retained ConnectionPoolUse spawn_credential_reauthentication to visit every active slotFailed slots stay installed with their previous identity while the remaining slots are attempted. For fail-closed rotation, call close, drop all clones, and rebuild the pool through the provider factory.
Direct or multiplexed Cluster clientUse the owner’s spawn_credential_reauthentication handleThe failed node socket/worker is removed from routing and a later command reconnects through the provider.
Direct SentinelClientUse spawn_credential_reauthentication for current data sockets; discovery sockets are short-livedA failed master is marked for rediscovery before the next command, and failed replicas are removed.
MultiplexedSentinelClientUse spawn_credential_reauthentication for current data workersFailed workers stay routed with their previous identity. Stop routing, drop every client clone/update handle, and rebuild the client before resuming traffic when rotation must fail closed.
Fresh blocking or transaction connectionFinish/cancel the operation, then reconnect through the provider factoryInserting AUTH into connection-local operation state is not a generic safe boundary. An ambiguous operation is never replayed.
Pub/SubReconnect through the provider factory; confirmed subscriptions are replayedSubscription mode is stateful, and an arbitrary callback must not inject AUTH. Messages during the gap are lost.
MONITORTerminate and create a new MonitorStream through the provider factoryMONITOR owns a one-way event stream, has no resume cursor, and loses events during the gap.

The generic update task is deliberately best-effort: it does not expose an error channel, renders neither provider nor callback errors, and continues after a failure. Where an owner exposes reauthenticate_all, call it directly when the result is needed synchronously. The table above is the authoritative retirement policy; do not assume that every owner evicts a socket after a rejected update.

PubSubConnection::connect_with, BinaryPubSubConnection::connect_with, and MonitorStream::connect_with open their dedicated sockets through any ConnectionFactory, including CredentialConnectionFactory. A streaming provider owner can select between its update stream and the session stream:

use std::sync::Arc;
use redis_tower::{BinaryPubSubConnection, StreamingCredentialProvider};
use tokio_stream::StreamExt;

let mut pubsub = BinaryPubSubConnection::connect_with(&factory).await?;
pubsub.subscribe_bytes(&[b"events"]).await?;
let mut credential_updates = Arc::clone(&streaming_provider).subscribe();

loop {
    tokio::select! {
        update = credential_updates.next() => {
            update.ok_or("credential stream ended")??;
            // The provider cache is current before it emits. Reconnect rather
            // than sending AUTH in subscription mode.
            pubsub.reconnect_with(&factory).await?;
        }
        message = pubsub.next() => {
            let Some(message) = message else { break };
            handle_message(message?);
        }
    }
}

Dropping a CredentialReauthenticationHandle cancels and aborts its provider stream; shutdown().await performs the same cancellation and waits for task exit. Dedicated-session select loops are owned directly by the application and stop when that owner is dropped.

AWS ElastiCache IAM

redis-tower-auth-aws implements the ElastiCache IAM SigV4 flow for provisioned replication groups and serverless caches. It creates a presigned elasticache:Connect request with the documented 15-minute lifetime, caches it for reconnect fan-out, and emits a replacement after 75% of that lifetime.

ElastiCache IAM requires TLS. The provider creates credentials; the client must still be configured with the appropriate TLS hostname and trust roots.

use std::sync::Arc;
use aws_config::BehaviorVersion;
use redis_tower::{
    AutoPipelineConfig, ConnectionConfig, CredentialConnectionFactory,
    MultiplexedClient, ProtocolVersion, StreamingCredentialProvider,
};
use redis_tower::auto_pipeline::AutoPipelineReconnectConfig;
use redis_tower_auth_aws::{
    ElastiCacheIamProvider, ElastiCacheResourceType,
};

let aws = aws_config::load_defaults(BehaviorVersion::latest()).await;
let provider = ElastiCacheIamProvider::new(
    "redis-app",
    "production-cache",
    "us-west-2",
    ElastiCacheResourceType::ReplicationGroup,
    aws.credentials_provider().expect("AWS credentials").clone(),
)?;

let factory = CredentialConnectionFactory::new(
    "production-cache.example.cache.amazonaws.com:6379",
    provider.clone(),
)
.with_connection_config(
    ConnectionConfig::new().with_protocol(ProtocolVersion::Resp3),
)
.with_tls("production-cache.example.cache.amazonaws.com", tls_config);

let client = MultiplexedClient::from_factory(
    factory,
    AutoPipelineConfig::default(),
    AutoPipelineReconnectConfig::default(),
).await?;
let updates: Arc<dyn StreamingCredentialProvider> = Arc::new(provider);
let auth_handle = client.spawn_credential_reauthentication(updates);

Use ElastiCacheResourceType::ServerlessCache for serverless ElastiCache. The signed request then includes ResourceType=ServerlessCache. The IAM user name and user ID must match, and the IAM policy must permit both the cache resource and user resource.

Microsoft Entra ID

redis-tower-auth-azure requests the standard Azure Redis scope, https://redis.azure.com/.default. The managed identity or service-principal object ID becomes the Redis username and its access token becomes the password. The provider caches a token and emits its replacement at approximately 75% of the observed lifetime.

use std::sync::Arc;
use redis_tower::{
    AutoPipelineConfig, CredentialConnectionFactory, MultiplexedClient,
    StreamingCredentialProvider,
};
use redis_tower::auto_pipeline::AutoPipelineReconnectConfig;
use redis_tower_auth_azure::EntraIdProvider;

let provider = EntraIdProvider::managed_identity(
    "00000000-0000-0000-0000-000000000000",
    None,
)?;
let factory = CredentialConnectionFactory::new(
    "my-cache.westus.redis.azure.net:6380",
    provider.clone(),
)
.with_tls("my-cache.westus.redis.azure.net", tls_config);

let client = MultiplexedClient::from_factory(
    factory,
    AutoPipelineConfig::default(),
    AutoPipelineReconnectConfig::default(),
).await?;
let updates: Arc<dyn StreamingCredentialProvider> = Arc::new(provider);
let auth_handle = client.spawn_credential_reauthentication(updates);

EntraIdProvider::new accepts any Azure SDK TokenCredential, so workload identity and service-principal credentials use the same cache and push stream. with_scope is available for sovereign-cloud or compatibility deployments.

The downstream redis-database-mcp-rs#92 contract keeps Azure SDK dependencies behind the server’s optional Entra feature. Construct one EntraIdProvider in the server layer, wrap or clone it into every DirectRedis* connection factory, and keep static URL authentication as the no-feature/default path. Its acceptance test injects a unique token into a failing provider, invokes a real MCP tool, and proves the serialized tool result or JSON-RPC error, returned metadata, and captured tracing/log output omit the token while retaining is_authentication_error classification. MCP policy and that tool-boundary evidence stay downstream.

Pools, Cluster, and Sentinel

Use one cloneable provider instance for both socket creation and streaming updates. Provider clones share their token cache.

// Pool: the factory covers new/replacement slots; the handle covers live slots.
let factory = CredentialConnectionFactory::new(address, provider.clone());
let pool = ConnectionPool::connect_with_factory(pool_config, factory).await?;
let auth_handle = pool.spawn_credential_reauthentication(Arc::new(provider.clone()));

// Cluster: every node setup/reconnect fetches credentials.
let cluster = MultiplexedClusterClient::builder(seed)
    .credentials(provider.clone())
    .connect()
    .await?;
let auth_handle = cluster.spawn_credential_reauthentication(Arc::new(provider.clone()));

// Sentinel discovery credentials are independent from Redis data credentials.
// Only data sockets persist, so the update handle targets node credentials.
let sentinel = SentinelClient::builder(sentinels, service_name)
    .sentinel_credentials(sentinel_provider)
    .node_credentials(provider.clone())
    .connect()
    .await?;
let auth_handle = sentinel.spawn_credential_reauthentication(Arc::new(provider));

Direct ClusterClient, SentinelClient, MultiplexedClusterClient, and MultiplexedSentinelClient expose the same owned update mechanism, but their failure retirement differs. Both Cluster owners remove a failed node from routing so its normal provider-backed reconnect path can rebuild it. Direct SentinelClient forces rediscovery after a master failure and removes failed replicas. MultiplexedSentinelClient retains a worker that rejects AUTH with its previous identity; stop routing and rebuild that owner to fail closed.

Secret handling and failure behavior

Credentials redacts passwords and tokens from Debug output and zeroizes its owned username/password buffers on drop. The AWS and Azure providers retain their cached Redis credentials in this type, so replacing or dropping a cached token zeroizes that allocation. Avoid cloning or logging token strings in application provider implementations.

Credential update streams are deliberately best-effort. Provider and AUTH errors are logged without rendering third-party error text, and later emissions are still consumed. Provider failures returned from setup are redacted to an AUTH_PROVIDER ... failed message and are recognized by is_authentication_error; Redis NOAUTH and WRONGPASS retain their server classification. Keep the owned handle alive for as long as the client should receive proactive updates, and call shutdown().await during graceful shutdown. A provider should delay after an emitted error; this prevents a broken credential source from creating a busy loop.

Manual Azure Managed Redis verification

The normal suite uses deterministic fake providers; no cloud credential is required. Before shipping a downstream Entra integration, run this manual check against a disposable Azure Managed Redis database:

  1. Create a Redis data-access-policy assignment for the managed identity or service principal and record its object ID. Start with the least-privilege command/key policy required by the application.
  2. Configure rediss://<host>:10000/0?protocol=resp3 without URL credentials. Build EntraIdProvider from the host’s Azure TokenCredential, then build a URL-backed CredentialConnectionFactory with a finite setup timeout.
  3. Exercise an ordinary multiplexed command, one new dedicated connection, and Cluster node discovery if the target exposes Cluster mode. Confirm no token appears in logs, errors, traces, or MCP tool output.
  4. Leave the process running through one proactive refresh. Verify ordinary sockets reauthenticate, a Pub/Sub owner reconnects and restores confirmed subscriptions, and a MONITOR owner terminates/reopens with its documented observation gap.
  5. Revoke or misconfigure the assignment and confirm the application reports a redacted authentication-category failure without retrying an ambiguous write, transaction, or blocking operation.

Client-side caching

redis-tower combines Redis server-assisted tracking with bounded local response caches for standalone Redis and master-routed Redis Cluster deployments. Use CachedMultiplexedClient for standalone work or redis_tower_cluster::CachedMultiplexedClusterClient for Cluster. Both are cheap to clone and share one coherent cache across their clones.

use std::time::Duration;
use redis_tower::{
    CacheTrackingMode, CachedClientConfig, CachedMultiplexedClient,
    commands::{Get, Set},
};

let config = CachedClientConfig::new()
    .max_entries(50_000)
    .client_ttl(Some(Duration::from_secs(15)))
    .tracking_mode(CacheTrackingMode::broadcast_with_prefixes(["user:"]));

let client =
    CachedMultiplexedClient::connect_with_config("127.0.0.1:6379", config).await?;
client.execute(Set::new("user:42", "Ada")).await?;

let first = client.execute(Get::new("user:42")).await?; // Redis miss
let second = client.clone().execute(Get::new("user:42")).await?; // local hit
assert_eq!(first, second);

let statistics = client.cache_statistics().await;
println!("hits={}, misses={}", statistics.hits, statistics.misses);
client.shutdown().await;
Ok::<(), redis_tower::RedisError>(())

Standalone connection setup

The address constructor is the shortest path for plain standalone Redis. connect_url applies URL authentication, database selection, TLS, or a Unix socket to both the data connection and invalidation receiver. The *_with_connection_config variants also propagate keepalive, connect timeout, and RESP decode limits while forcing the RESP3 protocol required by tracking.

For custom TLS roots, mTLS, rotating credentials, or application-specific setup, pass a ConnectionFactory to from_factory. To reuse an existing data connection, use from_connection_with_factory; its receiver factory must connect to the same standalone Redis server and reproduce the required authentication/session setup. The data worker is intentionally fixed rather than silently reconnecting without replaying tracking state; construct a new cached client after a data-connection failure.

This two-connection design requires Redis Open Source-compatible REDIRECT support. Redis Software and Redis Cloud currently do not support two-connection tracking, so a rediss:// URL alone does not make this client compatible with those services.

Tracking modes

CachedClientConfig supports three Redis tracking modes:

  • CacheTrackingMode::Broadcast receives invalidations for every key, or only for configured binary-safe prefixes. It uses no per-client key table on the Redis server and is the default.
  • CacheTrackingMode::ServerDefault asks Redis to remember the keys this data connection reads. It reduces invalidation traffic when the working set is narrow, at the cost of server-side tracking memory.
  • CacheTrackingMode::OptIn tracks only cache misses selected by the client. redis-tower submits CLIENT CACHING YES and the read as one atomic worker request, so another clone cannot interleave a command between them.

For standalone CachedMultiplexedClient, all modes use two RESP3 connections. A dedicated receiver owns invalidation pushes; the data connection redirects tracking messages to its client ID and uses NOLOOP. Writes through the standalone cached client therefore invalidate locally both before and after dispatch. The second invalidation also rejects an old read that raced the write.

The standalone cached client owns that connection-local state. Caller-issued CLIENT TRACKING, CLIENT CACHING, CLIENT REPLY, HELLO, RESET, and QUIT commands are rejected, including inside explicit pipelines and transactions. Use a dedicated uncached connection for session administration. Read-only diagnostics such as CLIENT TRACKINGINFO, CLIENT GETREDIR, and CLIENT ID remain available.

The local response cache currently supports GET, HGET, HGETALL, LRANGE, SMEMBERS, ZRANGE, and TYPE. Other commands still use the same client and auto-pipeline worker, but bypass the local response cache. With prefix-filtered broadcast tracking, cacheable reads outside the configured prefixes also bypass both the cache and its hit/miss statistics; Redis would not send invalidations for those keys, so caching them would be unsafe.

Staleness and failure behavior

The local cache has two independent safety bounds:

  • Redis invalidation pushes remove all cached command variants that depend on a changed key.
  • client_ttl caps how long any entry can be served even when no invalidation arrives. The safe default is 30 seconds. Standalone clients may pass None only when the deployment can tolerate relying exclusively on tracking; Cluster clients reject None.

Each cache miss snapshots a per-key invalidation epoch and a global generation. If an invalidation, explicit clear, or local write happens while Redis is answering, the late response is returned to its caller but is not inserted. Epoch bookkeeping is bounded alongside the cache.

If the standalone invalidation receiver disconnects, redis-tower immediately clears and disables the cache. Reads pass through to Redis while a supervisor reconnects, obtains a new receiver ID, and installs the new redirect on the data worker. Caching resumes only after that setup succeeds.

The data connection has stricter semantics because its tracking state and in-flight command ordering cannot be reconstructed transparently. A clean idle socket loss is detected by the auto-pipeline worker without waiting for a cache miss or write; redis-tower marks caching unhealthy and clears the cache before it releases any affected callers. The fixed worker then remains closed, so construct a new cached client. Silent network black holes remain bounded by the configured TCP keepalive policy or the next command deadline rather than by an application heartbeat.

For the standalone client, is_caching_healthy() includes both receiver tracking and data-worker health and is suitable for readiness or diagnostics. It never treats an open cache as healthy after the fixed data worker has stopped.

Redis Cluster

CachedMultiplexedClusterClient keeps one cache above cluster routing and one dedicated RESP3 invalidation receiver for every current master. Its builder accepts the same CachedClientConfig and forces RESP3 across seed discovery, data nodes, redirects, topology refreshes, receivers, and reconnects.

use redis_tower::{CacheTrackingMode, CachedClientConfig, commands::Get};
use redis_tower_cluster::CachedMultiplexedClusterClient;
use std::time::Duration;

let config = CachedClientConfig::new()
    .client_ttl(Some(Duration::from_secs(30)))
    .tracking_mode(CacheTrackingMode::OptIn);
let client = CachedMultiplexedClusterClient::builder("127.0.0.1:7000")
    .cache_config(config)
    .connect()
    .await?;

let value = client.execute(Get::new("user:42")).await?;
let _ = value;
client.shutdown().await;
Ok::<(), redis_tower::RedisError>(())

Cache use opens only after every current master has a healthy data worker and CLIENT TRACKING ... REDIRECT points at a live receiver. Data or receiver loss, an ambiguous timeout, and any receiver/topology coverage rebuild close the gate and clear the cache before it can reopen. MOVED and ASK close the gate before the router awaits a new connection; slot ownership epochs reject responses that began under an older owner. Opt-in dispatch remains one reserved worker submission: [CLIENT CACHING YES, command]. Redis’s one-shot ASKING and CLIENT CACHING YES flags consume one another, so ASK closes the gate and retries as [ASKING, command]; that migrated response is not cached.

The initial Cluster implementation intentionally supports ReadPreference::Master only. Replica and prefer-replica policies are rejected until equivalent invalidation coverage can be proven for replica reads. Cluster caching also requires a finite client_ttl. An empty slot can move without producing a redirect or invalidation visible to this client, leaving an old-owner cached miss that only the TTL can eventually expire. This Cluster-specific check runs before the seed connection is opened; standalone cached clients continue to permit client_ttl(None).

Capacity and observability

max_entries bounds local response entries; 0 makes the cache unbounded and is not recommended for production. Eviction is intentionally simple and bounded rather than an application-level replacement policy.

Every cached client exposes aggregate hits, misses, invalidations, and evictions. Supplying a MetricsRecorder in CachedClientConfig emits the same bounded events; MetricsFacadeRecorder publishes them as redis_tower.cache.events with a four-value event label. Keys and command arguments are never metric labels.

Tower and exclusive-client forms

CacheLayer / CacheService remain available for custom Tower stacks. Prefer constructors that attach and own an invalidation stream. A cache service built without invalidations is only safe when the caller explicitly clears it after every possible external write; its constructor documents that hazard loudly. Recorder-enabled stream constructors are available when custom Tower stacks need the same cache-event metrics as CachedClientConfig.

Tower calls poll_ready before supplying a request, so an inner service may already hold bounded capacity when CacheService discovers a local hit. Cache backends must implement ReleaseReadiness to return that capacity without an inner call; redis-tower implements the contract for AutoPipelineService, FrameService, and ReconnectService. Put CacheLayer directly above one of those backends and place tracing, metrics, timeouts, and other middleware outside the cache. Custom middleware placed inside the cache must explicitly propagate ReleaseReadiness to its inner service. There is no blanket no-op implementation because that could silently leak permits and stall a bounded service.

CachedClient is the lower-throughput serialized compatibility form. It uses the same cache service and connection actor with a one-request batch limit, so it shares the clone-safe tracking, failure, race-protection, and shutdown semantics without auto-batching concurrent misses. Existing CachedClient::connect callers keep the safe broadcast defaults. Both standalone cached client types expose shutdown() so the final clone can explicitly join their worker and invalidation lifecycles. The cached Cluster client provides the same final-clone shutdown behavior for all per-master workers and receivers.

Distributed primitives

The redis-tower-primitives crate builds locks, leader election, expirable semaphores, countdown latches, delayed queues, block-allocated IDs, and shared rate limits on the generic RedisExecutor interface. It works with standalone, multiplexed, pooled, resilient, cluster, Sentinel, and universal clients that implement that interface. Every script is a public, line-documented constant and runs through the EVALSHA-first Script helper.

Distributed lock

DistributedLock::new(lock_key, fencing_key, ttl) requires its TTL. Acquisition atomically checks the lock, increments the fencing counter, and performs SET NX PX. A lease can release or extend only while its random owner token still matches.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::DistributedLock;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let lock = DistributedLock::new(
    "{job:17}:lock",
    "{job:17}:fence",
    Duration::from_secs(15),
)?;

if let Some(lease) = lock.acquire(&mut client).await? {
    let fencing_token = lease.fencing_token();
    // The protected service must reject tokens lower than the greatest token
    // it has already accepted.
    write_with_fence(fencing_token).await?;
    lease.release(&mut client).await?;
}
Ok(())
}
async fn write_with_fence(_token: u64) -> Result<(), Box<dyn std::error::Error>> { Ok(()) }

Lock acquisition touches two keys. In Redis Cluster they must share a hash tag, as {job:17}:lock and {job:17}:fence do.

Renewal is opt-in. LockLease::spawn_renewal consumes the lease and starts a task only at that explicit call. Its owned handle cancels and aborts renewal on drop. shutdown().await stops cleanly and returns the lease for release.

Lock failure mode

The TTL bounds ownership; it does not prove that a paused or partitioned owner has stopped. Such an owner can resume after a higher-token replacement begins work. The protected resource must enforce fencing tokens. A lost acquisition response is indeterminate, and Redis failover can roll back an unreplicated fencing increment if deployment durability permits it.

Leader election

LeaderElection::new(key, ttl, renewal_interval) requires both timing choices. No task starts during construction. A successful campaign(executor) call atomically claims the single election key and returns a Campaign containing the owned Leadership renewal handle and a separate LeadershipEvents receiver.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::{LeaderElection, LeadershipEvent};

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let candidate = MultiplexedClient::connect("127.0.0.1:6379").await?;
let election = LeaderElection::new(
    "{scheduler}:leader",
    Duration::from_secs(15),
    Duration::from_secs(5),
)?;

if let Some(campaign) = election.campaign(candidate).await? {
    let (leadership, mut events) = campaign.into_parts();
    assert_eq!(events.recv().await, Some(LeadershipEvent::Elected));
    run_one_scheduler_term().await?;
    let _outcome = leadership.abdicate().await?;
    assert_eq!(events.recv().await, Some(LeadershipEvent::Demoted));
}
Ok(())
}
async fn run_one_scheduler_term() -> Result<(), Box<dyn std::error::Error>> { Ok(()) }

LeadershipEvent::RenewalFailed means the process must stop leader-only work immediately. LeadershipEvent::Demoted reports expiration, replacement, or completed abdication. abdicate().await confirms cleanup. Dropping leadership requests the same compare-and-delete operation in the detached owned task, and the TTL bounds the key if that best-effort cleanup cannot run.

Leader-election failure mode

A TTL lease cannot prove that a paused or partitioned process has stopped. Redis failover can also lose an unreplicated campaign or renewal. Applications must make leader work idempotent or protect downstream writes with another stale-owner mechanism. A lost campaign response is indeterminate, so the caller must not infer election without a successful return and Elected event.

Expirable semaphore

ExpirableSemaphore::new(key, permit_limit, ttl) stores random permit tokens in one sorted set. Acquisition uses Redis TIME, removes expired holders, and checks capacity atomically. A SemaphorePermit supports token-checked renew and release; drop leaves recovery to the required TTL.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::ExpirableSemaphore;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let semaphore = ExpirableSemaphore::new(
    "{worker-pool}:permits",
    16,
    Duration::from_secs(30),
)?;

if let Some(permit) = semaphore.try_acquire(&mut client).await? {
    run_bounded_work().await?;
    permit.release(&mut client).await?;
}
Ok(())
}
async fn run_bounded_work() -> Result<(), Box<dyn std::error::Error>> { Ok(()) }

All users of a semaphore key must agree on its limit and TTL. There is no automatic permit-renewal task; the holder chooses whether and when to call renew.

Semaphore failure mode

Expiry recovers capacity after a crash, but a paused holder can resume after a replacement acquires the freed permit. The protected operation must tolerate stale overlap. Lost acquire, renew, and release responses are indeterminate and must not be blindly retried when duplicate bounded work would be unsafe.

Countdown latch

CountDownLatch::new(key, initial_count, ttl) requires a positive count and an absolute TTL. initialize never resets an existing latch. count_down uses DECR without allowing the count below zero. wait requires a poll interval and timeout, starts no task, and reports count-zero release, expiry, and timeout as distinct outcomes. Its timeout bounds polling sleeps but does not cancel an in-flight Redis request.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::{CountDownLatch, LatchWaitOutcome};

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let latch = CountDownLatch::new("{deploy:42}:ready", 3, Duration::from_secs(60))?;
latch.initialize(&mut client).await?;

match latch
    .wait(
        &mut client,
        Duration::from_millis(100),
        Duration::from_secs(30),
    )
    .await?
{
    LatchWaitOutcome::Released => start_traffic().await?,
    LatchWaitOutcome::Expired | LatchWaitOutcome::TimedOut { .. } => abort_deploy().await?,
}
Ok(())
}
async fn start_traffic() -> Result<(), Box<dyn std::error::Error>> { Ok(()) }
async fn abort_deploy() -> Result<(), Box<dyn std::error::Error>> { Ok(()) }

Latch failure mode

The TTL is an abandonment bound, not a successful release signal. Waiters receive Expired for a missing key and must decide how to recover. A lost countdown response is indeterminate because the decrement may already have committed; retrying can release the latch earlier than intended.

Delayed queue

DelayedQueue::new(key, retention) requires the post-deadline interval during which a message remains claimable. enqueue accepts binary payloads and a caller-selected delay, then uses Redis TIME to calculate its sorted-set score. Random member prefixes preserve duplicate payloads. claim_due atomically removes a bounded batch in deadline order and starts no poll task.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::DelayedQueue;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let queue = DelayedQueue::new(
    "{mail}:delayed",
    Duration::from_secs(60),
)?;
queue
    .enqueue(&mut client, b"message-42", Duration::from_secs(5))
    .await?;

let claim = queue.claim_due(&mut client, 32).await?;
for payload in claim.payloads() {
    deliver(payload).await?;
}
record_expired(claim.expired());
Ok(())
}
async fn deliver(_payload: &[u8]) -> Result<(), Box<dyn std::error::Error>> { Ok(()) }
fn record_expired(_count: u64) {}

The queue touches one key and is cluster-safe. Every participant sharing that key must use the same retention. The key TTL tracks the latest scheduled deadline plus retention; a claim also reports messages it actively pruned for exceeding retention. Redis may expire the whole key without a later claim, so that reported count is diagnostic rather than a complete loss counter.

Delayed-queue failure mode

Claims are deliberately at-most-once: selected messages are removed before the script returns. A lost claim response can therefore lose the batch. A lost enqueue response is indeterminate and retrying can create a duplicate. The caller owns polling, processing, retries, and any dead-letter policy; use Redis Streams or another acknowledged queue when at-least-once delivery is required.

Block-allocated ID generator

IdGenerator::new(key, block_size) reserves a contiguous positive range with exactly one INCRBY. IdBlock then yields those IDs locally without more Redis I/O or a background task.

use redis_tower::MultiplexedClient;
use redis_tower_primitives::IdGenerator;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let generator = IdGenerator::new("{orders}:id", 1_000)?;
let mut block = generator.allocate(&mut client).await?;

while let Some(id) = block.next() {
    create_order(id).await?;
}
Ok(())
}
async fn create_order(_id: u64) -> Result<(), Box<dyn std::error::Error>> { Ok(()) }

The counter intentionally has no TTL. The key must be reserved for compatible ID generators and kept for at least as long as issued IDs can matter.

ID-generator failure mode

A lost INCRBY response makes one block indeterminate. Allocating another block is uniqueness-safe but leaves a gap; callers must never infer the unknown range. Deleting, expiring, or restoring an older counter can reissue IDs, and Redis failover can roll back an increment when durability permits it. Configure persistence and replication for the required uniqueness boundary.

GCRA rate limiter

GcraRateLimiter::new(key, quota, window) requires both quota and window. Its single Lua call reads Redis TIME, prunes obsolete virtual arrivals from one sorted set, and atomically admits or denies a request. A quota of 100 over one second sustains 100 requests per second and permits an initial burst of 100.

use std::time::Duration;
use redis_tower::MultiplexedClient;
use redis_tower_primitives::GcraRateLimiter;

async fn example() -> Result<(), Box<dyn std::error::Error>> {
let mut client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let limiter = GcraRateLimiter::new("{tenant:7}:rate", 100, Duration::from_secs(1))?;
let decision = limiter.check(&mut client).await?;
if !decision.is_allowed() {
    tokio::time::sleep(decision.retry_after()).await;
}
Ok(())
}

The script touches one key and is cluster-safe. A shared Redis limiter enforces cross-process quota, while tower-resilience-ratelimiter can put its RateLimiterLayer in backpressure mode for process-local admission pressure; production services commonly use both.

Rate-limit failure mode

Redis failures are surfaced without a fail-open/fail-closed policy. Applications must choose that policy at their boundary. A connection loss is indeterminate: the request may have consumed a cell before the response disappeared, so an automatic retry can double-consume quota. Failover can lose recent admissions when Redis durability permits data loss.

Production tuning

redis-tower’s defaults favor predictable behavior and broad compatibility. Production tuning is the process of putting explicit bounds around your own latency, concurrency, payload, and failure budgets. There is no universally correct pool size or timeout: measure a representative workload, change one limit at a time, and keep enough observability to tell saturation from Redis latency or network failure.

This page describes the current public configuration surface. The values in examples are starting points to test, not recommendations to copy unchanged.

Start with the right client

Client choice has a larger effect than most numeric knobs:

WorkloadStart withWhy
Many tasks issuing short, independent commandsMultiplexedClientOne auto-pipelined connection; cheap clones; real queue backpressure
Long-running service that must reconnectFactory-backed MultiplexedClientRetains high concurrency while rebuilding the worker’s connection
Repeated standalone reads with tolerable bounded stalenessCachedMultiplexedClientCloneable auto-pipelined misses plus local hits and RESP3 invalidation tracking
Modest traffic where the simplest reconnecting handle matters mostResilientRedisClientBuilt-in reconnect and single-flight recovery, but commands serialize through one mutex
Blocking commands or expensive reply parsingConnectionPool<RedisConnection>Multiple independent connections isolate head-of-line blocking
Stateful sequence, pub/sub, or MONITORDedicated RedisConnection-based APIExclusive protocol/session ownership
High-concurrency Redis ClusterMultiplexedClusterClientPer-node auto-pipeline workers, redirects, and topology refresh
Repeated master-routed Cluster reads with bounded stalenessCachedMultiplexedClusterClientOne slot-aware shared cache with complete per-master RESP3 invalidation coverage
Sentinel-managed primary with failoverMultiplexedSentinelClient::connect_with_reconnectRe-discovers and verifies the primary after failure or READONLY

Never send BLPOP, BRPOP, blocking XREAD / XREADGROUP, or another blocking command through a shared MultiplexedClient. One blocked Redis connection stalls its pipeline worker and every clone. Isolate blocking traffic on dedicated or pooled connections, and ensure the pool has enough independent slots for the maximum intended number of simultaneous blockers.

Likewise, keep PubSubConnection and MonitorStream dedicated. Redis changes the connection’s mode for pub/sub and MONITOR, and MONITOR itself has a large server-side throughput cost.

Plan for Smart Client Handoff maintenance

Planned-maintenance handoff is opt-in for a factory-backed MultiplexedClient. Construct it with MultiplexedClient::from_factory_with_maintenance and retain the returned MaintenanceListenerHandle. The initial connection and every replacement must be RESP3 and must accept CLIENT MAINT_NOTIFICATIONS ON moving-endpoint-type none; construction or reconnection fails if registration does not succeed.

For each valid MOVING notification, redis-tower waits until half of the server-supplied TTL and then creates a replacement through the original factory. That factory must reproduce all required authentication, TLS, URL, and other per-connection session setup. The none endpoint mode means the server does not select a destination: service discovery or endpoint changes must happen inside the factory. Set its connect timeout and reconnect budget so it can establish a usable connection within the remaining half of the TTL.

Once MOVING has been accepted, the worker lets any active batch make one completion attempt and does not replay it for the handoff. Newly queued work is held until replacement succeeds; if the reconnect budget is exhausted, that work fails instead of being sent on an unverified connection. Account for this pause when choosing queue capacity and end-to-end deadlines.

MIGRATING is observational only and does not reconnect. Use from_factory_with_maintenance_and_events and its lifecycle event bus if the application needs the notification metadata. This support does not relax command timeouts and does not cover pools, Cluster, blocking connections, or Pub/Sub.

Dropping the maintenance handle disables future handling without shutting down the client and cancels a half-TTL wait that has not committed. Prefer MaintenanceListenerHandle::shutdown().await during orderly shutdown: it waits for worker acknowledgement and, if replacement has already begun, for a connected or terminal result. Shut down the client separately.

Establish a baseline before tuning

Capture at least these workload characteristics under realistic concurrency:

  • operations per second and the mix of command names;
  • p50, p95, p99, and maximum command latency;
  • number of concurrent application requests;
  • request and response byte distributions, including worst-case collections;
  • fraction of commands that block or perform expensive server work;
  • auto-pipeline batch size and queue depth;
  • client-cache hit/miss, invalidation, eviction, size, and tracking-health data;
  • pool acquisition wait and in-flight commands, when using a pool;
  • reconnect state, offline queue depth, lifecycle-event lag, timeout, cluster redirect, and topology refresh rates;
  • Redis CPU, memory, evictions, connected clients, and slow-log entries.

Tune against the tail, not only average latency. A setting that improves mean throughput while growing the queue can make p99 latency and overload recovery worse.

Tune automatic pipelining

MultiplexedClient wraps AutoPipelineService. Its defaults are:

SettingDefaultEffect
max_batch_size100 framesUpper bound on one flush
batch_windowzeroFlush immediately after draining work already available
queue_capacity1024 requestsBound on pending worker requests
shed_load_on_fullfalseApply backpressure instead of returning QueueFull
response_timeoutNoneNo deadline for a whole batch response

Configure the worker before wrapping the connection:

use std::time::Duration;
use redis_tower::{AutoPipelineConfig, MultiplexedClient, RedisConnection};

let connection = RedisConnection::connect("127.0.0.1:6379").await?;
let client = MultiplexedClient::from_connection_with_config(
    connection,
    AutoPipelineConfig {
        max_batch_size: 128,
        batch_window: Duration::from_micros(250),
        queue_capacity: 2048,
        shed_load_on_full: false,
        response_timeout: Some(Duration::from_millis(500)),
        ..AutoPipelineConfig::default()
    },
);

Batch size and window

  • Increase max_batch_size when many commands arrive concurrently and Redis or network round trips dominate. Watch response size and p99 latency: larger batches occupy the connection longer and one slow response delays everything behind it.
  • Keep batch_window at zero for latency-sensitive or lightly concurrent traffic. A short non-zero window can improve batching for bursty writes, but it deliberately adds that wait to the first request in a batch.
  • Use the redis_tower.pipeline.batch_size histogram to verify that a change actually creates batches larger than one. Raising a limit that workloads never reach only adds theoretical capacity.

Queue capacity and overload policy

The queue is a concurrency bound, not a throughput control. A larger queue can absorb short bursts, but it also permits more memory use and queueing latency. Start from a measured burst budget and alert well before sustained depth reaches capacity.

With shed_load_on_full: false, Tower readiness waits for capacity and paces producers. This is the preferred default when upstream callers honor backpressure. Set it to true only when the application has an explicit load-shedding response for RedisError::QueueFull and would rather reject than wait.

MultiplexedClient::queue_depth() gives an instantaneous snapshot. With the metrics feature, spawn_queue_depth_exporter publishes redis_tower.pipeline.queue_depth; give each pipeline a stable low-cardinality name.

Response deadline

response_timeout covers the complete round trip for a flushed batch. When it expires, the in-flight batch fails and a factory-backed worker discards and rebuilds its connection. Set the deadline above the slowest command permitted on that worker. A deliberately long blocking command and a 200 ms response deadline are incompatible; separate those workloads instead of continually raising the shared deadline.

Use pools for isolation, not by default

A pool adds independent connections and server client state. More connections are useful only when one connection is the bottleneck or commands must be isolated; they also cost sockets, Redis memory, TLS handshakes, authentication, health checks, and more opportunities for uneven load.

use std::time::Duration;
use redis_tower::{ConnectionConfig, ConnectionPool, RedisConnection};
use redis_tower::pool::{DispatchStrategy, PoolConfig};
use redis_tower::reconnect::AddrConnectionFactory;

let factory = AddrConnectionFactory::new("127.0.0.1:6379")
    .with_connection_config(
        ConnectionConfig::new()
            .with_connect_timeout(Some(Duration::from_secs(3))),
    );

let pool = ConnectionPool::<RedisConnection>::connect_with_factory(
    PoolConfig::default()
        .name("blocking-jobs")
        .bounds(2, 8)
        .dispatch(DispatchStrategy::LeastConnections)
        .health_check_interval(Duration::from_secs(30))
        .idle_timeout(Duration::from_secs(60))
        .acquisition_timeout(Duration::from_millis(250)),
    factory,
).await?;

// Background work is never implicit. Retain each handle for its lifetime.
let reaper = pool.spawn_idle_reaper(Duration::from_secs(15));
let prober = pool.spawn_health_prober(Default::default());
let _ = (reaper, prober);

Pool guidance:

  • RoundRobin is the cheapest and works well when command latency is uniform.
  • LeastConnections is a better starting point for mixed short and long work; it avoids deliberately sending new work to the busiest slot.
  • Keep the default bounded acquisition timeout or choose a service-specific bound. Disabling it can turn saturation into an invisible hang.
  • Configure a factory when health checks should replace a dead slot. A closure- built pool without a retained factory can detect failure but cannot restore capacity automatically.
  • Dynamic sizing is opt-in. bounds(min, max) starts with min connections and creates more only when every live slot is contended. An idle_timeout is inert until spawn_idle_reaper is called; dropping its handle stops shrinking immediately.
  • Active health probing is also explicit. The default PING probe, ROLE probe, replication-lag-via-INFO probe, and custom HealthProbe implementations update metrics and PoolStats; they do not silently become a second circuit breaker or change dispatch policy. Run the byte-lag probe against primaries: it compares primary-side offsets for directly connected replicas and marks missing/offline replicas unhealthy. Replica-local INFO cannot establish the upstream primary’s current byte offset and is rejected conservatively.
  • Size from observed concurrency and Redis capacity. For blocking commands, the minimum useful size is usually the maximum number of blockers you intend to allow, plus deliberate headroom for health/control traffic. Enforce a higher- level concurrency limit rather than letting callers grow without bound.
  • Use a stable, unique PoolConfig::name; it becomes the pool metrics label.

Inspect pool.stats() for current/min/max size, idle and in-flight counts, active-probe health, replication lag, and cumulative idle reaping. Acquisition wait increasing while Redis command latency stays flat indicates pool contention. Both increasing usually means Redis or the network is the bottleneck, and adding connections may make it worse.

Set timeouts at distinct failure boundaries

One timeout cannot represent all phases of a Redis operation:

BoundaryAPIWhat it limits
TCP/Unix connectConnectionConfig::with_connect_timeoutInitial transport establishment
Reconnect attemptReconnectConfig::connect_timeoutOne factory call during recovery
Auto-pipeline responseAutoPipelineConfig::response_timeoutA whole flushed batch’s response
End-to-end typed callWithDeadline<Cmd>One absolute budget carried through middleware, queueing, routing, retries, pool waiting, and I/O
Tower callCommandTimeoutLayerA static upper bound; .with_request_deadlines() lets an earlier WithDeadline shorten it
Pool acquisitionPoolConfig::acquisition_timeoutWaiting to reserve a pool slot

Choose each from a remaining end-to-end latency budget. For example, pool wait plus command execution plus application processing must fit inside the request deadline; making all three independently equal to that deadline silently multiplies the worst case.

Carry one caller budget with WithDeadline and put CommandTimeoutLayer outside ExecutorService. This ordering lets the timeout layer inspect the typed command before ExecutorService adapts it, while the pool sees the same absolute deadline during acquisition:

use std::time::Duration;
use redis_tower::{
    CommandTimeoutLayer, ConnectionPool, ExecutorService, RedisConnection,
    WithDeadline,
};
use redis_tower::commands::Get;
use tower_layer::Layer;
use tower_service::Service;

let pool = ConnectionPool::connect(4, || {
    RedisConnection::connect("127.0.0.1:6379")
}).await?;

let mut service = CommandTimeoutLayer::new(Duration::from_secs(1))
    .with_request_deadlines()
    .layer(ExecutorService::new(pool));

let command = WithDeadline::after(Get::new("key"), Duration::from_millis(250));
let value = service.call(command).await?;

The earliest limit wins rather than starting a fresh budget at each stage:

  • nested WithDeadline envelopes retain the earliest absolute instant, and cloning or retrying them does not reset it;
  • CommandTimeoutLayer::with_request_deadlines uses the earlier of that instant and its static duration, returning RedisError::CommandTimeout;
  • a pool uses the earlier of the command deadline and its static acquisition timeout. The command deadline returns CommandTimeout; the pool’s own limit returns PoolAcquisitionTimeout.

The plain CommandTimeoutLayer remains generic over arbitrary Tower request types and applies only its configured duration. Opting in to request deadline extraction adds the RequestDeadline bound; all typed commands implement it, and raw RESP frames report no typed deadline.

CommandAdapter is the typed-to-frame boundary. Frame middleware inside the adapter cannot inspect command metadata, although the adapter enforces the absolute command deadline around its inner frame call. Likewise, MultiplexedClient::execute applies it across readiness and dispatch. Put metadata-aware middleware outside the adapter (normally outside ExecutorService) when the middleware itself needs to observe the deadline.

An already-expired command is rejected before the inner service is called. This prevents newly enqueueing a side effect after its caller budget is gone. Cluster clients retain the same deadline through node lookup, routing, redirects, and pinned-node execution; multiplexed Sentinel dispatch and the resilient client’s offline queue retain it through their own waits as well. If a deadline interrupts a direct or pool-member exchange after bytes reach the wire, that socket is quarantined instead of allowing its late response to poison a later request. On MultiplexedClient, the caller abandons its response but the worker retains the in-flight batch, reads its replies for alignment, and can keep a still-usable connection. In both cases, server execution remains unknown to the timed-out caller. The caller-level timeout still is not a substitute for worker-level response_timeout, which detects a connection whose batch never returns. Use both when a multiplexed production client needs bounded caller latency and bounded connection recovery.

Know the request lifecycle and execution boundary

Cancellation is local; Redis execution is remote. Use these states when deciding whether a failed or abandoned operation may be attempted again:

StateWhat redis-tower knowsSafe conclusion
Waiting for Tower readiness or worker queue admissionThe request has not been accepted by that componentDropping the future prevents that component from sending it
Accepted by ConnectionPool, waiting for an available memberThe pool counts the operation as accepted for graceful drain, but no member has received itDropping the future prevents wire dispatch and releases the accepted-operation count
Accepted but still in the auto-pipeline batch windowThe worker owns the request, but prunes a closed response channel before building the wire batchA request proven pruned is unsent
Partially or fully writtenRedis may have received some or all request bytesExecution is unknown unless a Redis reply later establishes an outcome
Waiting for the replyRedis may already have applied the commandA caller timeout or dropped future does not roll it back
Complete reply decodedThe worker/connection knows the Redis outcomeDelivery to a now-cancelled caller can still be lost locally; do not replay merely because the caller disappeared
ReconnectingThe failed exchange is not implicitly reconstructedOnly an explicit typed retry/offline policy or a protocol redirect can send a command again
Shutdown requestedNew admission is closed; behavior for accepted work depends on the owning handleAwait the documented shutdown/close method when draining matters

The direct RedisConnection implementation quarantines its socket when a typed call is cancelled after I/O begins, so a late response cannot satisfy a later command. The auto-pipeline worker drops cancelled requests that are still queued; after a batch starts it owns response alignment and continues to read the batch while the transport is usable. A batch response timeout or transport failure discards that connection and fails the affected callers. A factory may create a replacement for later work, but it does not silently replay that failed batch.

Client families add these explicit policies:

  • RetryClient replays only when the command reports idempotent() == true, the error is classified as retryable, and its retry/deadline budget remains. RedisError::is_retryable() says an attempt may be useful; it does not say a mutation is safe to repeat.
  • ResilientRedisClient fails new work during reconnect by default. Its opt-in offline queue admits only idempotent typed commands, has bounded capacity and per-command replacement-wire attempts, removes cancelled queued work, and quarantines a socket when cancellation happens on the wire.
  • Cluster MOVED/ASK and bounded transient-error handling are protocol replies, not guesses after a lost reply. The Cluster connection never replays a transaction redirect; it updates topology for a newly built future attempt. A connection loss with no reply remains unknown execution.
  • A ConnectionPool reserves one member for one execute call and releases its in-flight accounting on cancellation. It exposes no checkout/lease API, so it cannot hold WATCH or other connection-local state across separate calls. ConnectionPool::close stops admission and waits for accepted operations to finish.

Connection replacement restores only state owned by its constructor/factory: authentication, protocol version, database, TLS, decode limits, and any custom setup performed there. The caching clients own and re-establish their tracking sessions as described in Client-side caching. PubSubConnection::reconnect_with explicitly replays confirmed subscriptions, but messages published during the gap are lost. MonitorStream has no replay or resume cursor. Ad hoc AUTH, SELECT, HELLO, or tracking commands do not modify a factory and therefore are not automatically restored.

The auto-pipeline queued-cancellation contract and lost-reply contract distinguish unsent work from a lost non-idempotent reply. Treat any behavior not covered by the owning client’s source/tests as unknown rather than assuming another client family’s guarantee.

Do not retry every timeout automatically. Redis may have executed a write before the response was lost. The typed retry wrapper checks command idempotency and retryable error classification before replaying:

use std::time::Duration;
use redis_tower::RetryPolicy;

let client = client.retry(
    RetryPolicy::default()
        .max_retries(2)
        .base_delay(Duration::from_millis(25))
        .max_delay(Duration::from_millis(250)),
);

Keep jitter enabled in production so a Redis restart does not synchronize all application instances into a reconnect or retry storm. RetryService also checks the command’s absolute deadline while waiting for readiness, executing each attempt, and sleeping between attempts. A retry layer cannot reset or sleep past the caller’s WithDeadline, regardless of its position relative to the timeout layer. RetryClient::execute also bounds its initial readiness wait. With the lower-level Tower Service API, poll_ready necessarily occurs before the request is supplied, so request metadata starts governing the retry service once call begins; place a separate outer bound around a potentially blocking caller-side readiness wait.

Make reconnection replay connection state

Configure logical client identity in ConnectionConfig, not with a one-time CLIENT SETNAME on a borrowed connection:

async fn example() -> Result<(), redis_tower::RedisError> {
use redis_tower::{ConnectionConfig, ResilientRedisClient};
use redis_tower_cluster::MultiplexedClusterClient;

let standalone = ResilientRedisClient::connect_url_with_connection_config(
    "redis://127.0.0.1:6379/0",
    ConnectionConfig::new().with_client_name("redis-mcp"),
).await?;
let cluster = MultiplexedClusterClient::builder("127.0.0.1:7000")
    .client_name("redis-mcp")
    .connect().await?;
let _ = (standalone, cluster);
Ok(())
}

For a richer policy use ConnectionSetup::new().with_client_name(...), optionally .with_client_no_evict(true) (Redis 7.0+) or .with_client_no_touch(true) (Redis 7.2+), then attach it with ConnectionConfig::with_setup or either Cluster builder’s connection_setup. Builder calls replace the corresponding setting; connection_config replaces the entire config, so put name/setup calls after it when overriding it.

Built-in factories retain this policy across initial and replacement sockets. Both Cluster clients apply it to seed discovery, all connected masters and replicas, MOVED/ASK targets, topology-discovered nodes, and replacements. MultiplexedClusterClient::connect_to_node and its dedicated-session factories also inherit it. Setup does not create replica connections when the selected read preference only needs masters.

The order is optional best-effort library CLIENT SETINFO, authentication, URL database selection, RESP negotiation, declared setup, then Cluster replica READONLY. Explicit setup is not best effort: SETNAME, NO-EVICT, and NO-TOUCH run sequentially, with each +OK checked before another is sent. Unsupported commands, ACL denial, unexpected replies, or transport/protocol failure close that socket and return RedisError::Connection. No ordinary traffic is sent on a partially configured socket. Setup cancellation also discards the socket. Already healthy Cluster nodes may continue serving while a new node fails setup; that failed node is never admitted unconfigured.

Debug output redacts names, and setup errors show the failing command but omit arguments and server reply text (which can echo values). Names still appear in Redis CLIENT LIST; never include secrets. Grant +client|setname and only the other configured subcommands in the ACL. There is no silent downgrade on older servers. Bound full setup with a factory/reconnect timeout and configure a finite reconnect budget for persistent ACL/configuration failures; the socket connect timeout alone does not bound Redis setup replies.

The typed policy intentionally excludes arbitrary commands: AUTH/SELECT/HELLO, transaction state, reply suppression, tracking, Pub/Sub, and blocking operations have separate lifecycle owners. Custom factories remain responsible for setup they add themselves and can call RedisConnection::apply_setup after completing their own handshake. One-time commands on a borrowed socket are not recorded for reconnect. Sentinel and topology-neutral client builders do not yet expose this policy; this contract covers the standalone and Cluster APIs above.

Use a connection factory for long-running multiplexed clients. The factory is responsible for recreating all required session state on every connection:

  • URL username/password and database selection;
  • TLS roots and client certificates;
  • RESP mode and decode limits;
  • rotating credentials supplied by CredentialConnectionFactory, or other initialization performed by a custom factory.
use std::time::Duration;
use redis_tower::{
    AutoPipelineConfig, ConnectionConfig, KeepaliveConfig, MultiplexedClient,
    RespLimits,
};
use redis_tower::auto_pipeline::AutoPipelineReconnectConfig;
use redis_tower::reconnect::{ReconnectConfig, UrlConnectionFactory};

let connection_config = ConnectionConfig::new()
    .with_client_name("redis-mcp")
    .with_connect_timeout(Some(Duration::from_secs(3)))
    .with_keepalive(
        KeepaliveConfig::new()
            .with_idle(Duration::from_secs(30))
            .with_interval(Duration::from_secs(10))
            .with_probes(3),
    )
    .with_resp_limits(RespLimits {
        max_frame_size: 16 * 1024 * 1024,
        max_depth: 64,
    });

let factory = UrlConnectionFactory::new(
    "redis://default:secret@127.0.0.1:6379/0",
).with_connection_config(connection_config);

let reconnect = ReconnectConfig::default()
    .base_delay(Duration::from_millis(100))
    .max_delay(Duration::from_secs(5))
    .connect_timeout(Duration::from_secs(3));

let client = MultiplexedClient::from_factory(
    factory,
    AutoPipelineConfig {
        response_timeout: Some(Duration::from_secs(1)),
        ..AutoPipelineConfig::default()
    },
    AutoPipelineReconnectConfig::new(reconnect),
).await?;

Use CredentialConnectionFactory when authentication comes from a dynamic provider rather than a static URL:

use redis_tower::{
    AutoPipelineConfig, ConnectionConfig, CredentialConnectionFactory,
    MultiplexedClient, ProtocolVersion, StaticCredentials,
};
use redis_tower::auto_pipeline::AutoPipelineReconnectConfig;

// Replace StaticCredentials with an application or cloud CredentialProvider.
let factory = CredentialConnectionFactory::new(
    "127.0.0.1:6379",
    StaticCredentials::password("token"),
)
.with_connection_config(
    ConnectionConfig::new().with_protocol(ProtocolVersion::Resp3),
);

let client = MultiplexedClient::from_factory(
    factory,
    AutoPipelineConfig::default(),
    AutoPipelineReconnectConfig::default(),
).await?;

The credential factory opens each transport in RESP2, fetches credentials, runs AUTH, and then negotiates the protocol requested by ConnectionConfig. The complete sequence runs on initial connection and every reconnect, so each fresh socket consults the provider. The same factory also implements PoolFactory, allowing lazy or replacement pool slots to use the same setup.

An authentication rejection during connection establishment is handled narrowly: NOAUTH or WRONGPASS calls the provider’s force_refresh() and retries AUTH once. Providers shared by multiple clients or pool slots must synchronize concurrent refreshes. A StreamingCredentialProvider can proactively send replacement credentials to established standalone, multiplexed, pooled, Cluster, and Sentinel data sockets. That update is an explicit AUTH, never a replay of the command that exposed an authentication error. If an already-established connection returns NOAUTH or WRONGPASS for a user command, redis-tower returns the error to the caller.

The companion redis-tower-auth-aws and redis-tower-auth-azure crates implement ElastiCache IAM and Microsoft Entra ID token acquisition, caching, and proactive refresh. See Cloud and rotating credentials for complete setup and failure semantics.

max_retries counts retries after the first reconnect attempt: zero still allows one attempt, and n allows at most n + 1 attempts. The initial connection is outside that retry count. base_delay is applied before the first reconnect attempt, and connect_timeout bounds every factory call, including initial construction and each reconnect.

TCP keepalive detects dead peers below the Redis protocol, but its effective timing is platform-dependent and usually much slower than an application deadline. It complements response and reconnect timeouts; it does not replace them.

Decide whether to queue while offline

ResilientRedisClient fails later work while reconnecting by default. For a workload that benefits from waiting through a short outage, enable its bounded offline queue explicitly:

use redis_tower::{OfflineQueueConfig, ResilientRedisClient};

let client = ResilientRedisClient::connect_url_with_offline_queue(
    "redis://default:secret@127.0.0.1:6379/0",
    OfflineQueueConfig::new(256).with_max_replay_attempts(3),
).await?;

The queue is shared across clones and replays admitted work in queue-ticket order after one reconnect campaign. Admission is serialized, but concurrently first-polled futures can receive tickets in either scheduler order. Admission is deliberately narrow:

  • only typed commands marked idempotent can wait and replay;
  • non-idempotent commands and raw frame-service calls fail offline;
  • overflow returns RedisError::QueueFull immediately;
  • reconnect exhaustion returns RedisError::ReconnectFailed to queued work;
  • canceling a queued future releases its slot; and
  • canceling after a request reaches the wire quarantines that socket and reconnects before the next ticket runs, preventing a late response from being consumed by the wrong caller;
  • each command has a finite replacement-wire replay budget (three by default), so repeated successful connects followed by immediate disconnects cannot keep the head ticket alive forever; and
  • capacity zero preserves fail-fast behavior while reconnect continues in the background.

A WithDeadline budget includes admission, queue wait, reconnect, replay, and wire I/O. Expiry removes the ticket, and an expiry after dispatch quarantines the socket before the next ticket can run.

Size capacity from the number of operations the service can safely absorb over the outage window, not from normal throughput. Alert on offline_queue_depth() before it reaches capacity and use is_reconnecting() as a diagnostic or readiness input. Queueing increases outage latency and memory and can create a recovery burst, so fail-fast remains the safer choice for many request/response services. The replay-attempt budget is distinct from ReconnectConfig::max_retries, which limits retries after the first connection attempt within one campaign. Configure a finite reconnect budget and connect_timeout as well if each queued operation needs a bounded recovery path through an unreachable endpoint.

For a frame-level Tower stack, tower::buffer::Buffer can sit in front of ReconnectService. It cannot inspect Command::idempotent() after a command has become a raw frame, so it does not provide the typed queue’s replay-safety guarantee. Do not treat a generic frame buffer as authorization to replay mutations.

Bound protocol memory

RespLimits constrains the maximum buffered frame size and nesting depth. The defaults preserve normal Redis compatibility. Tighten them for shared, untrusted, or memory-constrained environments, but keep max_frame_size above the largest legitimate response (large values, collection reads, module results, and administrative output are common surprises).

Install limits through ConnectionConfig before connecting. For reconnecting clients put that config on the factory, so recovery cannot silently return to unbounded/default decoding. Cluster and Sentinel builders expose .resp_limits(...) and retain the limits across topology changes and failover.

Limit large collection work at the command level as well: prefer bounded scans and pagination over commands that materialize an entire unbounded keyspace or collection in one response.

Retry and circuit-breaker policy

Retries consume capacity precisely when a dependency is unhealthy. Keep retry budgets small, exponential, and jittered, and reserve them for commands whose typed implementation reports that replay is safe. A non-idempotent command is not retried by RetryClient, even when its error is otherwise retryable.

The Redis-aware circuit breaker counts transport failures and command deadlines that expire after dispatch, not readiness-only waits or ordinary Redis command errors such as WRONGTYPE. Start with the default and tune only from incident/load-test evidence:

use redis_tower::RedisCircuitBreakerConfig;

let client = client.with_circuit_breaker(RedisCircuitBreakerConfig {
    failure_threshold: 5,
    recovery_probe_interval: std::time::Duration::from_secs(5),
});
let health = client.circuit_breaker_handle();

Export health.state(), health.health_status(), or its metrics rather than using a successful process check as Redis readiness. Keep liveness independent from Redis so an outage does not restart every application instance in a loop; use Redis health for readiness and traffic admission.

Cluster tuning

MultiplexedClusterClient creates an auto-pipeline worker per node. Its builder accepts the same AutoPipelineConfig, a per-node reconnect config, a redirect budget, read preference, credentials, TLS, and RESP limits.

  • Leave ReadPreference::Master in place unless the application accepts replica staleness. Replica and PreferReplica change consistency, not only performance.
  • Use AdaptiveReplicaRouting when replica reads should compose locality, latency, and health. Redis does not advertise availability zones through CLUSTER SLOTS; configure local_zone and each replica_zone explicitly, using the post-host_override/address_map address. Unmapped replicas remain cross-AZ fallback candidates.
  • Adaptive routing samples new and recovered candidates round-robin, then uses inverse-EWMA weighting. Lower alpha smooths transient latency spikes; higher alpha reacts faster. Measure with production traffic before changing the default 0.2.
  • Consecutive transport, protocol, reconnect, and deadline failures drive ejection. Redis command errors are healthy responses and reset the failure count. Recovery is evaluated lazily after ejection_duration; there is no probe task to retain. minimum_healthy_replicas is a max-ejection floor: if every candidate would otherwise be excluded, the soonest-recovering entries are made eligible. The default floor of one favors availability and can continue trying a failing singleton replica; set it to zero with PreferReplica when master fallback is more important than retaining a replica target.
  • max_redirects bounds both latency and repeated work during resharding. Raising it can hide unstable topology while increasing tail latency; monitor MOVED, ASK, and topology refresh metrics first.
  • Use host_override or address_map when CLUSTER SLOTS advertises internal addresses that clients cannot reach. Validate every failover and resharding path, not only initial seed discovery.
  • Per-node metric labels are disabled by default. Enable them only when the diagnostic value is worth the additional series; redis-tower bounds concrete node-address labels and folds overflow into _OTHER.
  • Call shutdown() after stopping producers so all per-node workers can drain.

Sentinel tuning

Use MultiplexedSentinelClient::builder(...).connect_with_reconnect() for production failover. Configure Sentinel-hop and Redis-node credentials/TLS separately when the deployment does; the client re-queries Sentinel, verifies ROLE, and reconnects after a transport failure or a READONLY response from a demoted primary.

Test the complete failure path under load:

  1. stop or demote the current primary;
  2. verify in-flight errors are visible and bounded;
  3. verify Sentinel reaches agreement and the client discovers the new primary;
  4. verify credentials, TLS, and RESP limits still apply;
  5. verify writes resume without a retry storm or silent duplicate mutation.

Sentinel discovery availability depends on the set of addresses you provide. Use multiple independently placed Sentinel endpoints and monitor discovery failures separately from data-node failures.

To observe verified primary changes, pass a ConnectionEventBus through MultiplexedSentinelClient::builder(...).connection_events(events) before connect_with_reconnect(), or use MultiplexedSentinelClient::connect_with_reconnect_and_events(...). The first ROLE-verified primary establishes the baseline; only a later verified address change publishes ConnectionEvent::Failover. The comparison uses the exact endpoint strings Sentinel returns after ROLE verification; it is not durable node identity. DNS/textual aliases for one server can therefore look like a change, while replacing a node behind the same endpoint does not emit a failover event.

Observability that drives tuning

Enable the metrics feature and install one application-level metrics exporter. The built-in recorder can report:

  • db.client.operation.duration and redis_tower.commands for command latency/count/outcome;
  • redis_tower.pipeline.batch_size and redis_tower.pipeline.queue_depth for worker efficiency and saturation;
  • redis_tower.cache.events for bounded hit, miss, invalidation, and eviction counts;
  • db.client.connection.wait_time, db.client.connection.pending_requests, connection count/max, and pool lifecycle counters, including active-probe outcomes, replication lag, and idle reaping. Treat replication lag as current only while its paired *_observed freshness gauge is 1;
  • redis_tower.cluster.redirects, topology refresh count, and topology refresh duration.

Use stable pool and pipeline names. Do not put keys, user IDs, request IDs, or unbounded host strings into metric labels. Pair client metrics with Redis INFO, latency monitoring, slow log, CPU, memory, eviction, and network data; client-side latency alone cannot distinguish its cause.

For cached clients, alert on unhealthy caching before optimizing hit rate. While the invalidation receiver is being replaced, redis-tower clears and disables the local cache, so a falling hit rate paired with healthy Redis latency is an expected safety response rather than silent staleness. A lost fixed data worker on standalone CachedMultiplexedClient also clears the cache but requires constructing a new cached client. The Cluster cached client instead rebuilds complete data-worker and receiver coverage across every master before reopening its global gate, and requires a finite client_ttl as a backstop for ownership changes it does not observe. is_caching_healthy() distinguishes these states. See the client-side caching guide for tracking modes and failure semantics.

Tracing can be sampled more aggressively than metrics. Set a slow-command threshold on TracingLayer to preserve useful tail diagnostics without making every successful GET equally prominent.

Observe connection lifecycle without gating it

Attach a bounded ConnectionEventBus when constructing a reconnecting client. Subscribe first if the initial connection result matters because the stream does not replay old events:

use redis_tower::{
    AutoPipelineConfig, ConnectionEventBus, ConnectionEventRecvError,
    MultiplexedClient,
};
use redis_tower::auto_pipeline::AutoPipelineReconnectConfig;
use redis_tower::reconnect::{ReconnectConfig, UrlConnectionFactory};

let events = ConnectionEventBus::new(256);
let mut stream = events.subscribe();

let client = MultiplexedClient::from_factory_with_events(
    UrlConnectionFactory::new("redis://127.0.0.1:6379"),
    AutoPipelineConfig::default(),
    AutoPipelineReconnectConfig::new(ReconnectConfig::default()),
    events,
).await?;

tokio::spawn(async move {
    loop {
        match stream.recv().await {
            Ok(event) => record_connection_event(event),
            Err(ConnectionEventRecvError::Lagged { skipped }) => {
                record_event_lag(skipped);
            }
            Err(ConnectionEventRecvError::Closed) => break,
            Err(_) => break,
        }
    }
});

ConnectionEventBus::default() retains 64 events per subscriber; choose a larger explicit capacity only from observed event bursts and consumer latency. Publishing is synchronous and non-waiting, and every subscriber has an independent cursor. A slow consumer receives an explicit Lagged { skipped } error but never backpressures reconnect or failover. Treat the stream as an observability feed rather than a durable audit log, and ensure consumer work cannot grow an unbounded secondary queue.

Events distinguish initial connect failure, disconnect reason, each scheduled and failed reconnect attempt, successful reconnection, exhausted retry budget, intentional worker shutdown, and topology-confirmed endpoint failover. Disconnected { reason: Shutdown } is a producer’s terminal transition and is emitted separately after an earlier outage disconnect; when clients share one bus, other producers may continue publishing afterward. Human-readable connect/disconnect error strings can contain endpoint names, server responses, and other deployment details; redact them before exporting to systems with a broader audience. Aggregate events into stable counters and durations, and never put their free-form error text or addresses into unbounded metric labels. Sentinel publishes a single verified primary endpoint change automatically. Redis Cluster has multiple slot-scoped primaries, so its topology manager should call ConnectionEventBus::publish only when the application can state an honest, specific failover transition; do not infer one global failover from ordinary slot churn.

Graceful shutdown

Shutdown ordering prevents accepted work from disappearing in background queues:

  1. stop accepting new application requests and stop background producers;
  2. await tasks that own client clones, or retain a MultiplexedClient::shutdown_handle() before placing clones in a router;
  3. stop queue/pool metrics exporters that retain client clones;
  4. call ConnectionPool::close() to reject new pool work and drain accepted operations;
  5. consume the last multiplexed client with shutdown().await (cluster and Sentinel clients also expose shutdown);
  6. flush tracing/metrics providers and exit.

Calling shutdown() while another multiplexed clone remains alive returns without stopping the shared worker. Make ownership explicit in the application’s shutdown coordinator, and test SIGTERM behavior while commands are queued and in flight. For clone-heavy or trait-erased hosts, call the retained AutoPipelineShutdownHandle::shutdown().await instead. It atomically stops new admission across all clones, drains already accepted work while the connection is usable, cancels reconnect/backoff work, and joins the worker. Surviving client clones then return ConnectionClosed and can be dropped without the unfinished-worker warning.

A repeatable tuning loop

  1. Define an end-to-end latency SLO and an overload response.
  2. Load test the correct client shape with representative command and payload distributions.
  3. Set connect, acquisition, response, and caller deadlines from that budget; use WithDeadline when one absolute caller budget must cross pool waiting.
  4. Choose backpressure, explicit QueueFull shedding, and fail-fast versus bounded idempotent offline queueing; test saturation and recovery.
  5. Adjust one of batch size, batch window, queue capacity, pool size, or dispatch strategy at a time.
  6. Compare throughput and latency distributions together with queue/pool/Redis utilization.
  7. Inject connection loss, a black-holed endpoint, Redis restart, cluster resharding, and Sentinel failover as applicable.
  8. Verify idempotent retry behavior and ensure mutations are never duplicated.
  9. Exercise graceful shutdown with live work.
  10. Record the chosen values, workload assumptions, and alert thresholds next to application configuration so future changes have a baseline.

Serverless and scale-to-zero deployments

Serverless processes are often created speculatively, frozen between requests, or started while Redis is unavailable. Opening Redis during global initialization adds cold-start latency and can prevent an otherwise healthy process from starting. MultiplexedClient therefore has lazy constructors that create its lightweight worker immediately but defer DNS and socket work until the first command. Construct the client after entering a Tokio runtime (for example inside an async main before starting the handler loop); the synchronous constructor panics when no runtime is entered.

Lazy multiplexed client

For a Redis URL, including credentials, TLS, database selection, or a Unix socket, construct the client synchronously:

use redis_tower::MultiplexedClient;
use redis_tower::commands::Ping;

async fn invoke() -> Result<(), redis_tower::RedisError> {
let client = MultiplexedClient::connect_url_lazy(
    "redis://default:secret@cache.internal:6379/0",
);

// The first network connection is made here, not above.
client.execute(Ping::new()).await?;
Ok(())
}

Use MultiplexedClient::connect_lazy("host:port") for an unauthenticated TCP endpoint. For custom connection setup or reconnect policy, pass a ConnectionFactory to MultiplexedClient::from_lazy_factory:

use std::time::Duration;
use redis_tower::{AutoPipelineConfig, MultiplexedClient};
use redis_tower::auto_pipeline::AutoPipelineReconnectConfig;
use redis_tower::reconnect::{ReconnectConfig, UrlConnectionFactory};

let factory = UrlConnectionFactory::new("redis://cache.internal:6379/0");
let client = MultiplexedClient::from_lazy_factory(
    factory,
    AutoPipelineConfig::default(),
    AutoPipelineReconnectConfig::new(
        ReconnectConfig::default()
            .connect_timeout(Duration::from_secs(2))
            .base_delay(Duration::from_millis(50)),
    ),
);
let _ = client;

Lazy construction cannot report a connection error because it performs no connection attempt. The command that triggers a failed first attempt receives RedisError::ConnectionClosed; a later command tries the factory again. After the first successful connection, ordinary factory-backed reconnect behavior applies when that connection is lost.

Lazy, scale-to-zero pool

When commands need independent connections, a factory-backed pool can also start empty. It grows on command demand and acquisition contention, up to the configured maximum:

use std::time::Duration;
use redis_tower::{ConnectionPool, PoolConfig};
use redis_tower::reconnect::UrlConnectionFactory;

let pool = ConnectionPool::connect_lazy(
    PoolConfig::default()
        .max_size(8)
        .idle_timeout(Duration::from_secs(60)),
    UrlConnectionFactory::new("redis://cache.internal:6379/0"),
);
assert_eq!(pool.size(), 0); // no DNS or socket work yet

// Reaping is explicit. Retain this handle for as long as the task should run;
// dropping it stops the task and lets the pool remain at its current size.
let reaper = pool.spawn_idle_reaper(Duration::from_secs(15));
let _ = reaper;

The lazy pool’s minimum is zero. An idle reaper may therefore return it to zero after the configured timeout. Pool construction never starts the reaper implicitly. Use connect_with_factory plus PoolConfig::bounds(min, max) when a warm minimum is more important than scale-to-zero behavior.

Health and lifecycle semantics

A lazy client’s connection-health snapshot starts unhealthy. This means “not connected yet,” not that construction failed. It becomes healthy just before the first ConnectionEvent::Connected event. With from_lazy_factory_with_events:

  • construction emits no lifecycle event;
  • a failed deferred attempt emits ConnectionEvent::ConnectFailed;
  • the first successful deferred attempt emits ConnectionEvent::Connected;
  • failures after that point use the normal disconnect and reconnect events.

is_connection_healthy() reads that local snapshot without opening a socket; subscribe_connection_health() observes later changes. In contrast, health_check() sends PING and therefore triggers the deferred connection.

Do not use a Redis PING as the platform’s process-liveness check: it forces a cold instance to connect and defeats lazy initialization. Keep liveness local. Use a Redis-backed readiness or dependency check only when the platform should route traffic based on current Redis availability.

Invocation and shutdown pattern

Create one client per warm process and clone it into invocation handlers. Clones share one worker and connection, preserving auto-pipelining across concurrent invocations. Avoid constructing a client for every request.

Bound command duration using command deadlines or an outer timeout appropriate to the function’s remaining execution budget. Configure ReconnectConfig::connect_timeout; otherwise an unreachable endpoint may wait for the operating system’s TCP timeout.

When the runtime offers a reliable shutdown hook, call shutdown() after all client clones are gone so accepted work drains and the worker is joined. Many serverless platforms freeze or terminate without such a hook, so correctness must not depend on shutdown running.

Practical checklist

  • Store credentials in the platform’s secret store and prefer a redis:// or rediss:// URL so AUTH and SELECT are replayed after reconnect.
  • Keep the client outside the per-invocation handler and clone it cheaply.
  • Set a finite connection timeout below the invocation deadline.
  • Expect the first command on a cold process to include connection latency.
  • Keep process liveness local; use Redis-backed readiness deliberately.
  • Size concurrency so one auto-pipelined connection is appropriate, or use a bounded pool when individual blocking commands need independent sockets.
  • Never send blocking commands through MultiplexedClient; use a dedicated connection or a pool sized for the maximum simultaneous blockers. The pool dispatches commands; it does not expose a checkout for multi-call state.

See Production tuning for queue sizing, response timeouts, reconnect policy, and graceful shutdown details.

Pool health probing design

Status: accepted for issues #483 and #512.

Context

ConnectionPool already supports a lazy PING before using a connection that has been idle longer than PoolConfig::health_check_interval. That check is traffic-driven: it cannot report or repair a dead connection while the application is idle.

An earlier experiment at commits 4025d1f and 869556d integrated tower-resilience-healthcheck. The recovered design and review material show that package was built as a generic, health-aware multi-resource selector. It owned resource status, selection policy, thresholds, callbacks, and a background task. The pool subsequently developed different invariants:

  • each connection has a mutex and an in-flight reservation;
  • a retained PoolFactory can replace or add slots;
  • dispatch, acquisition deadlines, draining, and metrics are pool-native;
  • repository policy requires background work to be explicitly spawned and stopped by dropping an owned handle.

Putting the generic wrapper around the pool would duplicate selection state and would not let a probe safely coordinate with slot replacement or idle reaping. Putting each pool slot inside the wrapper would make two components compete to own dispatch and lifecycle state.

Decision

Active pool probing is implemented inside redis-tower rather than by adding tower-resilience-healthcheck as a dependency.

The public surface has two layers:

  1. HealthProbe<S> is a small async trait for one connection. Built-in PING, ROLE, and INFO-replication-lag probes cover the Redis-specific checks. The byte-lag probe targets a primary: it compares the primary offset with every directly connected replica offset reported by primary-side INFO. It treats no connected replicas, non-online replicas, and replica-local INFO as unhealthy rather than inventing a zero-byte lag.
  2. ConnectionPool::spawn_health_prober and ConnectionPool::spawn_health_prober_with are the only APIs that create a prober task. Both return HealthProberHandle; dropping the handle cancels the task, while shutdown cancels and joins it.

Creating a pool never starts a task. The existing lazy health check remains available for request-path validation and replacement. The active prober is observational: it updates per-slot state, PoolStats, and MetricsRecorder, but it does not silently change dispatch policy. Applications that need to stop sending work after an unhealthy result should compose the pool with the existing circuit breaker.

Dynamic pool shrinking follows the same lifecycle rule. Configuring an idle timeout does not start work; spawn_idle_reaper returns an owned IdleReaperHandle. Reaping only removes idle slots above the configured minimum. Scale-up happens synchronously on acquisition contention, so it does not require a background task.

Consequences

  • Probe and reaper tasks share the pool’s slot synchronization and cannot invalidate an in-flight connection.
  • Existing fixed-size configurations remain fixed by default (min == max == size). Dynamic sizing is opt-in through explicit bounds and requires a retained PoolFactory.
  • A lazy factory-backed pool can start at zero connections and creates its first slot on the first command.
  • The library owns a small amount of Redis-specific parsing for ROLE and INFO, but avoids a second resource-selection model and an additional dependency.
  • Health state is intentionally telemetry, not a second circuit breaker.

Command cookbook

This cookbook connects redis-tower’s typed command categories to the client or session that should execute them. Command structs remain available from redis_tower::commands::*; the same structs are also grouped into browsable category modules such as commands::strings, commands::streams, and commands::search.

Use the repository examples when you want a complete runnable program. The snippets here focus on the contract that is easy to miss when choosing a command, response type, or connection shape.

The Rust snippets require an external Redis server/topology, so mdBook marks them ignore instead of trying to execute them. Their matching functions live in examples/command_cookbook.rs, which CI compiles with the advertised workspace crates and features.

Pick ownership before commands

WorkRecommended ownerImportant constraint
Concurrent independent requestsMultiplexedClientOne shared auto-pipelined socket; do not run blocking or stateful multi-call sequences on it
One exclusive sequenceRedisConnectionMutable, single-owner socket; required for WATCH/read/build retries
Blocking requests spread across socketsConnectionPoolDispatches each command to an independent member, but exposes no checkout/lease API
Cluster or Sentinel trafficTheir multiplexed clientsRouting/failover is topology-aware; Cluster multi-key atomic work must stay in one slot
Subscription streamPubSubConnection / BinaryPubSubConnectionConsumes a dedicated connection and owns subscription state
MONITOR streamMonitorStreamPermanently changes its dedicated socket until it is dropped

See Production tuning for queueing, deadlines, reconnection, and shutdown. See Cloud and rotating credentials and Client-side caching for those specialized lifecycles.

Typed commands and response shapes

Command types declare their response at compile time. Nil, empty, integer, and server-error replies are not interchangeable:

async fn example() -> Result<(), Box<dyn std::error::Error>> {
use bytes::Bytes;
use redis_tower::{
    MultiplexedClient,
    commands::{Get, Incr, Set, SetOutcome, SetPreviousValue},
};

let client = MultiplexedClient::connect("127.0.0.1:6379").await?;

let missing_or_value: Option<Bytes> = client.execute(Get::new("profile:1")).await?;
let next: i64 = client.execute(Incr::new("visits")).await?;

// Finish with `with_outcome()` when conditional code needs to know whether
// Redis applied the write independently from the previous value.
let outcome: SetOutcome = client
    .execute(Set::new("profile:1", "ready").nx().get().with_outcome())
    .await?;
match &outcome.previous {
    SetPreviousValue::NotRequested => unreachable!("GET was requested"),
    SetPreviousValue::Missing => println!("the key did not exist"),
    SetPreviousValue::Value(value) => println!("previous bytes: {value:?}"),
}
let _ = (missing_or_value, next, outcome);
Ok(())
}

Get distinguishes a missing key (None) from an empty value (Some(Bytes::new())). Collection commands similarly preserve their documented ordering and optional entries. Redis command errors remain RedisError::Redis; they are not converted into an empty success value.

Set needs extra care because its options alter the wire reply. The base command keeps its original Option<Bytes> response for compatibility: ordinary OK and a rejected conditional write both become None. Conditional code should call with_outcome() after all other builder methods. Its SetOutcome separates SetStatus::{Applied, NotApplied} from SetPreviousValue::{NotRequested, Missing, Value}; an empty value is Value(Bytes::new()), not Missing.

Builder modestatusprevious
ordinary SETAppliedNotRequested
NX / XX without GETApplied or NotApplied from OK / nullNotRequested
ordinary SET GETalways AppliedMissing or Value
NX GETApplied when missing; otherwise NotAppliedMissing or Value
XX GETNotApplied when missing; otherwise AppliedMissing or Value

The strings, hashes, lists, sets, and sorted-set pages summarize their main result shapes.

Binary keys and values

Redis keys, fields, members, and stored values are byte strings. Typed builders accept byte slices, Vec<u8>, and Bytes through CommandArg; replies that represent Redis byte strings remain Bytes.

async fn example() -> Result<(), Box<dyn std::error::Error>> {
use bytes::Bytes;
use redis_tower::{MultiplexedClient, commands::{Get, HSet, Set}};

let client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let key = b"user:\xff".as_slice();

client.execute(Set::new(key, b"\x00\xfe\xff".as_slice())).await?;
client.execute(HSet::new(key, b"field".as_slice(), b"value\xff".as_slice())).await?;
let value: Option<Bytes> = client.execute(Get::new(key)).await?;
let _ = value;
Ok(())
}

Grammar remains typed or textual: cursors, stream IDs, JSON paths, Search queries, addresses, and SHA-1 digests are not arbitrary payload bytes. See Binary data and typed arguments for ownership and the exact per-family inventory.

Cursor iteration

SCAN is a cursor protocol, not a one-shot snapshot. Process every returned page, including pages with zero results, and stop only at cursor 0. A changing keyspace can produce duplicates.

async fn example() -> Result<(), Box<dyn std::error::Error>> {
use redis_tower::{MultiplexedClient, commands::Scan};

let client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let mut cursor = "0".to_string();
loop {
    let page = client
        .execute(Scan::new().cursor(cursor).match_pattern("user:*").count(100))
        .await?;
    let finished = page.is_finished();
    let next_cursor = page.cursor.clone();
    for key in page.results {
        println!("{key:?}");
    }
    if finished {
        break;
    }
    cursor = next_cursor;
}
Ok(())
}

On Redis Cluster, a keyless SCAN routed normally visits only one node. Use redis_tower_cluster::ScanClusterStream / ClusterScan for a cluster-wide walk and account for resharding semantics documented on that type.

Pipelines, transactions, and Cluster slots

A Pipeline batches network I/O but is not atomic. A Transaction submits a known MULTI/EXEC body atomically. Both keep typed results by index:

async fn example() -> Result<(), Box<dyn std::error::Error>> {
use bytes::Bytes;
use redis_tower::{Pipeline, RedisConnection, Transaction, TransactionResult};
use redis_tower::commands::{Get, Incr, Set};

let mut connection = RedisConnection::connect("127.0.0.1:6379").await?;

let pipeline = Pipeline::new()
    .push(Set::new("{account:1}:name", "Ada"))
    .push(Get::new("{account:1}:name"))
    .execute(&mut connection)
    .await?;
let name: &Option<Bytes> = pipeline.get(1)?;

let transaction = Transaction::new()
    .watch(["{account:1}:counter"])
    .push(Incr::new("{account:1}:counter"))
    .execute(&mut connection)
    .await?;
match transaction {
    TransactionResult::Committed(mut replies) => {
        let counter: i64 = replies.take(0)?;
        println!("{counter}");
    }
    TransactionResult::Aborted => { /* rebuild if appropriate */ }
}
let _ = name;
Ok(())
}

A direct Transaction::watch is safe on supported shared clients because the already-built WATCH/MULTI/EXEC exchange is submitted as one operation. A read/compute/write retry loop spans separate calls and must hold a dedicated RedisConnection for the complete WATCH/read/build/EXEC window. A ConnectionPool does not expose a connection lease and cannot provide that ownership.

Cluster atomic transactions and multi-key commands require every key to hash to the same slot. Hash tags such as {account:1} make that relationship explicit. A split helper can distribute independent reads, but splitting is not atomic and is not a substitute for co-located keys.

Run the complete examples with:

cargo run -p redis-tower-examples --example pipeline
cargo run -p redis-tower-examples --example transaction

Streams, consumer groups, and blocking reads

The application owns acknowledgement. Create a group once, read on an isolated connection, finish processing, then acknowledge the exact entry ID:

async fn example() -> Result<(), Box<dyn std::error::Error>> {
use redis_tower::RedisConnection;
use redis_tower::commands::{XAck, XGroupCreate, XReadGroup};

let mut connection = RedisConnection::connect("127.0.0.1:6379").await?;
connection
    .execute(XGroupCreate::new("events", "workers", "0").mkstream())
    .await?;

let streams = connection
    .execute(XReadGroup::new("workers", "worker-1", "events").count(10).block(1_000))
    .await?;
for (_stream, entries) in streams {
    for entry in entries {
        // Persist the side effect before acknowledging this ID.
        process(&entry).await?;
        connection.execute(XAck::new("events", "workers", entry.id)).await?;
    }
}
async fn process(_entry: &redis_tower::commands::StreamEntry) -> Result<(), Box<dyn std::error::Error>> { Ok(()) }
Ok(())
}

XREAD/XREADGROUP BLOCK and list blocking commands hold their socket until a reply or Redis timeout. Do not put them on a MultiplexedClient, where they would stop the shared worker. Prefer a finite Redis blocking timeout so the task can observe shutdown; dropping a dedicated connection closes the socket. A caller timeout after bytes reach Redis cannot prove that no entry was delivered, so processing and acknowledgement must be idempotent where a lost reply matters.

See examples/streams.rs for a runnable basic stream flow and Production tuning for shutdown ordering.

Raw, custom, and module replies

RawCommand is the binary-safe escape hatch. It returns Frame by default; .query::<T>() applies a FromFrame decoder without changing the request:

async fn example() -> Result<(), Box<dyn std::error::Error>> {
use redis_tower::{MultiplexedClient, commands::RawCommand};

let client = MultiplexedClient::connect("127.0.0.1:6379").await?;
let count: i64 = client
    .execute(RawCommand::new("SCARD").arg("members").query())
    .await?;
let members: Vec<bytes::Bytes> = client
    .execute(RawCommand::new("SMEMBERS").arg("members").query())
    .await?;
let _ = (count, members);
Ok(())
}

Choose the decoder from the documented RESP shape: Option<T> for nil, Vec<T> for ordered arrays, tuples/maps only when the protocol shape warrants them, and raw Frame when a module varies by protocol version or options. Do not normalize nil into empty bytes or unordered results into an invented ordering.

The Cluster router knows the layouts of supported typed commands. An unknown custom command uses the legacy first-argument key convention for ordinary single-command routing; that is safe only when the first argument really is the complete routing key. Keyless node-local administration should use an explicit dedicated node connection. Unknown multi-key atomic layouts should not be guessed.

Feature modules have both Cargo and server prerequisites. Applications using the redis-tower facade need its commands-* feature names; applications depending directly on redis-tower-commands use the shorter names:

redis-tower featureredis-tower-commands featurePublic categoryServer capability
commands-jsonjsoncommands::jsonRedisJSON
commands-searchsearchcommands::searchRedis Search
commands-bloombloomcommands::bloomRedisBloom Bloom/Cuckoo
commands-sketchsketchcommands::sketchRedisBloom CMS/Top-K
commands-tdigesttdigestcommands::tdigestRedisBloom T-Digest
commands-timeseriestimeseriescommands::timeseriesRedisTimeSeries
commands-vector-setsvector-setscommands::vector_setsRedis Vector Sets

The facade’s default commands-stack feature and the command crate’s default stack feature each enable all seven command families. Neither installs those capabilities on the Redis server.

Pub/Sub and MONITOR

Publishing is an ordinary typed command. Receiving messages changes the socket into a stateful session, so construct the session from a fresh connection:

async fn example() -> Result<(), Box<dyn std::error::Error>> {
use redis_tower::{BinaryPubSubConnection, RedisConnection};
use tokio_stream::StreamExt;

let connection = RedisConnection::connect("127.0.0.1:6379").await?;
let mut subscriber = BinaryPubSubConnection::from_connection(connection)?;
subscriber.subscribe_bytes(&[b"events\xff".as_slice()]).await?;

if let Some(message) = subscriber.next().await {
    let message = message?;
    println!("{:?}: {:?}", message.channel, message.payload);
}
Ok(())
}

reconnect_with installs a replacement connection and replays subscriptions that were previously confirmed. Messages published while disconnected or before Redis processes the replayed subscription are not recovered. Messages that arrive after server-side subscription processing but before the client reads the confirmation are buffered and delivered. Cluster regular Pub/Sub is tied to an explicitly selected node; sharded Pub/Sub follows slot ownership through its separate API.

MonitorStream::new likewise consumes a fresh connection. There is no resume cursor and no automatic recovery of commands observed during a gap. MONITOR is an expensive debugging facility, not an application event stream.

Run the complete Pub/Sub example with:

cargo run -p redis-tower-examples --example pubsub

Standalone, Cluster, Sentinel, and Universal entry points

Choose the topology at construction; command types stay the same:

async fn example() -> Result<(), Box<dyn std::error::Error>> {
use redis_tower::MultiplexedClient;
use redis_tower_cluster::MultiplexedClusterClient;
use redis_tower_sentinel::MultiplexedSentinelClient;
use redis_tower_client::UniversalClient;

let standalone = MultiplexedClient::connect_url("redis://127.0.0.1:6379").await?;
let cluster = MultiplexedClusterClient::connect("127.0.0.1:7000").await?;
let sentinel = MultiplexedSentinelClient::connect(
    &["127.0.0.1:26379"],
    "mymaster",
).await?;
let universal = UniversalClient::connect_url("redis://127.0.0.1:6379").await?;
let _ = (standalone, cluster, sentinel, universal);
Ok(())
}

Use the explicit builders when Sentinel-hop and data-node credentials differ, when Cluster TLS/address mapping needs customization, or when read routing and reconnect policy matter. UniversalClient is the convenient application-level enum; topology-specific APIs remain necessary for cluster-wide scan, dedicated node sessions, sharded Pub/Sub, and advanced failover controls.

The runnable cluster and sentinel examples document their local fixtures. TLS/authentication and rotating credentials remain canonical in Cloud and rotating credentials.

Administration and diagnostics

commands::server, commands::acl, commands::cluster, and commands::diagnostics contain typed operator commands. Managed services often restrict CONFIG, DEBUG, MODULE, ACL mutation, failover, and persistence commands. Detect permission/server errors rather than assuming local Redis configuration.

Connection-state commands deserve extra caution. Configure authentication, RESP version, selected database, tracking, and decode limits through the connection/factory APIs so every reconnect repeats the same setup. Issuing AUTH, SELECT, HELLO, or tracking commands ad hoc on one shared socket does not update its factory’s future connections.

For continuous operations, prefer the metrics and lifecycle-event APIs over polling expensive diagnostic output. Production tuning is the canonical guide for observability, health probes, reconnect state, and graceful shutdown.

Feature-gated and versioned families

Cargo feature availability, Redis server availability, and command version are three separate checks. Building with commands-stack on the redis-tower facade (or stack on redis-tower-commands) proves that the Rust builders are present; it does not prove a deployment has RedisJSON, Search, Bloom, or TimeSeries. Native Array commands require Redis 8.8+, and Vector Sets require a supporting Redis 8 server.

Treat ERR unknown command or module-specific capability errors as deployment configuration/version failures. Gate startup with an explicit capability probe when a feature is required, and keep fallback behavior visible rather than silently dropping an option or changing a reply shape.

Redis command capability ledger

This report separates four claims that cannot be reduced to one coverage percentage: command-name availability, typed semantics, RESP wire support, and session/topology behavior. A typed name does not prove every option, reply shape, protocol form, or routing mode.

The machine-readable source is conformance/command-capabilities.json. Unverified fields are explicit backlog, not implied support.

Pinned provenance

  • Redis documentation version: 8.8
  • Redis documentation revision: ad12d9cd6d10
  • Pinned metadata entries: 554
  • Scoped command names: 506
  • Redis documentation metadata license: CC-BY-NC-SA-4.0 (separate from the Rust crates’ MIT/Apache-2.0 license)
  • Ordinary generation and CI are offline; each vendored metadata file is SHA-256 verified from the detail manifest. The manifest records both the upstream byte digest and the digest after documented trailing-whitespace normalization.

Availability dispositions

DispositionScoped namesMeaning
dedicated_api6Supported by a stateful or higher-level API, not a stateless builder.
intentionally_unsupported2Deliberately outside the client API contract.
internal_protocol2Issued internally by routing/session machinery.
raw_only8Available only through an explicitly routed raw command.
typed_builder488Dedicated typed Command implementation.

These counts describe availability only. They are not combined with semantic, wire, or operational verification into a headline score.

Scoped names without typed builders

CommandDispositionPublic APIRationale
ASKINGinternal_protocolredis_tower_cluster::ClusterConnection
redis_tower_cluster::MultiplexedClusterClient
Cluster redirect handling sends ASKING and the redirected command as one connection-local exchange; a standalone builder would make unsafe separation easy.
CLUSTER COUNT-FAILURE-REPORTSraw_onlyredis_tower_commands::RawCommandNode-local cluster administration remains an explicitly routed raw operation.
CLUSTER MIGRATIONraw_onlyredis_tower_commands::RawCommandRedis 8 cluster migration administration is not part of automatic client routing and must target a deliberate node.
CLUSTER MYSHARDIDraw_onlyredis_tower_commands::RawCommandThis node-local topology inspection command requires an explicitly selected cluster node.
CLUSTER SLOT-STATSraw_onlyredis_tower_commands::RawCommandThis node-local Redis 8.2 administration command is available through a deliberately routed raw request.
CLUSTER SYNCSLOTSraw_onlyredis_tower_commands::RawCommandCluster slot synchronization mutates node-local state and is intentionally not exposed as an ordinary routed builder.
JSON.DEBUG HELPraw_onlyredis_tower_commands::RawCommandJSON.DEBUG MEMORY has a typed builder; the module help variant is an operator-facing raw reply and remains raw-only.
PSUBSCRIBEdedicated_apiredis_tower::PubSubConnection
redis_tower::BinaryPubSubConnection
redis_tower_cluster::ClusterPubSubConnection
redis_tower_cluster::BinaryClusterPubSubConnection
Pattern subscription changes a dedicated socket into Pub/Sub mode and is owned by the streaming session API.
PSYNCintentionally_unsupported—Replication handshakes and replication-stream consumption are outside the request/response client contract.
PUNSUBSCRIBEdedicated_apiredis_tower::PubSubConnection
redis_tower::BinaryPubSubConnection
redis_tower_cluster::ClusterPubSubConnection
redis_tower_cluster::BinaryClusterPubSubConnection
Pattern unsubscription is part of the stateful Pub/Sub session and its acknowledgement stream.
READONLYinternal_protocolredis_tower_cluster::MultiplexedClusterClientBuilderReplica-read connections issue READONLY during connection setup so reconnects restore the state before routing reads.
READWRITEraw_onlyredis_tower_commands::RawCommandThe public cluster clients own their read-routing mode; ad hoc connection-state reversal is only available on an explicitly owned raw session.
SSUBSCRIBEdedicated_apiredis_tower::PubSubConnection
redis_tower::BinaryPubSubConnection
redis_tower_cluster::ShardedClusterPubSubConnection
redis_tower_cluster::BinaryShardedClusterPubSubConnection
Sharded Pub/Sub uses a dedicated standalone session or a slot-aware Cluster session that follows the channel slot’s owner.
SUBSCRIBEdedicated_apiredis_tower::PubSubConnection
redis_tower::BinaryPubSubConnection
redis_tower_cluster::ClusterPubSubConnection
redis_tower_cluster::BinaryClusterPubSubConnection
Subscription changes the connection protocol mode and is exposed through a dedicated streaming session rather than a stateless command builder.
SUNSUBSCRIBEdedicated_apiredis_tower::PubSubConnection
redis_tower::BinaryPubSubConnection
redis_tower_cluster::ShardedClusterPubSubConnection
redis_tower_cluster::BinaryShardedClusterPubSubConnection
Sharded unsubscription is owned by the dedicated standalone or slot-aware Cluster Pub/Sub session and its confirmation handling.
SYNCintentionally_unsupported—Legacy replication handshakes and replication-stream consumption are outside the request/response client contract.
TRIMSLOTSraw_onlyredis_tower_commands::RawCommandThis node-local cluster maintenance operation is deliberately excluded from automatic key routing.
UNSUBSCRIBEdedicated_apiredis_tower::PubSubConnection
redis_tower::BinaryPubSubConnection
redis_tower_cluster::ClusterPubSubConnection
redis_tower_cluster::BinaryClusterPubSubConnection
Unsubscription and its confirmations are part of the stateful Pub/Sub session.

Audited command semantics and operations

19/506 scoped names currently have a command-specific semantic or operational audit. The remaining rows are recorded as unverified in the machine ledger.

CommandSemanticsBinary inputsOptions / responseRouting / sessionEvidence
ASKINGnot_applicablenot_applicableOwned by redirect machinery rather than an application command.ASK target node; same connection and immediately followed by redirected command; One-shot connection state must not be separated from the redirected command.topology.cluster-redirect
AUTHverifiedpartialpassword or username/password; status; Prefer constructor credential configuration so reconnect restores authentication.connection-local; connection setup; Ad hoc AUTH does not mutate a connection factory.session.connection-setup
session.credential-provider
BLPOPverifiedverifiedone or more keys plus timeout; optional key/value pair; None on Redis timeout; Caller cancellation after write cannot prove that no element was delivered.same slot in Cluster; exclusive connection recommended; a multiplexed response deadline can fail a slow command and retire its worker; Standalone value and Redis-timeout replies are verified. Cluster routing and non-stalling concurrent isolation are not command-specifically verified here.command.blocking-dedicated
CLIENT TRACKINGpartialnot_applicableowned by caching clients; raw ad hoc variants are not replayed; status; Caching clients enforce their own tracking configuration and invalidation lifecycle.per data connection; connection-local tracking state; Tracking state is restored only by the caching client that owns it.session.client-tracking
EVALpartialverifiedkeys must be declared separately from arguments; raw Frame; The script defines the reply shape; applications must decode deliberately.all declared keys in one Cluster slot; stateless command; Declared-key encoding is verified in standalone differential tests; Cluster routing remains unaudited here.command.eval-declared-keys
FT.SEARCHpartialpartialFtSearch supports NOCONTENT, VERBATIM, WITHSCORES, LIMIT, RETURN and SORTBY; other documented options are not exposed by this builder; raw Frame; The separate high-level SearchClient::search API performs typed result deserialization. Both require the search Cargo feature and Redis Search server capability.module/topology dependent; stateless command; Module availability is proven separately from compilation.module.behavior-differential
GETverifiedverifiednone; Option<Bytes>; nil and empty values remain distinct; RESP2/RESP3 byte strings normalize to the same public shape.single key slot; stateless command; Read-only and idempotent.command.binary-and-scalar
topology.cluster-core-routing
HELLOverifiedpartialRESP version, optional auth and client name; server capability map; Prefer connection protocol configuration so reconnect repeats negotiation.connection-local; connection setup; Protocol version is a socket property.session.connection-setup
HGETALLverifiedverifiednone; Vec<(Bytes, Bytes)> parsed from RESP2 flat arrays or RESP3 maps; Binary fields and values are preserved; the public response is an ordered pair vector, not a map type.single key slot; stateless command; Large hashes can produce large replies.command.collection-stream-shapes
topology.cluster-core-routing
JSON.GETpartialverified for keysJSONPath and formatting grammar are textual; optional raw JSON bytes; Requires the JSON Cargo feature and RedisJSON server capability.single key slot; stateless command; Module capability and standalone behavior are verified; Cluster module routing is not audited here.module.behavior-differential
MGETverifiedverifiedone or more keys; ordered Vec<Option<Bytes>> preserving nil and empty entries; The result position corresponds to the input key position.same slot for atomic server command; explicit split helpers are non-atomic; stateless command; Standalone response order is verified; cross-slot split-helper order is verified separately, but same-slot Cluster MGET remains unaudited here.command.binary-and-scalar
topology.cluster-core-routing
MONITORverifiednot_applicablenone; stream of monitor lines; MONITOR permanently changes the socket mode until it is dropped.explicit server node; dedicated MonitorStream; No reconnect resume cursor; commands during a gap are lost.session.monitor-dedicated
session.credential-provider
SCANpartialpartialcursor, MATCH and COUNT are live-tested; TYPE is encoded by the builder but is not live-tested here; cursor plus a possibly empty page of binary keys; Continue until cursor zero; changing keyspaces can repeat elements.one selected node; use ClusterScan for all nodes; cursor sequence without connection affinity; A normal Cluster SCAN is not cluster-wide.command.scan-cursor
SELECTverifiednot_applicabledatabase index; status; Cluster does not support multiple logical databases.connection-local; connection setup; Configure the database in the connection factory for reconnect replay.session.connection-setup
session.credential-provider
SETpartialverifiedEX/PX, NX/XX and GET are typed; EXAT/PXAT/KEEPTTL and Redis 8.4 IFEQ/IFNE/IFDEQ/IFDNE are unimplemented; Set::with_outcome returns independent Applied/NotApplied status and NotRequested/Missing/Value previous-value state; base Set preserves the legacy Option<Bytes> response; RESP2/RESP3 live and parser coverage exercises ordinary SET plus all applied/rejected NX/XX branches with and without GET, including missing, empty and binary previous values. Expiry combinations are not covered.single key slot; stateless command; Non-idempotent option combinations are not replayed by typed retry policy.command.set-options
command.binary-and-scalar
topology.cluster-core-routing
SSUBSCRIBEverifiedverifiedone or more shard channels in one slot; subscription confirmations and message stream; Messages during a disconnect before server-side resubscription are not recovered.standalone server or current Cluster owner of the shared slot; dedicated standalone or sharded Cluster Pub/Sub connection; Cluster cross-slot subscription sets are rejected before dispatch.session.pubsub-reconnect
session.credential-provider
SUBSCRIBEverifiedverifiedone or more channels; subscription confirmations and message stream; Confirmed subscriptions can be replayed, but disconnected-gap messages are lost.explicit standalone or Cluster node; dedicated Pub/Sub connection; Not a stateless multiplexed command.session.pubsub-reconnect
session.credential-provider
WATCHverifiedverifiedone or more keys; status before EXEC commit/abort outcome; Use the Transaction abstraction for an already-built atomic exchange.same Cluster slot; dedicated connection for read/compute/write retry loops; A pool has no multi-call connection lease.command.pipeline-transaction
command.binary-and-scalar
topology.cluster-core-routing
XREADGROUPpartialpartialgroup, consumer, streams, IDs, COUNT and BLOCK; NOACK is not exposed by the builder; typed stream/entry vector for the tested non-blocking consumer-group lifecycle; The builder accepts byte-preserving CommandArg inputs, but the live XREADGROUP evidence uses text identifiers. Blocking timeout shape is not command-specifically verified here.same slot in Cluster; exclusive connection recommended when BLOCK is used; Standalone consumer-group behavior is verified; blocking isolation and Cluster routing remain unaudited here.command.stream-consumer-group

Wire protocol contracts

ContractStatusCurrent behaviorEvidence
configured-resource-limitssupportedFrame wire size, aggregate cardinality and nesting depth are preflighted before recursive frame materialization, independently of fragmentation.protocol.boundaries
protocol.fragmentation
fragmentation-equivalencesupportedWhole-buffer and arbitrarily fragmented decoding have the same semantic outcome and error category for the tested corpus.protocol.fragmentation
resp2-framessupportedRESP2 scalars, nulls and aggregates are decoded within configured size/depth limits.protocol.boundaries
protocol.fragmentation
resp3-core-framessupportedSupported RESP3 scalar and fixed aggregate forms are decoded and normalized by typed response parsers.protocol.boundaries
command.collection-stream-shapes
resp3-fixed-attributesrejectedFixed attributes fail closed without consuming bytes; attachment to the following value remains planned in issue #705.protocol.boundaries
resp3-streamed-formsrejectedStreamed string/aggregate headers, chunks and terminators fail closed rather than becoming independent replies; bounded assembly remains planned in issue #705.protocol.boundaries
protocol.fragmentation

Logical behavior evidence

A source test, a configured workflow selector, and a dated successful run are separate evidence states. A later checkout must use its own attached checks rather than inheriting the observed result below.

Behavior command binary and scalar

Logical ID: command.binary-and-scalar.

Raw RESP2/RESP3 scalar, nil, empty, binary and numeric replies are compared with redis-rs. Separate typed live tests verify byte-preserving builders and public GET/MGET decoding, including ordered nil-versus-empty results, under forced RESP2 and RESP3.

Behavior command blocking dedicated

Logical ID: command.blocking-dedicated.

Typed BLPOP returns a key/value pair when data is available and None after a Redis timeout on an exclusive connection. A separate multiplexed raw BLPOP test proves the configured response deadline returns CommandTimeout; the differential BLMOVE case is immediate because its source list is preloaded.

Behavior command collection stream shapes

Logical ID: command.collection-stream-shapes.

Raw hash, collection and Stream replies are compared with redis-rs using independent keys and controlled normalization. Separate typed live and parser tests verify HGETALL’s public pair-vector shape, empty response and RESP3 map normalization.

Behavior command eval declared keys

Logical ID: command.eval-declared-keys.

Typed EVAL preserves binary declared keys and arguments and matches redis-rs for the tested nested reply.

Behavior command pipeline transaction

Logical ID: command.pipeline-transaction.

Pipeline ordering and transaction commit/abort outcomes are compared with redis-rs without reusing mutating datasets.

Behavior command scan cursor

Logical ID: command.scan-cursor.

Standalone SCAN cursor iteration exercises MATCH and COUNT until cursor zero; Cluster tests prove a plain keyless SCAN reaches one node while ClusterScan covers the full keyspace.

Behavior command set options

Logical ID: command.set-options.

SET outcome mode distinguishes applied and rejected NX/XX writes independently from unrequested, missing, empty and binary previous values. RESP2/RESP3 parser and live tests cover the complete supported outcome matrix; NX+GET and XX+GET branches are also compared with redis-rs. Expiry combinations remain outside this behavior row.

Behavior command stream consumer group

Logical ID: command.stream-consumer-group.

A typed XREADGROUP call reads new entries in a standalone consumer-group lifecycle, after which typed pending, acknowledgement, claim and autoclaim operations verify ownership transitions.

Behavior module behavior differential

Logical ID: module.behavior-differential.

Module capability preflight is followed by RedisJSON/Search/TimeSeries/probabilistic/Vector behavior and selected redis-rs reply comparisons.

Behavior protocol boundaries

Logical ID: protocol.boundaries.

Independent wire fixtures prove frame limits, aggregate preflight, nesting limits and fail-closed handling for fixed attributes and streamed forms.

Behavior protocol fragmentation

Logical ID: protocol.fragmentation.

Property and fuzz oracles compare whole-buffer decoding with arbitrary fragmentation plans and stable error categories.

Behavior session client tracking

Logical ID: session.client-tracking.

Managed caching clients install tracking, invalidate external writes, reject unsafe connection-state commands and restore a lost redirect connection.

Behavior session connection setup

Logical ID: session.connection-setup.

Authentication, RESP negotiation and selected database state are exercised through connection setup and reconnect-aware paths.

Behavior session credential provider

Logical ID: session.credential-provider.

One shared credential provider backs fresh standalone and Cluster sockets plus dedicated Pub/Sub and MONITOR construction. Provider URL setup preserves transport, selected database and RESP negotiation; lookup, forced-refresh, transport and session setup can share one deadline. A forced-refresh failure is redacted and stops setup before SELECT, HELLO or application traffic. Established ordinary clients can reauthenticate through owned tasks with explicitly documented owner-specific failure retirement, while Pub/Sub reconnects with confirmed binary subscriptions and MONITOR terminates/reopens rather than accepting unsafe generic in-band AUTH. Provider errors and refresh-task tracing omit third-party error text and retain authentication classification.

Behavior session monitor dedicated

Logical ID: session.monitor-dedicated.

A dedicated MONITOR session parses streamed events and preserves binary command arguments while another connection executes the command.

Behavior session pubsub reconnect

Logical ID: session.pubsub-reconnect.

Dedicated regular and sharded Pub/Sub sessions preserve binary names, replay confirmed subscriptions and follow tested Cluster ownership changes.

Behavior topology cluster core routing

Logical ID: topology.cluster-core-routing.

Cluster live tests route binary single-key GET/SET, HGETALL, ordered non-atomic split MGET helpers and slot-pinned WATCH transactions through public APIs.

Behavior topology cluster redirect

Logical ID: topology.cluster-redirect.

Cluster clients keep ASKING with the redirected command, follow ASK then MOVED, and update slot ownership during a live reshard.

Offline validation and refresh

Validate references, dispositions, source builders, pinned file hashes, and generated output without network access:

python3 scripts/generate_command_capabilities.py --check
python3 scripts/generate_command_coverage.py --check

To review a newer Redis documentation snapshot, fetch an explicit commit into a separate directory and compare it before changing provenance:

python3 scripts/fetch_command_metadata.py --ref <redis-docs-commit> \
  --output-dir /tmp/redis-command-metadata
python3 scripts/generate_command_capabilities.py \
  --compare-metadata /tmp/redis-command-metadata

The comparison reports added/removed names and any command-definition change, including scope, routing keys/flags, arity, versions, summaries, history, and arguments/options. Updating the five vendored files, their hashes, and every new disposition is a reviewed repository change. The downstream MCP project’s Redis 8.10.1 application ledger has a different denominator and is not substituted for this client ledger.

Binary data and typed command arguments

Redis keys and most data arguments are byte strings. They are not required to be UTF-8. redis-tower-commands uses CommandArg for binary-safe typed inputs wherever Redis treats an argument as opaque data: keys, stored values, fields, members, patterns, script arguments, channels, and module payloads.

use bytes::Bytes;
use redis_tower::commands::{Get, HSet, Set, XAdd};

let string_key = b"user:\xff".as_slice();
let hash_key = b"profile:\xff".as_slice();
let stream_key = b"events:\xff".as_slice();

client
    .execute(Set::new(string_key, vec![0x00, 0xfe, 0xff]))
    .await?;
let value: Option<Bytes> = client.execute(Get::new(string_key)).await?;

client
    .execute(HSet::new(
        hash_key,
        b"embedding".as_slice(),
        vector_bytes,
    ))
    .await?;
client
    .execute(XAdd::new(stream_key).field(b"payload", payload))
    .await?;

Ownership and compatibility

CommandArg owns shared Bytes storage so command values remain Send + 'static and can be cloned for Tower middleware. Conversions behave as follows:

InputStorage behavior
Bytesshares the existing allocation
Vec<u8>takes ownership without copying the payload
Stringtakes ownership without copying the UTF-8 payload
&[u8], byte arrays, &str, &Stringcopies once into owned storage
CommandArgmoves the value; cloning shares storage

Ordinary string calls such as Get::new("key") and Set::new(String::from("k"), "v") remain direct. Generic wrappers that want to accept both text and bytes should use impl Into<CommandArg>. A wrapper intentionally restricted to text can retain impl Into<String> and pass value.into() to a builder. As with the old impl Into<String> API, avoid an unnecessary .into() on a literal at the call site because an impl Trait parameter gives the conversion no unique target; pass the literal directly or use CommandArg::from(...).

The standard inputs accepted through Into<String> remain accepted directly, including Box<str>, char, and &mut str. A downstream newtype that only implemented Into<String> is the direct source-compatibility exception: either convert it to String before calling the builder or implement From<YourType> for CommandArg. This intentional break must not ship in a 0.1.x patch release. It requires redis-tower-commands 0.2.0 and redis-tower 0.2.0, with workspace and downstream dependency requirements updated when that release is prepared.

Stream field names and stream names in XREAD/XREADGROUP results are now Bytes, as are StreamMessage.stream and its field names. The previous String representation used lossy UTF-8 conversion and could not round-trip an input that the builder now accepts. Consumer and group names in stream metadata are also Bytes. Stream IDs remain String because Redis defines their grammar as decimal numbers separated by -.

Semantic inventory

The table is an argument-semantic inventory, not a search for Rust String fields. “Text” means Redis assigns grammar or an encoding to the value; “opaque” means Redis compares, stores, hashes, or routes the exact bytes.

FamilyOpaque inputsDeliberately structured/text inputsTyped status
Stringskeys and stored values, comparison valuesinteger/float increments, expiry numbers, digest syntaxCommandArg throughout
Hasheskeys, field names, field valuesincrements and expiry numbersCommandArg throughout
Lists and non-blocking multi-list commandskeys, elements, pivotsindexes, counts, LEFT/RIGHT and BEFORE/AFTER tokensCommandArg throughout
Setskeys and memberscounts and limitsCommandArg throughout
Sorted setskeys and members; lex bounds contain a binary member after their Redis prefixscores, rank indexes, aggregation and direction tokensall builder byte strings use CommandArg; Redis validates bound grammar
Streamsstream keys, field names/values, group and consumer namesentry IDs/cursors, trim thresholds and option tokensbuilder byte strings use CommandArg; stream keys and field names decode as Bytes
Keys and key lifecyclekeys, rename/copy destinations, patterns, serialized RESTORE payloadTTLs, database numbers, SORT grammaropaque positions use CommandArg throughout
Blocking commandslist/zset keystimeouts and direction tokenskeys use CommandArg throughout
Scancollection keys and glob patternscursors, count, TYPE tokenkeys/patterns use CommandArg; results are Bytes
Scripting and functionsscript bodies, KEYS and ARGV, function payloadsSHA-1 hex digests, function/library names and subcommand grammaropaque positions use CommandArg; cached Script helpers also provide binary argument methods
Pub/Sub commands and sessionschannels, patterns, payloadsPUBSUB subcommandstyped commands and dedicated sessions are binary-safe
Geo and HyperLogLogkeys, members/elementscoordinates, unitsopaque positions use CommandArg throughout
Bitmap, transaction, diagnostics, and server helperskeys, WATCH keys, tracking prefixes and data-bearing argumentsnumeric offsets plus administrative grammaropaque positions use CommandArg; PING/ECHO echo responses are exact Bytes
ACL and cluster administrationACL passwords and simulated command arguments; keys supplied to CLUSTER KEYSLOTACL usernames/rules/categories, node IDs, addresses, command grammar and numeric slotsdeliberate split: opaque data uses CommandArg; operational identifiers remain text-first
Redis 8.8 arrayskeys, values, predicatesindexes and option grammaralready binary-safe
Bloom/Cuckoo, sketches, t-digestkeys and item payloadscapacities, probabilities and numeric observationsopaque positions use CommandArg throughout
JSONkeysJSONPath and serialized JSON syntaxkeys use CommandArg; structured JSON interfaces stay text/serde-oriented
Searchdocument keys, suggestion payloads, tag values and vector blobsindex/schema/query syntax and autocomplete suggestion textopaque inputs and outputs preserve bytes; Redis Search can terminate NUL-containing suggestions and normalize malformed UTF-8
Time serieskeys and label names/valuestimestamps, reducers and retention/configurationopaque inputs and returned keys/labels preserve bytes
Vector setskeys and element names; vector bytesJSON attributes and filter/query expression grammarkeys/elements use CommandArg; returned elements are Bytes

RawCommand::arg remains the universal escape hatch for commands or extension syntax outside the typed surface. It accepts AsRef<[u8]>, can retain a typed response with .query::<T>(), and does not perform lossy conversion.

Response decoding

Responses corresponding to opaque values preserve exact bytes. This includes keys, members, stream names/fields, Search documents/suggestions/payloads/tag values, TimeSeries keys/labels, Vector Set elements, TopK items, and echoed PING/ECHO messages.

Redis-defined textual metadata remains String: scan cursors, stream IDs, SHA-1 digests, JSON text, query plans, server diagnostic reports, ACL usernames and rules, Cluster node descriptions and IDs, addresses, and command/config names. Some of these existing metadata parsers use UTF-8 replacement for a malformed server reply; that behavior cannot affect an opaque value because those positions are classified separately above.

ACL usernames/rules/categories and Cluster node IDs/addresses intentionally remain text-first. Redis exposes them as operational configuration grammar, not application key/value payloads. In contrast, ACL passwords and ACL DRYRUN arguments, and the key accepted by CLUSTER KEYSLOT, preserve exact bytes.

The client serializes Search suggestion strings as exact bulk-string bytes, but Redis Search implements them as text: an embedded NUL is a terminator and malformed UTF-8 can be normalized. Suggestion dictionary keys and payloads remain binary-safe. Suggestion responses preserve the exact bytes returned by the server rather than applying another client-side conversion.

Cluster routing

Typed commands serialize a CommandArg directly into RESP bulk strings. The Cluster key extractor reads those same frame bytes, so slot hashing and hash-tag extraction happen before any decoding and use the exact key bytes. Braces inside invalid UTF-8 keys therefore have the same routing meaning as braces in an ASCII key.

RESP codec support and validation

The protocol crate decodes one bounded RESP reply at a time. Its public Frame type comes from resp-rs and can represent more wire forms than the client currently accepts. This page is the disposition contract used by the property tests, fuzz oracle, and reviewed seed corpus.

Frame disposition

Wire formDecode dispositionGenerated and fixture coverage
Simple strings and errorsSupported; line contents cannot contain an unescaped CRLFGenerated leaves and independent pipeline fixtures
IntegersSupported across the full i64 range; overflow is rejectedGenerated leaves plus malformed/overflow boundaries
Bulk strings and RESP2 null bulk stringsSupported with byte-exact binary payloadsGenerated binary leaves and framing-looking payload fixtures
Blob errorsSupported with byte-exact payloadsGenerated leaves and mixed aggregate fixtures
Finite doublesSupportedGenerated finite values
inf, -inf, and nan doublesSupported and normalized to canonical SpecialFloat framesGenerated canonical values and independent wire fixtures
Booleans and RESP3 nullSupportedGenerated leaves
Big numbersSupported as their wire bytesGenerated signed decimal values and independent fixtures
Verbatim stringsSupported when the payload has a three-byte format followed by :Generated binary content under the txt format and malformed fixtures
Fixed arrays, null arrays, sets, maps, and pushesSupported recursively within the configured size and depth limitsGenerated mixed aggregates, raw fixtures, and exact boundary tests
Fixed attributesRejected without consuming inputDeterministic top-level and nested rejection fixtures
Streamed string/blob/verbatim headers, aggregate headers, chunks, and terminatorsRejected without consuming inputEvery token at top level and in aggregates, across every split point
Malformed tags, scalars, lengths, terminators, or overflowRejected without consuming inputDeterministic regressions and arbitrary-byte fuzzing
Frames over max_frame_size or max_depthRejected before materialization and without consuming inputExact limits, every split point, declared cardinality, and nested maps

Encoding a Frame is intentionally less restrictive because outbound frames are application-built. In particular, the re-exported enum can serialize attributes and streamed variants that this decoder rejects. Do not infer receive support from the variants exposed by resp-rs.

Allocation and ownership

The codec preflights the first reply without materializing a frame. Once that reply is complete and within its configured size/depth limits, decoding copies exactly that bounded first frame into immutable Bytes for resp-rs; it does not clone or copy the entire unread receive batch. Aggregate elements still have decoded representation overhead, so max_frame_size is a wire-size bound, not an exact heap budget. This is a bounded-copy path, not an end-to-end zero-copy claim. The decode ownership and copy evidence compares the historical whole-buffer clone, the production first-frame copy, and a split/shared alternative, including retained-frame lifetime behavior.

The declared-cardinality regression uses a test-only scan observer: an array or map declaring ten million entries visits exactly its one available header and never spills the inline nesting stack. This deterministically verifies constant scan work without attempting a dangerous allocation. A separate test-only materialization counters wrap the public RespCodec::decode path: modest incomplete array and map declarations leave both frame and byte counts at zero, while a complete array is the positive control. A retained mixed pipeline proves the byte count is exactly the sum of completed frame extents, and byte-at-a-time delivery proves incomplete fragments are never copied for parser input.

Fragmentation oracle

The decode_chunked fuzz target decodes each input twice under identical limits: once with the entire wire available and once with a repeating sequence of arbitrary one-to-64-byte chunks. It compares:

  • every successful frame in order, represented by its canonical re-encoding;
  • the bytes retained at the first incomplete or rejected frame; and
  • the terminal disposition, including the exact parser error variant.

Canonical bytes avoid treating NaN != NaN as a protocol disagreement. A frame-size error compares by disposition rather than the diagnostic size field: a prefix can prove that the cap is exceeded before the complete extent is known. If incremental decoding rejects a prefix, the harness appends the not-yet-delivered bytes without retrying the decoder, matching a connection that closes on a protocol error while preserving a whole-buffer-comparable remainder.

The property suite separately builds up to sixteen valid replies, including nested mixed aggregates, then checks arbitrary partitions and truncation points. Raw wire fixtures are kept independent of the crate encoder so a paired encoder/decoder defect cannot be the only evidence.

Fuzz input and retained evidence

Both fuzz targets use this compact envelope:

[max_depth, max_size_hi, max_size_lo, plan_length, plan..., RESP wire...]

The plan is capped at 16 bytes and each value maps to a non-zero chunk size. Inputs shorter than four bytes are treated as raw wire with conservative limits. Reviewed seeds live as named hexadecimal fixtures under fuzz/corpus-seeds/; generated corpus entries and crash artifacts remain ignored working state. Every decode_chunked seed starts with a one-byte plan, so its intended boundary behavior is exercised before mutation discovers new partitions.

Pull requests run ten seconds per target. The weekly and manually dispatchable Scheduled Fuzzing workflow accepts a one-to-3,600-second duration per target. Each job retains its reviewed starting seeds, final corpus, crash artifacts, log, toolchain versions, source SHA, resolved Cargo manifests and lockfiles, their hashes, duration, exit code, and terminal status for 90 days. The manifest is replaced with running before seed preparation and changes to passed only after a zero process exit, the requested wall time, and libFuzzer’s normal completion record. Graceful signal exits are explicitly rejected, so an interruption cannot masquerade as a completed pass.

For local reproduction:

python3 scripts/prepare_fuzz_corpus.py \
  fuzz/corpus-seeds/decode_chunked fuzz/corpus/decode_chunked
cargo +nightly fuzz run decode_chunked fuzz/corpus/decode_chunked -- \
  -max_total_time=300

Historical mutation triage

The September 21 protocol report predated the current preflight limit scanner and scored a library-only mutation scope at 58.5%. Its survivors are evidence to classify, not a score that can be compared with a differently scoped run.

Historical survivor familyClassification and current evidence
Delete or alter map depth traversalReal assertion gap. nested_maps_consume_the_same_depth_budget_as_other_aggregates now rejects a two-level map at depth one and accepts it at depth two. Fixed attributes are separately unsupported.
Scan inside a blob instead of skipping its declared payloadReal framing and depth gap. Binary fixtures contain streaming and aggregate tokens, while unit and integration regressions prove those bytes remain opaque.
Alter CRLF scanning at a fragment boundaryReal fragmentation gap. Every strict prefix of independent frames is incomplete, exact split tests cover bounded frames, and the generated/fuzz oracles compare arbitrary partitions.
Change length/cardinality arithmetic or overflow handlingA real gap when the mutation changes rejection, consumption, or allocation bounds; deterministic tests cover oversized declarations, impossible cardinality, integer overflow, and malformed terminators. A mutation that changes only the early diagnostic lower-bound size while preserving its documented class is equivalent under this contract.
Timeout or mutation outside a reachable supported dispositionTool limitation or excluded protocol feature, to be reported separately rather than counted as covered. Attributes and streaming remain intentional fail-closed exclusions.

A future score comparison must record the exact source SHA, cargo-mutants version, test command, package scope, and raw outcomes. The scheduled mutation workflow already retains that evidence; this work does not relabel the old number as a current baseline.

RESP decode ownership and copy evidence

RespCodec deliberately copies one complete, bounded frame after preflight. It does not copy the entire unread receive buffer, and it does not claim a zero-copy decode path. This page records why that ownership model is the current production choice and how to reproduce its evidence.

Decision

The historical decoder passed src.clone().freeze() to resp-rs for every reply. BytesMut::clone() copies the full unread slice, so a batch of equal frames caused the shrinking remainder to be copied repeatedly. The current decoder first determines the complete frame extent, copies only those bytes, parses that immutable frame, and advances the receive buffer only after parse success. Total parser-input copying is therefore linear in decoded wire bytes.

A split_to(frame_len).freeze() design removes that copy and is faster in the focused harness, but it couples response lifetime to receive-buffer lifetime. A small response retained after the unread tail is drained can pin the entire original allocation. It also complicates the codec’s promise that parse errors leave input unchanged: restoring an already-split prefix requires rebuilding the buffer on the exceptional path.

The bounded first-frame copy is the selected tradeoff because it:

  • avoids the historical quadratic copied-byte mechanism;
  • limits a retained response to frame-sized backing storage rather than the whole receive batch;
  • preserves unchanged bytes on incomplete input and all errors; and
  • requires no unsafe code in the production protocol crate.

The first-frame implementation arrived with the earlier bounded-preflight work. The evidence here validates and regression-tests that choice; it is not a claim that this evidence round alone produced an end-to-end client speedup.

Retained evidence

The checked raw evidence was recorded on macOS arm64 at source b7256ba04acda830368632af1f2450f746029c81 with Rust 1.98.1, Cargo 1.98.1, bytes 1.12.1, resp-rs 0.1.8, tokio-util 0.7.19, and Criterion 0.5.1. The source tree was clean when the run began. Allocation values are requested layout bytes observed through an instrumented System allocator, not RSS. “Allocated” is the sum of successful initial layout sizes plus positive growth during reallocation; “live” is the change in requested live layout bytes. Neither value includes allocator metadata or size-class rounding. Durations are 30 raw within-process samples of 20 iterations with allocation instrumentation active. They are directional mechanism evidence, not Redis client throughput or a cross-host performance claim.

The retained-batch rows below keep every decoded frame until the batch is complete. “Copied wire” counts only bytes copied to create parser input; “allocated” and “live” also include the requested layout sizes for receive storage, Frame values, and aggregate vectors.

Scenario and strategyCopied wireAllocated through decodeLive after decodeMedian time
512 five-byte replies; historical whole-buffer copy656,640 B708,352 B708,352 B54,307 ns
512 five-byte replies; production first-frame copy2,560 B54,272 B54,272 B39,685 ns
512 five-byte replies; split/shared comparison0 B39,464 B39,464 B12,722 ns
128 mixed replies (71,296 wire bytes); historical whole-buffer copy4,630,640 B4,718,832 B3,576,560 B99,510 ns
128 mixed replies; production first-frame copy71,296 B158,720 B158,544 B16,425 ns
128 mixed replies; split/shared comparison0 B85,160 B85,160 B6,389 ns

The lifetime counterexample is the reason the fastest microbenchmark strategy is not the production default. The input contains a 23-byte first frame and a 262,155-byte unread tail. After the unread buffer is dropped while the head frame remains alive:

StrategyCopied wireBytes pinned by the retained headMedian time
Historical whole-buffer copy262,178 B262,202 B8,819 ns
Production first-frame copy23 B47 B3,275 ns
Split/shared comparison0 B262,218 B3,637 ns

These numbers should not be extrapolated to a complete Redis workload. The production path also performs the bounded preflight scan, while the comparison strategies receive known fixture extents. Later parser or buffer changes should rerun the same artifact rather than preserving these numbers as a permanent marketing baseline.

Regression and benchmark coverage

The protocol unit suite counts both completed materializations and their exact wire bytes. A mixed pipeline containing binary bulk data, a push, scalars, and an aggregate proves that production materializes the sum of frame sizes once, not the sum of shrinking unread suffixes. A byte-at-a-time case proves that no copy happens until a frame is complete. Existing boundary tests continue to cover limits, malformed input, unchanged error bytes, binary payloads, pushes, and response order.

Criterion now includes:

  • dropped and batch-retained lifetimes for 100 simple replies;
  • dropped and batch-retained mixed payloads from zero through 4 KiB; and
  • a smaller mixed fixture with payloads through 256 bytes, delivered in 1-, 7-, 64-, and 1,024-byte fragments.

The raw allocation artifact repeats those four fragment sizes over the same smaller 112-frame fixture, retains the decoded batch, and records receive-buffer reallocations, net acquired requested layout bytes, and the change in requested live layout bytes. Every fragmentation plan decodes frames equivalent to the contiguous production path and reports exactly one frame’s wire bytes per completed materialization.

Delivery chunkCopied wireAllocated through decodeBuffer reallocationsMedian time
1 B5,616 B20,720 B686,262 ns
7 B5,616 B20,720 B624,997 ns
64 B5,616 B20,720 B312,590 ns
1,024 B5,616 B22,256 B111,136 ns

To reproduce the checked allocation/copy artifact from a clean checkout, use an absolute output path because Cargo runs the custom benchmark from the package directory:

evidence_output="$PWD/conformance/codec-copy-evidence.json"
cargo bench -p redis-tower-protocol --bench codec_allocation -- \
  --evidence --samples 30 --iterations 20 --check \
  --output "$evidence_output"

The --check mode verifies equivalent decoded frames, exact deterministic copy counts, lower production allocation than the historical mechanism, and the retained-tail tradeoff. cargo test --benches runs a short silent smoke; an explicit --evidence invocation is required for retained timing samples. Run the Criterion timing suite separately with:

cargo bench -p redis-tower-protocol --bench codec

Client comparison

This page compares documented client contracts, not feature counts. A checkmark can hide the questions that matter in production: whether a connection is exclusive or multiplexed, which commands are replayed after reconnect, whether a cache is managed for you, and whether a “typed” API fixes the response type or lets the caller choose one.

The comparison was checked on 2026-09-25 against the versions and revisions below. Links point to upstream documentation or pinned source. Treat a claim as unknown when the cited material does not establish it; do not infer absence from an empty cell.

ClientBaseline used here
redis-tower186719e; published workspace versions range from 0.1.1 to 0.1.3
redis-rs1.7.0 (2a29a8f)
Fred10.1.0 (29d4790)
Lettuce4461844 plus the rolling Lettuce wiki, checked on the date above
go-redis2fc3ccd
StackExchange.Redis0a92ae4
ioredisb597303
node-redisd3eac3d
redis-pyba6976b

Rust clients

redis-tower, redis-rs, and Fred overlap heavily, but expose different contracts.

Contractredis-towerredis-rs 1.7Fred 10.1
Typed command APICommand values fix their response types; pinned Get returns Option<Bytes>. Raw commands can opt into a decoder.AsyncTypedCommands provides typed convenience methods; Cmd::query_async::<T> retains caller-selected conversion.Command traits expose typed arguments and caller-selected response types; see the versioned interfaces module.
Composition pointFrame services implement Tower Service; pinned middleware sources cover timeouts, circuit breaking, tracing, and metrics.Unknown / not compared: the 1.7 API index documents commands and connections, but this review did not establish a general middleware contract.Config and the versioned interfaces are client-specific extension points; no cross-client middleware equivalence is asserted here.
Shared connectionPinned MultiplexedClient is cloneable and auto-pipelines concurrent commands over one connection.MultiplexedConnection is cloneable and multiplexed.Client is a cloneable handle over the driver’s connections.
PoolPinned ConnectionPool owns independent connections with fixed, dynamic, or lazy population and explicit health/lifecycle controls.Optional r2d2 and bb8 integrations are listed in the pinned 1.7 feature definitions. A multiplexed connection is not itself a checkout pool.Pool distributes commands across clients; stateful interfaces are intentionally omitted.
TopologyStandalone, Cluster, and Sentinel are separate clients; pinned UniversalClient provides one enum-backed entry point.Standalone plus the cluster-async and sentinel features are defined in the pinned 1.7 manifest.ServerConfig covers centralized, Cluster, Sentinel, and Unix deployments.
Reconnect and replayPinned ResilientRedisClient single-flights reconnects with bounded backoff. Ordinary in-flight commands are not silently replayed; stateful APIs document their own restoration rules.ConnectionManager reconnects in the background; the command that observes the dropped connection errors and later commands wait for the replacement connection.ReconnectPolicy controls reconnect delay. Fred’s pinned ConnectionConfig defaults to three command attempts, and reconnect_once flushes previously in-flight commands from the retry buffer. The redelivery metric counts requests sent again after close while awaiting a response; pinned command policy and attempt checks show that fail-fast, attempt exhaustion, or no reconnect policy stops redelivery.
Client-side cachingPinned standalone and master-routed Cluster caching manages tracking, invalidations, bounds, metrics, and reconnect fail-closed behavior.The experimental cache-aio feature integrates caching with multiplexed, connection-manager, and async Cluster connections.TrackingInterface exposes RESP3 tracking and invalidation commands; it is not documented as a managed local cache.
Redis Stack and newer familiesPinned redis-tower-modules covers JSON, Search, TimeSeries, Bloom/Cuckoo, and Vector Sets.The pinned 1.7 feature definitions include JSON, Bloom, and Vector Sets; Search is marked unfinished.The versioned interfaces module documents JSON and Search interfaces. Other families remain unverified here.
CredentialsPinned CredentialProvider supports reconnect/push rotation; sibling crates provide AWS IAM and Microsoft Entra providers.The pinned 1.7 feature definitions include token-based authentication and Microsoft Entra support.Versioned Config exposes a credential provider when its feature is enabled.
ObservabilityPinned production guidance documents stable tracing fields, metrics-facade recording, pool/queue snapshots, and bounded opt-in per-node Cluster labels.Unknown / not compared: no client-wide tracing/metrics contract was verified from the 1.7 API index; application wrappers are outside this comparison.Pinned TracingConfig is feature-gated; versioned MetricsInterface exposes redelivery, queue, latency, and payload-size measurements. Export to an external metrics backend remains application work.
RESP behaviorPinned redis-tower-protocol documents RESP2/RESP3 negotiation, push demultiplexing, and explicit rejection of unsupported attributes/streamed aggregates.RESP2 and RESP3 are supported; URL selection is documented by versioned ConnectionInfo.Pinned Config::version defaults to RESP2 and can require RESP3; TrackingInterface requires RESP3 for tracking.

The important typed-API distinction is not “typed versus untyped.” All three clients provide typed conveniences. redis-tower command values own a specific decoder, while redis-rs and Fred also make caller-selected response conversion a common path.

Lessons from widely used clients

These clients are useful design references, not direct feature-score opponents.

ClientDocumented contract worth carrying into a review
LettuceIts execution reliability documentation treats reconnect replay, duplicates, ordering, and transaction behavior as separate guarantees. A generic “reconnects automatically” claim is insufficient.
go-redisThe pinned README documents automatic pooling, experimental streaming credentials, RESP3 client caching limits, and OpenTelemetry instrumentation. Its experimental auto-pipeline warns that retrying a whole batch can duplicate non-idempotent work.
StackExchange.RedisConnectionMultiplexer is designed to be shared and reused; cheap database handles and multiplexing are deliberately different from checking out a socket per request.
ioredisThe pinned README documents standalone, Sentinel, and Cluster connections, offline queues, retry behavior, and auto-pipelining. Queueing before connect and replaying after a disconnect are different decisions.
node-redisIts pinned FAQ says already-sent commands reject when the socket closes because Redis may have executed them; unsent commands can remain queued for reconnect. That boundary is a useful retry model.
redis-pyThe pinned unified response proposal separates wire-protocol shape from the public response contract. Differential tests must normalize client-facing values, not merely compare raw RESP frames.

What this page does not claim

  • It does not rank clients. API fit, deployed topology, language ecosystem, and operational familiarity usually matter more than a total capability count.
  • It does not publish performance conclusions. The repository’s pinned benchmark publication protocol defines workloads, client versions and adapters, topology, connection count, timeouts, metadata, and reproducibility requirements. A result is evidence only for the measured environment and scenario.
  • It does not turn undocumented behavior into “no.” Unknown behavior stays unknown until a stable upstream source or a reproducible experiment supports a narrower statement.

When updating this page, pin the source revision, record the check date, link the precise upstream contract, and describe reconnect, retry, replay, pooling, caching, and response typing independently.

Test conformance

This report maps redis-tower’s test surface to the claims it supports. It is intended to be citable at a particular commit: the compiled inventory is generated from the checkout, while the matrices below link each broader claim to its test code and workflow. A compiled test is not automatically a passing test, and a scheduled job is not evidence that a particular run was green; use the checks attached to the commit for that conclusion.

This page inventories compiled tests, configured workflow expansion, and observed execution evidence. Those are distinct from the logical client behaviors they exercise. The generated command capability ledger records those behaviors and separately tracks command-name availability, typed semantics, RESP wire support, and session/topology claims; neither page treats one dimension as proof of another.

Compiled inventory

The generated block is the only place this page records test counts. Keeping the numbers mechanical prevents a prose headline from drifting as tests are added, renamed, or split into new binaries. Refresh it with python3 scripts/generate_test_conformance.py; CI checks that the committed block matches the current source tree.

Compiled integration inventory

The scoreboard-scoped packages compile 756 integration tests across 45 test binaries. 212 tests are marked #[ignore] because they need explicit infrastructure; workflow selectors, rather than the annotation alone, determine whether they run.

SurfacePackageBinariesCompiled#[ignore]Pull requestScheduledNo workflow selector
Standaloneredis-tower3252005202290
Clusterredis-tower-cluster311210611200
Sentinelredis-tower-sentinel288888800
Modulesredis-tower-modules7201820180
Distributed primitivesredis-tower-primitives11601600
Total457562127562470

Counts are unique compiled tests. A test selected by both pull-request and scheduled workflows appears in both cadence columns, but only once in the compiled total.

Cross-backend command parity

The shared command corpus produces 210 compiled parity tests: 35 test cases × 6 client/topology expansions.

SurfaceTest binary / variantCompiled cases
Standaloneintegration / resp335
Standaloneintegration / root expansion35
Clustercluster_integration / multiplexed35
Clustercluster_integration / root expansion35
Sentinelsentinel_integration / multiplexed35
Sentinelsentinel_integration / root expansion35

Independent-client differential coverage

The independent redis-rs corpus contains 12 compiled test entry points. These are counted separately from shared-client topology expansions.

SurfaceTest binaryCompiled testsDefault
Standalonedifferential_redis_rs9normal
Clustercluster_integration1ignored
Sentinelsentinel_integration1ignored
Modulesdifferential_module_replies1ignored

Server compatibility matrix

CadenceTargetImplementation source
Pull requestRedis 7.4.3source build
Pull requestRedis 8.0.6source build
Nightlyredis-7.2redis:7.2-alpine
Nightlyredis-7.4redis:7.4-alpine
Nightlyredis-8.0redis:8.0-alpine
Nightlyredis-8.2redis:8.2-alpine
Nightlyredis-8.4redis:8.4-alpine
Nightlyredis-8.6redis:8.6-alpine
Nightlyredis-8.8redis:8.8-alpine
Nightlyvalkey-8.1valkey/valkey:8.1-alpine

Fault and destructive-recovery inventory

SurfaceTest binaryCompiled testDefaultWorkflow cadence
Standaloneintegrationmultiplexed_client_reconnects_after_server_restartnormalpull request, nightly/scheduled
Standalonetest_resilience_integrationcircuit_breaker_ignores_redis_command_errorsnormalpull request
Standalonetest_resilience_integrationcircuit_breaker_opens_on_repeated_failuresnormalpull request
Standalonetest_resilience_integrationcommand_timeout_fires_on_slow_commandnormalpull request
Standalonetest_servercover_replicaof_and_failovernormalpull request
Clustercluster_integrationcached_cluster_fails_closed_and_recovers_after_master_connections_are_killedignoredpull request
Clustercluster_integrationcluster_pool_exhaustion_and_recoveryignoredpull request
Clustercluster_integrationmux_cluster_handles_ask_then_moved_during_live_reshardignoredpull request
Clustercluster_integrationmux_cluster_keyless_commands_survive_default_master_failoverignoredpull request
Clustercluster_integrationmux_cluster_replaces_killed_master_after_replica_promotionignoredpull request
Clustertopology_recoverykeyless_commands_follow_the_default_master_through_failovernormalpull request
Sentinelsentinel_failoversentinel_failover_sequenceignoredpull request
Sentinelsentinel_integrationsentinel_pool_exhaustion_and_recoveryignoredpull request

Reproducibility

The generator compiles only the client-facing integration-test packages with all features and --no-run, asks each emitted libtest binary for its full and ignored listings, and inspects checked-in workflow cargo test selectors. Regenerate or verify the marked section with:

python3 scripts/generate_test_conformance.py
python3 scripts/generate_test_conformance.py --check

The inventory describes what Cargo discovers under the generator’s documented feature set. It does not claim that every test runs in a normal cargo test, because infrastructure-backed suites are deliberately #[ignore]-gated and invoked by the workflows described below. It also cannot infer a runtime early return. The Redis 8.x command binary now discovers redis_version from INFO and fails when REDIS_8X_REQUIRED=1 cannot reach a suitable server; an optional REDIS_8X_VERSION is cross-checked against that discovery rather than treated as proof by itself.

Independent-client differential corpus

The compiled differential entry points compare redis-tower with a test-only, exactly pinned redis-rs 1.7.0 adapter. They are independent-client evidence, not additional expansions of the shared redis-tower command macro, and their generated count is therefore reported separately above.

The standalone corpus runs on every per-PR Redis 7.4.3 and 8.0.6 leg under both RESP2 and RESP3. Mutations use equivalent fresh namespaces, failures retain the case/protocol/live server/client version and deterministic seed, and normalization is limited to documented unordered or pair-shaped replies. The module comparison remains #[ignore]-gated. It runs in the path-filtered per-PR module gate and the broader module-enabled nightly matrix. See the reviewed case ledger and lifecycle contract for exact inputs, dispositions, negative controls, and reproduction commands.

What superseded the original fault comparison

Issue #416 proposed running the former custom CircuitBreakerLayer and tower-resilience-circuitbreaker side by side under identical faults. That experiment was useful input while the implementation choice was still open. PR #636 subsequently made the choice: it replaced the custom circuit-breaker internals with the Redis-aware adapter in circuit_breaker.rs. There is no longer a maintained custom breaker to use as the second side of the proposed A/B matrix, so presenting that matrix as current conformance coverage would be misleading.

The maintained contract is narrower and directly tested:

  • tower-resilience-circuitbreaker owns circuit state, half-open admission, metrics, and state transitions. The adapter preserves RedisError, maps an upstream open rejection to RedisError::CircuitOpen, and exposes an operational handle.
  • The Redis classifier counts connection failures, connect timeouts, command timeouts, and recursively classified reconnect exhaustion. Redis command replies and caller errors do not degrade circuit health.
  • Unit tests in circuit_breaker.rs cover classification, open rejection, successful half-open recovery, and the distinction between a deadline exhausted while waiting for readiness and a timeout after dispatch.
  • Live tests in test_resilience_integration.rs cover refused TCP connections through open and half-open recovery, a real slow Redis command through the timeout layer, and a WRONGTYPE response that leaves the circuit closed.

This does not claim fault parity for every crate in the tower-resilience family. Redis-aware circuit breaking is the upstream-backed adapter in this repository; command timeouts and idempotency-aware retries remain redis-tower-owned policies. Upstream rate limiting and bulkhead layers can be composed by applications, but they are not part of this conformance matrix.

Continuous test tiers

TierCoverageCadenceEvidence
Unit and rustdocAll-feature library tests on stable and beta RustEvery pull request and push to mainCI workflow
StandaloneEvery redis-tower integration binary and the standalone UniversalClient path against source-built Redis 7.4.3 and 8.0.6Every pull request and push to mainCI workflow, standalone suite, universal-client suite
Universal topology entry pointsStandalone URL auth, authenticated Cluster URL selection, Sentinel URL selection, command execution, and recovery after a real Sentinel failoverStandalone on both standard per-PR Redis legs; dedicated Cluster/Sentinel fixture in the transport gateCI workflow, transport gate, UniversalClient topology suite
Independent differentialMCP-derived semantic cases through redis-tower and pinned redis-rs adapters, including RESP2/RESP3, binary/null/error, pipeline, transaction, blocking-session, topology, cancellation, and lost-reply contractsStandalone, Cluster, and Sentinel per PR; module replies in the path-filtered module gate and nightlycase ledger, standalone corpus, module corpus
ClusterShared command corpus plus routing, redirect, caching, reshard, TLS, pool, and failover scenarios against Redis 7.4.3 and 8.0.6Every pull request and push to main; invoked explicitly with --ignoredCI workflow, cluster suite
SentinelShared command corpus, discovery, routing, pool, reconnect, and isolated destructive failover scenarios against Redis 7.4.3 and 8.0.6Every pull request and push to main; invoked explicitly with --ignoredCI workflow, healthy suite, failover suite
Cluster fixture lifecyclePartial-start cleanup, promotion/reshard/kill cleanup, and hard deadlines while nodes are frozenEvery pull request and push to main; invoked explicitly with --ignoredCI workflow, fixture tests
Server compatibilityFocused integration.rs suite across supported Redis minor lines and Valkey; Redis 8 command tests on each Redis 8 legNightly and manual dispatchnightly compatibility workflow
Verified TLS transportsPrivate-CA trust and hostname verification, percent-encoded password auth, RESP3/database setup, rejection of a wrong hostname and untrusted CA, Cluster roundtrips, and bracketed IPv6 URL transportIsolated rustls-only and native-TLS-only live legs per PR; bracketed IPv6 required in the same transport gatetransport gate, standalone TLS suite, Cluster TLS suite
Redis modulesA fail-closed actual-version/COMMAND INFO preflight followed by JSON, Search, TimeSeries, probabilistic, Vector Set, and redis-rs differential assertionsPath-filtered per PR on Redis 8; broader Redis 8 and Redis Stack 7.x matrix nightly/manualmodule gate, nightly modules workflow, capability preflight
Soak fault smokeShort standalone SIGKILL/restart and cluster master-kill/promotion runs, including JSONL contract checksPull requests touching the harness/client paths, and manual dispatchsoak workflow, soak harness
RESP property and fuzzGenerated mixed RESP2/RESP3 frames, independent wire fixtures, whole-versus-fragmented semantic comparison, and arbitrary malformed inputProperty and ten-second fuzz smoke per PR; bounded weekly/manual campaignsprotocol contract, fuzz targets, scheduled workflow
RESP decode copy/allocationExact test-only first-frame copied-byte counts; simple, mixed, retained-lifetime, and fragmented Criterion cases; instrumented comparison of historical, production, and split/shared ownershipCopied-byte unit contract and Criterion regression suite per PR; raw allocation artifact retained when ownership changesownership decision, raw evidence, benchmark sources

The per-PR cluster and Sentinel commands intentionally include --ignored. In this repository, that flag marks an infrastructure boundary, not an optional quality tier. In the remaining tables, “per PR” also includes the same CI tier on pushes to main.

Client and protocol parity

The reusable command_tests! corpus exercises strings, keys, hashes, lists, sets, sorted sets, bitmaps, geospatial commands, HyperLogLog, and streams through the same typed-command API. It is instantiated along these paths:

TopologyClient pathProtocol policy exercised by the shared corpusAutomated cadence
StandaloneRedisConnectionForced RESP2Per PR on Redis 7.4.3 and 8.0.6; nightly version matrix
StandaloneRedisConnectionForced RESP3Per PR on Redis 7.4.3 and 8.0.6; nightly version matrix
ClusterClusterConnectionDefault Auto policy across routed connectionsPer PR on Redis 7.4.3 and 8.0.6
ClusterMultiplexedClusterClientDefault Auto policy across pipelined node connectionsPer PR on Redis 7.4.3 and 8.0.6
SentinelSentinelConnectionDefault Sentinel connection path; no forced-protocol parity claimPer PR on Redis 7.4.3 and 8.0.6
SentinelMultiplexedSentinelClientDefault Sentinel connection path; no forced-protocol parity claimPer PR on Redis 7.4.3 and 8.0.6

The generated parity total counts compiled expansions, not assertions that run unchanged on every server version. Six Vector Set cases probe capability and return early when the server does not support those commands.

Standalone also has focused negotiation coverage for forced RESP2, forced RESP3, Auto, and the default connection path in integration.rs. Cluster connection builders retain an explicit protocol policy across discovery, redirects, topology refresh, and reconnect, with focused coverage in connection.rs and multiplexed.rs. Sentinel does not yet expose equivalent explicit protocol selection, so its rows above must not be read as RESP2/RESP3 parity.

UniversalClient has focused public-entry tests for all three variants. The Cluster case proves percent-decoded authentication; the Sentinel case kills the original master and requires the already-created client to resume commands after election. These are topology/lifecycle probes, not a replay of the full shared command corpus, so this report does not claim full universal-client command parity.

On Unix CI, the live test_unix_url suite opens a percent-encoded socket path through every supported Unix URL alias, applies percent-decoded ACL credentials, selects a non-default database, negotiates RESP3, and repeats setup through a fresh factory connection. Unix sockets are not available on Windows, and the matrix does not claim otherwise.

Exclusive-session behavior is exercised through the constructors users call: transactions and blocking commands own a RedisConnection, while Pub/Sub converts an owned connection with PubSubConnection::from_connection and explicitly replays subscriptions through reconnect_with. Those session types do not promise transparent reconnect on their own; factory-backed multiplexed, Cluster, and Sentinel handles provide that lifecycle. The corresponding live tests run in the standalone, Cluster, and Sentinel per-PR suites, including transaction abort/commit, bounded blocking, subscription replay, and topology rediscovery. See the standalone integration suite and Cluster integration suite.

Server version matrix

ServerStandaloneClusterSentinelVersion-gated Redis 8 commandsModules
Redis 7.4.3Per PRPer PRPer PRNot applicableNo
Redis 8.0.6 source buildPer PRPer PRPer PRNot activated on the module-free source buildNo
Redis 8 module imageFocused Redis 8 command binary in the path-filtered gateNot covered by this image legNot covered by this image legPer PR when relevant; actual version discovered from INFOPer-PR capability preflight plus all module suites
Redis 7.2 and 7.4 container tagsNightlyNot in the nightly matrixNot in the nightly matrixNot applicableNo
Redis 8.0, 8.2, 8.4, 8.6, and 8.8 container tagsNightlyNot in the nightly matrixNot in the nightly matrixNightly, with the corresponding version inputRedis 8 module image in the separate nightly module workflow
Valkey 8.1 container tagNightlyNot coveredNot coveredNot applicableNo
Redis Stack 7.4.0-v3No core-client matrix claimNot coveredNot coveredNot applicableNightly JSON, Search, TimeSeries, and probabilistic suites

The exact per-PR patch versions and nightly image tags are defined in ci.yml, nightly.yml, and module-gate.yml, and nightly-modules.yml. Moving container tags intentionally test the current patch in their minor line; the capability log records the actual INFO redis_version and required command set, while the workflow log and image digest identify the image for a particular run. The nightly compatibility legs run the focused integration.rs suite, not every standalone test binary in the generated inventory.

Destructive and fault-injection inventory

Injected conditionRequired observationCadenceEvidence
Redis command exceeds its deadline (DEBUG SLEEP or a blocking command)A bounded CommandTimeout is surfaced before the slow command completesPer PRresilience tests, standalone integration
Connected TCP peer applies a write but withholds all or some pipeline repliesExplicit response timeout reports unknown execution, closes the old socket, and recovers on a replacement without replay or reply misalignmentPer PR, no Redis processauto-pipeline tests
Refused TCP connections, then a healthy endpointCircuit opens at the configured threshold, rejects immediately, admits a half-open probe, and closes after recoveryPer PRresilience tests
Managed standalone process stopped and restarted on the same portThe existing multiplexed client eventually resumes validated reads and writes against a fresh databasePer PR; deliberately bypassed in externally managed nightly service containersstandalone integration
Standalone REPLICAOF role changes and coordinated FAILOVERReplication completes, roles change, and acknowledged data survives promotionPer PRserver integration
Cluster slot migration held between importing/migrating and final ownershipClient follows ASK and then MOVED, while its redirect metrics record the expected orderPer PRcluster integration
Cached cluster data and invalidation connections killed on one masterCache fails closed, clears unsafe entries, returns current data, and restores complete tracking coveragePer PRcluster integration
Master killed in a dedicated cluster and its acknowledged replica promotedClient reaches a bounded first successful read, preserves data, and replaces the dead slot ownerPer PRcluster integration
One Sentinel killed, then the monitored master killedDiscovery survives one Sentinel loss while quorum remains, Sentinel elects a different master, the existing client rediscovers it, and a fresh connection finds itPer PR, in an isolated ordered test binarySentinel failover sequence
Pool contention with one available connectionConcurrent cluster and Sentinel callers serialize and recover rather than failing from exhaustionPer PRcluster suite, Sentinel suite
Cluster fixture startup expires, or all fixture nodes are frozenPartial processes and artifacts are reaped; the caller’s deadline stays hard; teardown resumes stopped nodes and removes only owned statePer PRfixture lifecycle tests
Real standalone SIGKILL/restart under continuous validated loadExact owned process dies, replacement starts on the same port, lifecycle event is observed, and useful work recovers within the boundPath-filtered PR smoke; long run by explicit publication protocolsoak workflow, soak harness
Real cluster master SIGKILL under continuous validated loadRedis promotes the replicated slot owner and the client completes a validated probe within the boundPath-filtered PR smoke; long run by explicit publication protocolsoak workflow, soak harness

Destructive Sentinel phases use a separate port range and a single orchestrating test because killing shared processes would make independent tests order-dependent. Dedicated cluster failover and reshard fixtures use their own topologies for the same reason.

Ignored-test policy

#[ignore] is reserved for tests that need infrastructure or intentionally mutate a shared topology: live cluster and Sentinel processes, module-enabled servers, Docker, destructive fault injection, or signal-cleanup checks. Each ignored test must state or inherit its prerequisite and must have one of these execution paths:

  • an explicit per-PR cargo test ... -- --ignored step;
  • a scheduled workflow matched to the required server image; or
  • a path-filtered/manual fault-harness workflow.

The mapping is visible in the CI, compatibility, per-PR module, nightly module, and soak workflows. A normal workspace test will therefore report ignored tests; that is expected. The conformance goal is no silently abandoned ignored tests, not a false claim that every environment can execute every test.

Reproduce the tiers

The process-backed commands require redis-server and redis-cli on PATH. Cluster, Sentinel, and destructive suites should be run single-threaded to match CI and avoid port/topology interference.

# Generated inventory and drift check
python3 scripts/generate_test_conformance.py
python3 scripts/generate_test_conformance.py --check

# Unit, rustdoc, standalone, and universal-client coverage
cargo test --lib --all-features
cargo test --doc --all-features
cargo test -p redis-tower --test '*' --all-features -- --test-threads=1
cargo test -p redis-tower-client --test '*' --all-features -- --test-threads=1

# Infrastructure-gated per-PR suites
cargo test -p redis-tower-client --test topologies --all-features -- \
  --ignored --test-threads=1
cargo test -p redis-tower-cluster --test cluster_integration \
  --all-features -- --ignored --test-threads=1
cargo test -p redis-tower-sentinel --test 'sentinel_*' \
  --all-features -- --ignored --test-threads=1
cargo test -p redis-tower-test --lib live_ -- --ignored --test-threads=1

# The upstream-backed circuit and timeout contract only
cargo test -p redis-tower --test test_resilience_integration \
  --all-features -- --test-threads=1

# Docker fixture used by nightly compatibility
REDIS_TEST_IMAGE=redis:8.8-alpine \
  cargo test -p redis-chaos-tests --test docker_smoke -- --ignored

# Module suites; preflight the actual server before behavior
REDIS_URL=redis://127.0.0.1:6399 REDIS_8X_REQUIRED=1 \
  cargo test -p redis-tower-modules --all-features --test capabilities \
  required_server_version_and_commands_are_present -- --ignored --exact --nocapture
REDIS_URL=redis://127.0.0.1:6399 REDIS_8X_REQUIRED=1 \
  cargo test -p redis-tower-modules --all-features -- --ignored

The exact short soak commands and the publication-grade long-run requirements live in the soak harness guide. Reproduce a nightly version leg by using the image, REDIS_URL, REDIS_8X_REQUIRED=1, and the expected REDIS_8X_VERSION shown in nightly.yml; that workflow is the canonical matrix definition.

Known limits

  • The old custom-vs-upstream circuit-breaker A/B from issue #416 is not runnable because the custom implementation was retired. The maintained adapter contract above is the replacement evidence.
  • There is no automated true network partition, Docker pause, or Toxiproxy latency/jitter leg today. Current fault tests cover refused connects, connection kills, process SIGKILL/restart, role changes, resharding, and command stalls. Deterministic connected-peer response-stall tests verify explicit timeout, quarantine and no replay, but do not simulate packet-level partitions or node-to-node quorum loss.
  • Valkey is exercised only through the standalone nightly suite. Cluster, Sentinel, module, and fault-injection parity are not claimed for Valkey.
  • The nightly version matrix is standalone-only; per-PR cluster and Sentinel coverage is limited to the two pinned Redis patch versions in CI.
  • Sentinel uses its default connection protocol path but does not expose the explicit protocol policy available to standalone and cluster builders.
  • The per-PR module gate is path-filtered and uses one Redis 8 image. The Redis Stack 7.x feature subset and broader image cadence remain nightly.
  • The short soak workflow proves harness and recovery behavior, not long-run performance, memory stability, or a publication result. Those claims require the sealed publication protocol and its retained artifacts.
  • Live Redis integration, module, and transport handshakes currently run on Linux. Publishable all-feature library tests also run on macOS arm64, Windows x64, and Linux arm64. Managed services still need separate validation; protocol compatibility alone is not provider certification.

Differential testing against redis-rs

The differential corpus uses redis-rs as an independent implementation oracle, not as the Redis specification. Both clients run commands against the same server version but use separate key namespaces for every mutation. When the clients disagree, Redis documentation and server behavior decide the expected contract.

The test-only dependency is pinned to redis = 1.7.0; the per-PR job records the resolved oracle package with cargo tree alongside the Redis server version. It is not a production dependency. Standalone command and conversion failures report the case, command or step, adapter side, selected RESP protocol, live Redis version, both client versions, and deterministic seed (0x52454449535f4d43). Connection failures use server=unavailable because no server version can be queried safely. A synthetic user/password regression drives both adapters through connection failure and verifies that neither the target nor raw error text reaches the captured panic message.

Initial MCP-derived case ledger

BehaviorInput and isolated setupExpected semanticsProtocol/server requirementredis-tower adapterredis-rs adapter
Scalar, null, empty, and binaryGET missing; binary-safe SET/GET/ECHO; MGET over present, missing, and empty values in per-client namespacesNull stays distinct from empty bytes; arbitrary bytes round-trip without UTF-8 lossRESP2 and RESP3; Redis 7.4.3 and 8.0.6 per PRRedisConnection + RawCommandasync MultiplexedConnection + Cmd
Remaining typed binary familiestyped RENAME, EVAL, ECHO, PUBLISH, GEOADD, PFADD, and BITOP with invalid UTF-8, NUL, CR/LF, and protocol-looking bytesTyped serialization and server-visible results agree without decoding opaque argumentsRESP2 and RESP3typed command buildersCmd with byte-slice arguments
Numeric boundaries and decodingStore u64::MAX, -1, and invalid UTF-8 as bulk valuesBoth decode the maximum as u64; both reject negative-to-u64 and invalid UTF-8-to-String conversionsRESP2 and RESP3TypedRawCommand<u64/String>FromRedisValue<u64/String>
Typed builder optionsExercise Set::nx().get() against an existing key and Set::xx().get() against a missing key; compare the returned old value and a follow-up GETBuilder options reach the wire: NX preserves the existing value and XX does not create a missing keyRESP2 and RESP3typed Set builderCmd with explicit NX/XX/GET arguments
Hashes and unordered setsInvalid-UTF-8 keys, binary hash value plus multiple fields; set members inserted in a different lexical orderHGETALL compares as sorted pairs; SMEMBERS compares as a sorted collectionRESP2 flat pairs / arrays and RESP3 maps / setstyped HSet / SAdd, raw response normalized after decodeCmd request and raw response
Lists and sorted setsInvalid-UTF-8 keys, ordered list including binary bytes; ZRANGE WITHSCORESList order is preserved; zset member/score pairs survive RESP2/RESP3 shape differencesRESP2 and RESP3typed RPush / ZAdd, raw responseCmd request and raw response
Streams and nested repliesInvalid-UTF-8 stream key, deterministic typed XADD 1-0, then XRANGEEntry IDs, field order, binary values, and nested structure agreeRedis 5+; RESP2 and RESP3typed XAdd, raw responseCmd request and raw response
Server and conversion errorsGET against a list; INCR at i64::MAX; typed range/UTF-8 failuresServer error codes agree (WRONGTYPE, ERR); client-side conversion failures remain failures without pretending their library-specific categories are identicalRESP2 and RESP3RedisError classificationredis::RedisError classification
Raw and administrative shapesLua returns null, empty aggregate, status, integer, and binary values; COMMAND INFO GET returns administrative nestingOrdered nested values agree without blanket flattening or lossy conversionRESP2 and RESP3RawCommandCmd
Pipeline outcome and alignmentSET, failing HSET, then GET, followed by PINGPer-command failure is retained, later reply stays paired with GET, and the next command remains alignedRESP2 and RESP3PipelinePipeline::ignore_errors
Transaction commit and abortAtomic SET/INCR/GET; separate clients mutate a watched key before EXECCommit results agree; WATCH conflict returns an aborted transaction and does not apply its bodyRESP2 and RESP3Transaction / exclusive connectionatomic Pipeline / exclusive connection
Blocking command ownershipSeed independent lists, then BLMOVE binary data with a finite timeoutResults agree while each blocking operation owns a dedicated, unshared sessionRedis 6.2+; RESP2 and RESP3dedicated RedisConnectionunshared async connection
Public topology entry pointsBinary SET/GET through live Cluster and Sentinel fixtures, with an independent redis-rs Cluster connection or direct connection to the Sentinel-discovered masterRouting/discovery does not change the binary result; mutations remain in independent namespacesPer-PR Redis 7.4.3 and 8.0.6 Cluster/Sentinel fixture legsClusterConnection / SentinelConnectionasync Cluster connection / direct async master connection
RedisJSON module repliesIndependent JSON documents; compare JSON.GET and unordered JSON.OBJKEYSJSON bytes agree and object-key order is the only normalizationPath-filtered per-PR and nightly module-enabled Redis 8; RESP2 and RESP3RawCommandCmd
Queued cancellationCancel single and multi requests before the configured batch window closesCancelled queue entries never reach the socketDeterministic in-memory/TCP fixture; no Redis processAutoPipelineServiceNot applicable: redis-tower lifecycle contract
Lost non-idempotent replyFake server records an INCR-like request, applies it once, then closes before replying; replacement socket receives a probeCaller gets an error, execution remains unknown, and reconnect never silently replays the writeDeterministic TCP fixture; no Redis processfactory-backed AutoPipelineServiceNot applicable: redis-tower lifecycle contract
Connected peer withholds repliesFake peer applies one INCR-like write while keeping TCP open; single request gets no reply, multi request gets only its first replyExplicit response timeout surfaces CommandTimeout; old socket closes, replacement probe stays aligned, and the write is not replayedBounded deterministic TCP fixture; no Redis process or packet-level partitionfactory-backed AutoPipelineServiceNot applicable: redis-tower lifecycle contract

The executable standalone cases live in differential_redis_rs.rs. The RedisJSON case lives in differential_module_replies.rs and is selected by the per-PR module gate and module-enabled nightly job.

Normalization policy

Normalization is deliberately narrow:

  • Sets are sorted because Redis does not promise member order.
  • RESP2 flat pair arrays, RESP3 pair arrays, and RESP3 maps become sorted pairs only for commands whose documented result is a map or pair collection.
  • Status strings and bulk strings preserve their bytes; OK is not converted through UTF-8 to compare successfully.
  • Null and empty values, ordered arrays, integer widths, binary bytes, and server error codes are never collapsed together.
  • Client-side conversion error taxonomies may differ; the shared contract is rejection, while server-originated error codes must agree.

The negative controls prove that the comparator rejects lossy binary conversion, reversed sorted-set rank order, and a removed NX option from the typed Set path. A replay-count control and the live lost-reply fixture fail if a non-idempotent request executes twice.

Cancellation and unknown execution

There are three materially different states:

  1. A request cancelled while still queued is unsent and has no server effect.
  2. A request whose complete reply was received has a known result.
  3. A request written before its connection loses the reply has unknown execution. Reconnection cannot determine whether Redis applied it.

redis-tower removes cancelled queue entries before flushing and quarantines a connection whose reply alignment is no longer knowable. It does not replay the lost non-idempotent request on the replacement socket. Applications that retry an unknown-execution write must supply their own idempotency mechanism.

An explicit AutoPipelineConfig::response_timeout also bounds a connected peer that stops replying, including after only part of a pipeline’s responses arrive. The deterministic TCP regressions keep that socket open until the timeout is observed, then verify quarantine and useful replacement work without replay. This is a client response-stall fixture, not a real network-partition test; the default has no response deadline, so applications must configure one.

Reproduction

cargo test -p redis-tower --test differential_redis_rs \
  --all-features -- --test-threads=1

# Requires REDIS_URL pointing at a RedisJSON-capable server.
cargo test -p redis-tower-modules --test differential_module_replies \
  --all-features -- --ignored --test-threads=1

Engineering hygiene

The Engineering Hygiene workflow keeps expensive diligence visible without putting every slow measurement on the ordinary Linux CI critical path.

Pull-request gates

Every pull request runs three classes of checks:

  • cargo-semver-checks compares all publishable crate APIs with the pull request target revision under a patch-compatible policy. Intentional API breaks must therefore be explicit rather than arriving as collateral edits.
  • Publishable library tests run with every feature on macOS arm64, Windows x64, and Linux arm64. The matrix excludes Unix-only benchmark packages, and the reusable cluster fixture is compiled only on Unix because its process lifecycle depends on POSIX signals. The all-feature builds exercise native TLS on each host; Unix-socket tests run on the Unix hosts.
  • check_release_hygiene.py discovers every publishable redis-tower* crate from the workspace manifest. It requires #![deny(missing_docs)], complete docs.rs metadata, standard package metadata, and expected package contents.

Run the release audit locally with:

python3 scripts/check_release_hygiene.py --check-package-contents

Mutation score

The scheduled and manual workflow runs cargo-mutants against the library tests of every publishable crate. Ordinary pull requests do not wait for this suite. Each package retains its complete mutants.out directory for 90 days, and the summary job emits per-package and combined JSON/Markdown reports.

The conservative headline score is:

caught / (caught + missed + timeout)

Unviable mutants are reported but excluded from the denominator. Timeouts are included because they need investigation rather than being silently treated as caught. The initial 70% floor is a tracked target, not a merge gate; establish a stable baseline before deciding whether to enforce it.

For a focused local run:

cargo mutants --package redis-tower-core --jobs 2
python3 scripts/mutation_score.py mutants.out --minimum-score 0.70

The scheduled workflow derives its package list from cargo metadata. It uses deterministic round-robin shards for suites that exceed one runner’s budget or need smaller retry units, then requires every shard before producing each package artifact and all 13 package reports before the workspace headline score.

Mutation failure and recovery

Treat three outcomes differently:

OutcomeMeaningRecovery
A mutant is missed or times outValid test-quality evidence for that packageInspect the retained shard outcome and add a focused assertion when the mutation is behaviorally meaningful. Do not relabel it as infrastructure failure.
A shard process or runner failsThe package result is incompleteKeep the uploaded successful shard artifacts for diagnosis, then rerun the workflow on the same source SHA. Do not combine shards from different workflow runs.
A package or headline aggregation rejects missing reportsThe fail-closed completeness guard workedFind the absent shard/package in the job graph, inspect its logs, and rerun the workflow. Never lower --expected-reports to manufacture a score.

Use the workflow’s Run workflow action for a clean recovery run. Record the source SHA and the replacement run URL in any issue or release evidence. The package and headline jobs intentionally aggregate only artifacts from their own run; partial results remain downloadable for 90 days but cannot silently enter a later score. The 70% target remains informational: a complete low score, an incomplete campaign, and a failing product test are three distinct states.

Protocol fuzzing

The RESP decoder has a separate weekly and manually dispatchable fuzz workflow. It runs both the arbitrary-input decoder and the whole-versus-fragmented semantic oracle for a bounded, configurable duration. Every campaign starts from the reviewed named seeds and retains the final corpus, crashes, source and dependency provenance, log, duration, and outcome for 90 days. See the protocol validation contract for the disposition table, input envelope, local commands, and historical mutation classification.

CI wall clock and flake signal

The scheduled report reads the latest 50 completed, non-cancelled runs of the main CI workflow. It records p50, p95, and maximum wall-clock duration plus:

  • a rerun/flake signal: runs whose GitHub run_attempt is greater than one;
  • a failure signal: completed runs whose conclusion is not success.

The enforced operating budgets are a 15-minute p95 and a 10% rerun rate. Failure rate is tracked but not budgeted because a failing pull-request commit usually represents a real code/configuration error, not a flaky test. Cancelled runs are excluded because superseded jobs and GitHub queue incidents otherwise distort elapsed time.

The reporter accepts captured GitHub API JSON for local reproduction:

python3 scripts/ci_health.py --input workflow-runs.json \
  --max-p95-minutes 15 --max-rerun-rate 0.10

Resource and build footprint

The existing Weekly Benchmarks workflow remains the source of comparative resource evidence. Its isolated probes compare redis-tower, redis-rs, and Fred for RSS per connection and CPU at a fixed offered rate, then compare cold-build time and stripped binary size. JSON results and the exact Cargo.lock are retained for 90 days. Hosted-runner output is trend/smoke evidence; release claims still require an otherwise-idle dedicated host as described in the benchmark publication protocol.

Release process

redis-tower uses the manual Release GitHub Actions workflow. Preparing a release pull request and publishing immutable crates are separate maintainer actions. Merging ordinary code never publishes a package.

Current published state

The complete 13-crate public workspace is available on crates.io. These versions were verified against the workspace manifests, crates.io search, and GitHub releases on 2026-10-22:

Package groupPublished version
redis-tower-protocol, redis-tower, redis-tower-cluster, redis-tower-sentinel0.1.4
redis-tower-core, redis-tower-commands, redis-tower-sync, redis-tower-modules, redis-tower-client, redis-tower-primitives0.1.3
redis-tower-auth-aws, redis-tower-auth-azure, redis-tower-test0.1.2

This is a dated status snapshot, not a version source of truth. Before a release, compare the manifests, crates.io, tags, GitHub releases, and each crate’s changelog.

Prepare

  1. Confirm that no open pull request already covers the release and that main is green.

  2. Run the local gates:

    cargo fmt --all -- --check
    cargo clippy --workspace --all-targets --all-features -- -D warnings
    cargo test --lib --all-features
    cargo test --workspace --doc --all-features
    RUSTDOCFLAGS="-D warnings" cargo doc --workspace --all-features --no-deps
    mdbook build
    mdbook test
    python3 scripts/test_check_docs_links.py -v
    python3 scripts/check_docs_links.py
    python3 scripts/check_release_hygiene.py --check-package-contents
    
  3. Ask release-plz to prepare version and changelog changes:

    gh workflow run Release --ref main -f command=release-pr
    
  4. Review the generated pull request. Check that every changed public package has the intended semver bump and changelog entry, workspace dependency requirements remain resolvable, and no unrelated package was bumped.

  5. Merge only after the release pull request and required checks are complete.

A downstream cargo publish --dry-run cannot resolve a new workspace dependency until that exact dependency version exists on crates.io. Dry-run the first dependency tier (redis-tower-protocol) and rely on the full workspace gates for downstream code before publication. The real workflow waits for each published dependency to become visible before continuing.

Publish

Publish only from a clean, fully validated main:

gh workflow run Release --ref main -f command=release

The workspace dependency graph requires this order:

  1. redis-tower-protocol
  2. redis-tower-core
  3. redis-tower-commands and redis-tower-test
  4. redis-tower
  5. Cluster, Sentinel, modules, primitives, sync, and cloud-auth crates
  6. redis-tower-client

GitHub serializes release workflow runs and skips publication in forks. The repository must retain a CARGO_REGISTRY_TOKEN Actions secret; the first publication of a crate cannot be bootstrapped with crates.io trusted publishing.

Do not retry a partially successful workflow blindly. Inspect the workflow, crates.io, tags, and releases first. Published versions are immutable: continue from the first missing tier, and bump only packages whose attempted version can no longer be published.

Verify

For every package reported as published:

  • confirm the expected version exists and is not yanked on crates.io;
  • confirm the docs.rs build completed with the expected features;
  • confirm the GitHub tag and release exist;
  • install the facade from crates.io in a fresh temporary project and run a basic PING against a supported Redis version;
  • verify README badges and public documentation links resolve to the release.

Keep publication manual. It changes external immutable state and remains a deliberate action after review and validation.

Recovery history

The initial June 2026 publication attempt left six versions yanked and seven packages unpublished. Yanked versions cannot be reused, and release-plz could not infer a next version when every registry version was yanked.

PR #690 merged on 2026-08-28 with explicit patch bumps, changelogs for all public crates, staged publication order, and the manual operation-specific workflow. The first complete staged relaunch was published on 2026-09-18; follow-up releases on 2026-09-24 established the current version groups above. That incident explains the recovery safeguards but is no longer the active release plan.

See release-plz’s yanked-package guidance if a future release encounters the same registry state.