Test conformance
This report maps redis-tower’s test surface to the claims it supports. It is intended to be citable at a particular commit: the compiled inventory is generated from the checkout, while the matrices below link each broader claim to its test code and workflow. A compiled test is not automatically a passing test, and a scheduled job is not evidence that a particular run was green; use the checks attached to the commit for that conclusion.
This page inventories compiled tests, configured workflow expansion, and observed execution evidence. Those are distinct from the logical client behaviors they exercise. The generated command capability ledger records those behaviors and separately tracks command-name availability, typed semantics, RESP wire support, and session/topology claims; neither page treats one dimension as proof of another.
Compiled inventory
The generated block is the only place this page records test counts. Keeping
the numbers mechanical prevents a prose headline from drifting as tests are
added, renamed, or split into new binaries. Refresh it with
python3 scripts/generate_test_conformance.py; CI checks that the committed block
matches the current source tree.
Compiled integration inventory
The scoreboard-scoped packages compile 756 integration tests across 45 test binaries. 212 tests are marked #[ignore] because they need explicit infrastructure; workflow selectors, rather than the annotation alone, determine whether they run.
| Surface | Package | Binaries | Compiled | #[ignore] | Pull request | Scheduled | No workflow selector |
|---|---|---|---|---|---|---|---|
| Standalone | redis-tower | 32 | 520 | 0 | 520 | 229 | 0 |
| Cluster | redis-tower-cluster | 3 | 112 | 106 | 112 | 0 | 0 |
| Sentinel | redis-tower-sentinel | 2 | 88 | 88 | 88 | 0 | 0 |
| Modules | redis-tower-modules | 7 | 20 | 18 | 20 | 18 | 0 |
| Distributed primitives | redis-tower-primitives | 1 | 16 | 0 | 16 | 0 | 0 |
| Total | 45 | 756 | 212 | 756 | 247 | 0 |
Counts are unique compiled tests. A test selected by both pull-request and scheduled workflows appears in both cadence columns, but only once in the compiled total.
Cross-backend command parity
The shared command corpus produces 210 compiled parity tests: 35 test cases × 6 client/topology expansions.
| Surface | Test binary / variant | Compiled cases |
|---|---|---|
| Standalone | integration / resp3 | 35 |
| Standalone | integration / root expansion | 35 |
| Cluster | cluster_integration / multiplexed | 35 |
| Cluster | cluster_integration / root expansion | 35 |
| Sentinel | sentinel_integration / multiplexed | 35 |
| Sentinel | sentinel_integration / root expansion | 35 |
Independent-client differential coverage
The independent redis-rs corpus contains 12 compiled test entry points. These are counted separately from shared-client topology expansions.
| Surface | Test binary | Compiled tests | Default |
|---|---|---|---|
| Standalone | differential_redis_rs | 9 | normal |
| Cluster | cluster_integration | 1 | ignored |
| Sentinel | sentinel_integration | 1 | ignored |
| Modules | differential_module_replies | 1 | ignored |
Server compatibility matrix
| Cadence | Target | Implementation source |
|---|---|---|
| Pull request | Redis 7.4.3 | source build |
| Pull request | Redis 8.0.6 | source build |
| Nightly | redis-7.2 | redis:7.2-alpine |
| Nightly | redis-7.4 | redis:7.4-alpine |
| Nightly | redis-8.0 | redis:8.0-alpine |
| Nightly | redis-8.2 | redis:8.2-alpine |
| Nightly | redis-8.4 | redis:8.4-alpine |
| Nightly | redis-8.6 | redis:8.6-alpine |
| Nightly | redis-8.8 | redis:8.8-alpine |
| Nightly | valkey-8.1 | valkey/valkey:8.1-alpine |
Fault and destructive-recovery inventory
| Surface | Test binary | Compiled test | Default | Workflow cadence |
|---|---|---|---|---|
| Standalone | integration | multiplexed_client_reconnects_after_server_restart | normal | pull request, nightly/scheduled |
| Standalone | test_resilience_integration | circuit_breaker_ignores_redis_command_errors | normal | pull request |
| Standalone | test_resilience_integration | circuit_breaker_opens_on_repeated_failures | normal | pull request |
| Standalone | test_resilience_integration | command_timeout_fires_on_slow_command | normal | pull request |
| Standalone | test_server | cover_replicaof_and_failover | normal | pull request |
| Cluster | cluster_integration | cached_cluster_fails_closed_and_recovers_after_master_connections_are_killed | ignored | pull request |
| Cluster | cluster_integration | cluster_pool_exhaustion_and_recovery | ignored | pull request |
| Cluster | cluster_integration | mux_cluster_handles_ask_then_moved_during_live_reshard | ignored | pull request |
| Cluster | cluster_integration | mux_cluster_keyless_commands_survive_default_master_failover | ignored | pull request |
| Cluster | cluster_integration | mux_cluster_replaces_killed_master_after_replica_promotion | ignored | pull request |
| Cluster | topology_recovery | keyless_commands_follow_the_default_master_through_failover | normal | pull request |
| Sentinel | sentinel_failover | sentinel_failover_sequence | ignored | pull request |
| Sentinel | sentinel_integration | sentinel_pool_exhaustion_and_recovery | ignored | pull request |
Reproducibility
The generator compiles only the client-facing integration-test packages with all features and --no-run, asks each emitted libtest binary for its full and ignored listings, and inspects checked-in workflow cargo test selectors. Regenerate or verify the marked section with:
python3 scripts/generate_test_conformance.py
python3 scripts/generate_test_conformance.py --check
The inventory describes what Cargo discovers under the generator’s documented
feature set. It does not claim that every test runs in a normal cargo test,
because infrastructure-backed suites are deliberately #[ignore]-gated and
invoked by the workflows described below. It also cannot infer a runtime early
return. The Redis 8.x command binary now discovers redis_version from INFO
and fails when REDIS_8X_REQUIRED=1 cannot reach a suitable server; an optional
REDIS_8X_VERSION is cross-checked against that discovery rather than treated
as proof by itself.
Independent-client differential corpus
The compiled differential entry points compare redis-tower with a test-only, exactly pinned redis-rs 1.7.0 adapter. They are independent-client evidence, not additional expansions of the shared redis-tower command macro, and their generated count is therefore reported separately above.
The standalone corpus runs on every per-PR Redis 7.4.3 and 8.0.6 leg under
both RESP2 and RESP3. Mutations use equivalent fresh namespaces, failures
retain the case/protocol/live server/client version and deterministic seed,
and normalization is limited to documented unordered or pair-shaped replies.
The module comparison remains #[ignore]-gated. It runs in the path-filtered
per-PR module gate and the broader module-enabled nightly matrix. See the reviewed
case ledger and lifecycle contract for exact inputs,
dispositions, negative controls, and reproduction commands.
What superseded the original fault comparison
Issue #416 proposed
running the former custom CircuitBreakerLayer and
tower-resilience-circuitbreaker side by side under identical faults. That
experiment was useful input while the implementation choice was still open.
PR #636 subsequently
made the choice: it replaced the custom circuit-breaker internals with the
Redis-aware adapter in
circuit_breaker.rs. There is
no longer a maintained custom breaker to use as the second side of the proposed
A/B matrix, so presenting that matrix as current conformance coverage would be
misleading.
The maintained contract is narrower and directly tested:
tower-resilience-circuitbreakerowns circuit state, half-open admission, metrics, and state transitions. The adapter preservesRedisError, maps an upstream open rejection toRedisError::CircuitOpen, and exposes an operational handle.- The Redis classifier counts connection failures, connect timeouts, command timeouts, and recursively classified reconnect exhaustion. Redis command replies and caller errors do not degrade circuit health.
- Unit tests in
circuit_breaker.rscover classification, open rejection, successful half-open recovery, and the distinction between a deadline exhausted while waiting for readiness and a timeout after dispatch. - Live tests in
test_resilience_integration.rscover refused TCP connections through open and half-open recovery, a real slow Redis command through the timeout layer, and aWRONGTYPEresponse that leaves the circuit closed.
This does not claim fault parity for every crate in the tower-resilience family. Redis-aware circuit breaking is the upstream-backed adapter in this repository; command timeouts and idempotency-aware retries remain redis-tower-owned policies. Upstream rate limiting and bulkhead layers can be composed by applications, but they are not part of this conformance matrix.
Continuous test tiers
| Tier | Coverage | Cadence | Evidence |
|---|---|---|---|
| Unit and rustdoc | All-feature library tests on stable and beta Rust | Every pull request and push to main | CI workflow |
| Standalone | Every redis-tower integration binary and the standalone UniversalClient path against source-built Redis 7.4.3 and 8.0.6 | Every pull request and push to main | CI workflow, standalone suite, universal-client suite |
| Universal topology entry points | Standalone URL auth, authenticated Cluster URL selection, Sentinel URL selection, command execution, and recovery after a real Sentinel failover | Standalone on both standard per-PR Redis legs; dedicated Cluster/Sentinel fixture in the transport gate | CI workflow, transport gate, UniversalClient topology suite |
| Independent differential | MCP-derived semantic cases through redis-tower and pinned redis-rs adapters, including RESP2/RESP3, binary/null/error, pipeline, transaction, blocking-session, topology, cancellation, and lost-reply contracts | Standalone, Cluster, and Sentinel per PR; module replies in the path-filtered module gate and nightly | case ledger, standalone corpus, module corpus |
| Cluster | Shared command corpus plus routing, redirect, caching, reshard, TLS, pool, and failover scenarios against Redis 7.4.3 and 8.0.6 | Every pull request and push to main; invoked explicitly with --ignored | CI workflow, cluster suite |
| Sentinel | Shared command corpus, discovery, routing, pool, reconnect, and isolated destructive failover scenarios against Redis 7.4.3 and 8.0.6 | Every pull request and push to main; invoked explicitly with --ignored | CI workflow, healthy suite, failover suite |
| Cluster fixture lifecycle | Partial-start cleanup, promotion/reshard/kill cleanup, and hard deadlines while nodes are frozen | Every pull request and push to main; invoked explicitly with --ignored | CI workflow, fixture tests |
| Server compatibility | Focused integration.rs suite across supported Redis minor lines and Valkey; Redis 8 command tests on each Redis 8 leg | Nightly and manual dispatch | nightly compatibility workflow |
| Verified TLS transports | Private-CA trust and hostname verification, percent-encoded password auth, RESP3/database setup, rejection of a wrong hostname and untrusted CA, Cluster roundtrips, and bracketed IPv6 URL transport | Isolated rustls-only and native-TLS-only live legs per PR; bracketed IPv6 required in the same transport gate | transport gate, standalone TLS suite, Cluster TLS suite |
| Redis modules | A fail-closed actual-version/COMMAND INFO preflight followed by JSON, Search, TimeSeries, probabilistic, Vector Set, and redis-rs differential assertions | Path-filtered per PR on Redis 8; broader Redis 8 and Redis Stack 7.x matrix nightly/manual | module gate, nightly modules workflow, capability preflight |
| Soak fault smoke | Short standalone SIGKILL/restart and cluster master-kill/promotion runs, including JSONL contract checks | Pull requests touching the harness/client paths, and manual dispatch | soak workflow, soak harness |
| RESP property and fuzz | Generated mixed RESP2/RESP3 frames, independent wire fixtures, whole-versus-fragmented semantic comparison, and arbitrary malformed input | Property and ten-second fuzz smoke per PR; bounded weekly/manual campaigns | protocol contract, fuzz targets, scheduled workflow |
| RESP decode copy/allocation | Exact test-only first-frame copied-byte counts; simple, mixed, retained-lifetime, and fragmented Criterion cases; instrumented comparison of historical, production, and split/shared ownership | Copied-byte unit contract and Criterion regression suite per PR; raw allocation artifact retained when ownership changes | ownership decision, raw evidence, benchmark sources |
The per-PR cluster and Sentinel commands intentionally include --ignored.
In this repository, that flag marks an infrastructure boundary, not an
optional quality tier. In the remaining tables, “per PR” also includes the
same CI tier on pushes to main.
Client and protocol parity
The reusable
command_tests! corpus
exercises strings, keys, hashes, lists, sets, sorted sets, bitmaps, geospatial
commands, HyperLogLog, and streams through the same typed-command API. It is
instantiated along these paths:
| Topology | Client path | Protocol policy exercised by the shared corpus | Automated cadence |
|---|---|---|---|
| Standalone | RedisConnection | Forced RESP2 | Per PR on Redis 7.4.3 and 8.0.6; nightly version matrix |
| Standalone | RedisConnection | Forced RESP3 | Per PR on Redis 7.4.3 and 8.0.6; nightly version matrix |
| Cluster | ClusterConnection | Default Auto policy across routed connections | Per PR on Redis 7.4.3 and 8.0.6 |
| Cluster | MultiplexedClusterClient | Default Auto policy across pipelined node connections | Per PR on Redis 7.4.3 and 8.0.6 |
| Sentinel | SentinelConnection | Default Sentinel connection path; no forced-protocol parity claim | Per PR on Redis 7.4.3 and 8.0.6 |
| Sentinel | MultiplexedSentinelClient | Default Sentinel connection path; no forced-protocol parity claim | Per PR on Redis 7.4.3 and 8.0.6 |
The generated parity total counts compiled expansions, not assertions that run unchanged on every server version. Six Vector Set cases probe capability and return early when the server does not support those commands.
Standalone also has focused negotiation coverage for forced RESP2, forced
RESP3, Auto, and the default connection path in
integration.rs. Cluster
connection builders retain an explicit protocol policy across discovery,
redirects, topology refresh, and reconnect, with focused coverage in
connection.rs and
multiplexed.rs. Sentinel
does not yet expose equivalent explicit protocol selection, so its rows above
must not be read as RESP2/RESP3 parity.
UniversalClient has focused public-entry tests for all three variants. The
Cluster case proves percent-decoded authentication; the Sentinel case kills the
original master and requires the already-created client to resume commands
after election. These are topology/lifecycle probes, not a replay of the full
shared command corpus, so this report does not claim full universal-client
command parity.
On Unix CI, the live
test_unix_url suite opens a
percent-encoded socket path through every supported Unix URL alias, applies
percent-decoded ACL credentials, selects a non-default database, negotiates
RESP3, and repeats setup through a fresh factory connection. Unix sockets are
not available on Windows, and the matrix does not claim otherwise.
Exclusive-session behavior is exercised through the constructors users call:
transactions and blocking commands own a RedisConnection, while Pub/Sub
converts an owned connection with PubSubConnection::from_connection and
explicitly replays subscriptions through reconnect_with. Those session types
do not promise transparent reconnect on their own; factory-backed multiplexed,
Cluster, and Sentinel handles provide that lifecycle. The corresponding live
tests run in the standalone, Cluster, and Sentinel per-PR suites, including
transaction abort/commit, bounded blocking, subscription replay, and topology
rediscovery. See the
standalone integration suite and
Cluster integration suite.
Server version matrix
| Server | Standalone | Cluster | Sentinel | Version-gated Redis 8 commands | Modules |
|---|---|---|---|---|---|
| Redis 7.4.3 | Per PR | Per PR | Per PR | Not applicable | No |
| Redis 8.0.6 source build | Per PR | Per PR | Per PR | Not activated on the module-free source build | No |
| Redis 8 module image | Focused Redis 8 command binary in the path-filtered gate | Not covered by this image leg | Not covered by this image leg | Per PR when relevant; actual version discovered from INFO | Per-PR capability preflight plus all module suites |
| Redis 7.2 and 7.4 container tags | Nightly | Not in the nightly matrix | Not in the nightly matrix | Not applicable | No |
| Redis 8.0, 8.2, 8.4, 8.6, and 8.8 container tags | Nightly | Not in the nightly matrix | Not in the nightly matrix | Nightly, with the corresponding version input | Redis 8 module image in the separate nightly module workflow |
| Valkey 8.1 container tag | Nightly | Not covered | Not covered | Not applicable | No |
| Redis Stack 7.4.0-v3 | No core-client matrix claim | Not covered | Not covered | Not applicable | Nightly JSON, Search, TimeSeries, and probabilistic suites |
The exact per-PR patch versions and nightly image tags are defined in
ci.yml,
nightly.yml, and
module-gate.yml, and
nightly-modules.yml. Moving
container tags intentionally test the current patch in their minor line; the
capability log records the actual INFO redis_version and required command set,
while the workflow log and image digest identify the image for a particular run.
The nightly compatibility legs run the focused integration.rs suite, not
every standalone test binary in the generated inventory.
Destructive and fault-injection inventory
| Injected condition | Required observation | Cadence | Evidence |
|---|---|---|---|
Redis command exceeds its deadline (DEBUG SLEEP or a blocking command) | A bounded CommandTimeout is surfaced before the slow command completes | Per PR | resilience tests, standalone integration |
| Connected TCP peer applies a write but withholds all or some pipeline replies | Explicit response timeout reports unknown execution, closes the old socket, and recovers on a replacement without replay or reply misalignment | Per PR, no Redis process | auto-pipeline tests |
| Refused TCP connections, then a healthy endpoint | Circuit opens at the configured threshold, rejects immediately, admits a half-open probe, and closes after recovery | Per PR | resilience tests |
| Managed standalone process stopped and restarted on the same port | The existing multiplexed client eventually resumes validated reads and writes against a fresh database | Per PR; deliberately bypassed in externally managed nightly service containers | standalone integration |
Standalone REPLICAOF role changes and coordinated FAILOVER | Replication completes, roles change, and acknowledged data survives promotion | Per PR | server integration |
| Cluster slot migration held between importing/migrating and final ownership | Client follows ASK and then MOVED, while its redirect metrics record the expected order | Per PR | cluster integration |
| Cached cluster data and invalidation connections killed on one master | Cache fails closed, clears unsafe entries, returns current data, and restores complete tracking coverage | Per PR | cluster integration |
| Master killed in a dedicated cluster and its acknowledged replica promoted | Client reaches a bounded first successful read, preserves data, and replaces the dead slot owner | Per PR | cluster integration |
| One Sentinel killed, then the monitored master killed | Discovery survives one Sentinel loss while quorum remains, Sentinel elects a different master, the existing client rediscovers it, and a fresh connection finds it | Per PR, in an isolated ordered test binary | Sentinel failover sequence |
| Pool contention with one available connection | Concurrent cluster and Sentinel callers serialize and recover rather than failing from exhaustion | Per PR | cluster suite, Sentinel suite |
| Cluster fixture startup expires, or all fixture nodes are frozen | Partial processes and artifacts are reaped; the caller’s deadline stays hard; teardown resumes stopped nodes and removes only owned state | Per PR | fixture lifecycle tests |
| Real standalone SIGKILL/restart under continuous validated load | Exact owned process dies, replacement starts on the same port, lifecycle event is observed, and useful work recovers within the bound | Path-filtered PR smoke; long run by explicit publication protocol | soak workflow, soak harness |
| Real cluster master SIGKILL under continuous validated load | Redis promotes the replicated slot owner and the client completes a validated probe within the bound | Path-filtered PR smoke; long run by explicit publication protocol | soak workflow, soak harness |
Destructive Sentinel phases use a separate port range and a single orchestrating test because killing shared processes would make independent tests order-dependent. Dedicated cluster failover and reshard fixtures use their own topologies for the same reason.
Ignored-test policy
#[ignore] is reserved for tests that need infrastructure or intentionally
mutate a shared topology: live cluster and Sentinel processes, module-enabled
servers, Docker, destructive fault injection, or signal-cleanup checks. Each
ignored test must state or inherit its prerequisite and must have one of these
execution paths:
- an explicit per-PR
cargo test ... -- --ignoredstep; - a scheduled workflow matched to the required server image; or
- a path-filtered/manual fault-harness workflow.
The mapping is visible in the CI, compatibility, per-PR module, nightly module, and soak workflows. A normal workspace test will therefore report ignored tests; that is expected. The conformance goal is no silently abandoned ignored tests, not a false claim that every environment can execute every test.
Reproduce the tiers
The process-backed commands require redis-server and redis-cli on PATH.
Cluster, Sentinel, and destructive suites should be run single-threaded to
match CI and avoid port/topology interference.
# Generated inventory and drift check
python3 scripts/generate_test_conformance.py
python3 scripts/generate_test_conformance.py --check
# Unit, rustdoc, standalone, and universal-client coverage
cargo test --lib --all-features
cargo test --doc --all-features
cargo test -p redis-tower --test '*' --all-features -- --test-threads=1
cargo test -p redis-tower-client --test '*' --all-features -- --test-threads=1
# Infrastructure-gated per-PR suites
cargo test -p redis-tower-client --test topologies --all-features -- \
--ignored --test-threads=1
cargo test -p redis-tower-cluster --test cluster_integration \
--all-features -- --ignored --test-threads=1
cargo test -p redis-tower-sentinel --test 'sentinel_*' \
--all-features -- --ignored --test-threads=1
cargo test -p redis-tower-test --lib live_ -- --ignored --test-threads=1
# The upstream-backed circuit and timeout contract only
cargo test -p redis-tower --test test_resilience_integration \
--all-features -- --test-threads=1
# Docker fixture used by nightly compatibility
REDIS_TEST_IMAGE=redis:8.8-alpine \
cargo test -p redis-chaos-tests --test docker_smoke -- --ignored
# Module suites; preflight the actual server before behavior
REDIS_URL=redis://127.0.0.1:6399 REDIS_8X_REQUIRED=1 \
cargo test -p redis-tower-modules --all-features --test capabilities \
required_server_version_and_commands_are_present -- --ignored --exact --nocapture
REDIS_URL=redis://127.0.0.1:6399 REDIS_8X_REQUIRED=1 \
cargo test -p redis-tower-modules --all-features -- --ignored
The exact short soak commands and the publication-grade long-run requirements
live in the soak harness guide. Reproduce a
nightly version leg by using the image, REDIS_URL, REDIS_8X_REQUIRED=1, and
the expected REDIS_8X_VERSION shown in
nightly.yml; that workflow is the
canonical matrix definition.
Known limits
- The old custom-vs-upstream circuit-breaker A/B from issue #416 is not runnable because the custom implementation was retired. The maintained adapter contract above is the replacement evidence.
- There is no automated true network partition, Docker pause, or Toxiproxy latency/jitter leg today. Current fault tests cover refused connects, connection kills, process SIGKILL/restart, role changes, resharding, and command stalls. Deterministic connected-peer response-stall tests verify explicit timeout, quarantine and no replay, but do not simulate packet-level partitions or node-to-node quorum loss.
- Valkey is exercised only through the standalone nightly suite. Cluster, Sentinel, module, and fault-injection parity are not claimed for Valkey.
- The nightly version matrix is standalone-only; per-PR cluster and Sentinel coverage is limited to the two pinned Redis patch versions in CI.
- Sentinel uses its default connection protocol path but does not expose the explicit protocol policy available to standalone and cluster builders.
- The per-PR module gate is path-filtered and uses one Redis 8 image. The Redis Stack 7.x feature subset and broader image cadence remain nightly.
- The short soak workflow proves harness and recovery behavior, not long-run performance, memory stability, or a publication result. Those claims require the sealed publication protocol and its retained artifacts.
- Live Redis integration, module, and transport handshakes currently run on Linux. Publishable all-feature library tests also run on macOS arm64, Windows x64, and Linux arm64. Managed services still need separate validation; protocol compatibility alone is not provider certification.