Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Test conformance

This report maps redis-tower’s test surface to the claims it supports. It is intended to be citable at a particular commit: the compiled inventory is generated from the checkout, while the matrices below link each broader claim to its test code and workflow. A compiled test is not automatically a passing test, and a scheduled job is not evidence that a particular run was green; use the checks attached to the commit for that conclusion.

This page inventories compiled tests, configured workflow expansion, and observed execution evidence. Those are distinct from the logical client behaviors they exercise. The generated command capability ledger records those behaviors and separately tracks command-name availability, typed semantics, RESP wire support, and session/topology claims; neither page treats one dimension as proof of another.

Compiled inventory

The generated block is the only place this page records test counts. Keeping the numbers mechanical prevents a prose headline from drifting as tests are added, renamed, or split into new binaries. Refresh it with python3 scripts/generate_test_conformance.py; CI checks that the committed block matches the current source tree.

Compiled integration inventory

The scoreboard-scoped packages compile 756 integration tests across 45 test binaries. 212 tests are marked #[ignore] because they need explicit infrastructure; workflow selectors, rather than the annotation alone, determine whether they run.

SurfacePackageBinariesCompiled#[ignore]Pull requestScheduledNo workflow selector
Standaloneredis-tower3252005202290
Clusterredis-tower-cluster311210611200
Sentinelredis-tower-sentinel288888800
Modulesredis-tower-modules7201820180
Distributed primitivesredis-tower-primitives11601600
Total457562127562470

Counts are unique compiled tests. A test selected by both pull-request and scheduled workflows appears in both cadence columns, but only once in the compiled total.

Cross-backend command parity

The shared command corpus produces 210 compiled parity tests: 35 test cases × 6 client/topology expansions.

SurfaceTest binary / variantCompiled cases
Standaloneintegration / resp335
Standaloneintegration / root expansion35
Clustercluster_integration / multiplexed35
Clustercluster_integration / root expansion35
Sentinelsentinel_integration / multiplexed35
Sentinelsentinel_integration / root expansion35

Independent-client differential coverage

The independent redis-rs corpus contains 12 compiled test entry points. These are counted separately from shared-client topology expansions.

SurfaceTest binaryCompiled testsDefault
Standalonedifferential_redis_rs9normal
Clustercluster_integration1ignored
Sentinelsentinel_integration1ignored
Modulesdifferential_module_replies1ignored

Server compatibility matrix

CadenceTargetImplementation source
Pull requestRedis 7.4.3source build
Pull requestRedis 8.0.6source build
Nightlyredis-7.2redis:7.2-alpine
Nightlyredis-7.4redis:7.4-alpine
Nightlyredis-8.0redis:8.0-alpine
Nightlyredis-8.2redis:8.2-alpine
Nightlyredis-8.4redis:8.4-alpine
Nightlyredis-8.6redis:8.6-alpine
Nightlyredis-8.8redis:8.8-alpine
Nightlyvalkey-8.1valkey/valkey:8.1-alpine

Fault and destructive-recovery inventory

SurfaceTest binaryCompiled testDefaultWorkflow cadence
Standaloneintegrationmultiplexed_client_reconnects_after_server_restartnormalpull request, nightly/scheduled
Standalonetest_resilience_integrationcircuit_breaker_ignores_redis_command_errorsnormalpull request
Standalonetest_resilience_integrationcircuit_breaker_opens_on_repeated_failuresnormalpull request
Standalonetest_resilience_integrationcommand_timeout_fires_on_slow_commandnormalpull request
Standalonetest_servercover_replicaof_and_failovernormalpull request
Clustercluster_integrationcached_cluster_fails_closed_and_recovers_after_master_connections_are_killedignoredpull request
Clustercluster_integrationcluster_pool_exhaustion_and_recoveryignoredpull request
Clustercluster_integrationmux_cluster_handles_ask_then_moved_during_live_reshardignoredpull request
Clustercluster_integrationmux_cluster_keyless_commands_survive_default_master_failoverignoredpull request
Clustercluster_integrationmux_cluster_replaces_killed_master_after_replica_promotionignoredpull request
Clustertopology_recoverykeyless_commands_follow_the_default_master_through_failovernormalpull request
Sentinelsentinel_failoversentinel_failover_sequenceignoredpull request
Sentinelsentinel_integrationsentinel_pool_exhaustion_and_recoveryignoredpull request

Reproducibility

The generator compiles only the client-facing integration-test packages with all features and --no-run, asks each emitted libtest binary for its full and ignored listings, and inspects checked-in workflow cargo test selectors. Regenerate or verify the marked section with:

python3 scripts/generate_test_conformance.py
python3 scripts/generate_test_conformance.py --check

The inventory describes what Cargo discovers under the generator’s documented feature set. It does not claim that every test runs in a normal cargo test, because infrastructure-backed suites are deliberately #[ignore]-gated and invoked by the workflows described below. It also cannot infer a runtime early return. The Redis 8.x command binary now discovers redis_version from INFO and fails when REDIS_8X_REQUIRED=1 cannot reach a suitable server; an optional REDIS_8X_VERSION is cross-checked against that discovery rather than treated as proof by itself.

Independent-client differential corpus

The compiled differential entry points compare redis-tower with a test-only, exactly pinned redis-rs 1.7.0 adapter. They are independent-client evidence, not additional expansions of the shared redis-tower command macro, and their generated count is therefore reported separately above.

The standalone corpus runs on every per-PR Redis 7.4.3 and 8.0.6 leg under both RESP2 and RESP3. Mutations use equivalent fresh namespaces, failures retain the case/protocol/live server/client version and deterministic seed, and normalization is limited to documented unordered or pair-shaped replies. The module comparison remains #[ignore]-gated. It runs in the path-filtered per-PR module gate and the broader module-enabled nightly matrix. See the reviewed case ledger and lifecycle contract for exact inputs, dispositions, negative controls, and reproduction commands.

What superseded the original fault comparison

Issue #416 proposed running the former custom CircuitBreakerLayer and tower-resilience-circuitbreaker side by side under identical faults. That experiment was useful input while the implementation choice was still open. PR #636 subsequently made the choice: it replaced the custom circuit-breaker internals with the Redis-aware adapter in circuit_breaker.rs. There is no longer a maintained custom breaker to use as the second side of the proposed A/B matrix, so presenting that matrix as current conformance coverage would be misleading.

The maintained contract is narrower and directly tested:

  • tower-resilience-circuitbreaker owns circuit state, half-open admission, metrics, and state transitions. The adapter preserves RedisError, maps an upstream open rejection to RedisError::CircuitOpen, and exposes an operational handle.
  • The Redis classifier counts connection failures, connect timeouts, command timeouts, and recursively classified reconnect exhaustion. Redis command replies and caller errors do not degrade circuit health.
  • Unit tests in circuit_breaker.rs cover classification, open rejection, successful half-open recovery, and the distinction between a deadline exhausted while waiting for readiness and a timeout after dispatch.
  • Live tests in test_resilience_integration.rs cover refused TCP connections through open and half-open recovery, a real slow Redis command through the timeout layer, and a WRONGTYPE response that leaves the circuit closed.

This does not claim fault parity for every crate in the tower-resilience family. Redis-aware circuit breaking is the upstream-backed adapter in this repository; command timeouts and idempotency-aware retries remain redis-tower-owned policies. Upstream rate limiting and bulkhead layers can be composed by applications, but they are not part of this conformance matrix.

Continuous test tiers

TierCoverageCadenceEvidence
Unit and rustdocAll-feature library tests on stable and beta RustEvery pull request and push to mainCI workflow
StandaloneEvery redis-tower integration binary and the standalone UniversalClient path against source-built Redis 7.4.3 and 8.0.6Every pull request and push to mainCI workflow, standalone suite, universal-client suite
Universal topology entry pointsStandalone URL auth, authenticated Cluster URL selection, Sentinel URL selection, command execution, and recovery after a real Sentinel failoverStandalone on both standard per-PR Redis legs; dedicated Cluster/Sentinel fixture in the transport gateCI workflow, transport gate, UniversalClient topology suite
Independent differentialMCP-derived semantic cases through redis-tower and pinned redis-rs adapters, including RESP2/RESP3, binary/null/error, pipeline, transaction, blocking-session, topology, cancellation, and lost-reply contractsStandalone, Cluster, and Sentinel per PR; module replies in the path-filtered module gate and nightlycase ledger, standalone corpus, module corpus
ClusterShared command corpus plus routing, redirect, caching, reshard, TLS, pool, and failover scenarios against Redis 7.4.3 and 8.0.6Every pull request and push to main; invoked explicitly with --ignoredCI workflow, cluster suite
SentinelShared command corpus, discovery, routing, pool, reconnect, and isolated destructive failover scenarios against Redis 7.4.3 and 8.0.6Every pull request and push to main; invoked explicitly with --ignoredCI workflow, healthy suite, failover suite
Cluster fixture lifecyclePartial-start cleanup, promotion/reshard/kill cleanup, and hard deadlines while nodes are frozenEvery pull request and push to main; invoked explicitly with --ignoredCI workflow, fixture tests
Server compatibilityFocused integration.rs suite across supported Redis minor lines and Valkey; Redis 8 command tests on each Redis 8 legNightly and manual dispatchnightly compatibility workflow
Verified TLS transportsPrivate-CA trust and hostname verification, percent-encoded password auth, RESP3/database setup, rejection of a wrong hostname and untrusted CA, Cluster roundtrips, and bracketed IPv6 URL transportIsolated rustls-only and native-TLS-only live legs per PR; bracketed IPv6 required in the same transport gatetransport gate, standalone TLS suite, Cluster TLS suite
Redis modulesA fail-closed actual-version/COMMAND INFO preflight followed by JSON, Search, TimeSeries, probabilistic, Vector Set, and redis-rs differential assertionsPath-filtered per PR on Redis 8; broader Redis 8 and Redis Stack 7.x matrix nightly/manualmodule gate, nightly modules workflow, capability preflight
Soak fault smokeShort standalone SIGKILL/restart and cluster master-kill/promotion runs, including JSONL contract checksPull requests touching the harness/client paths, and manual dispatchsoak workflow, soak harness
RESP property and fuzzGenerated mixed RESP2/RESP3 frames, independent wire fixtures, whole-versus-fragmented semantic comparison, and arbitrary malformed inputProperty and ten-second fuzz smoke per PR; bounded weekly/manual campaignsprotocol contract, fuzz targets, scheduled workflow
RESP decode copy/allocationExact test-only first-frame copied-byte counts; simple, mixed, retained-lifetime, and fragmented Criterion cases; instrumented comparison of historical, production, and split/shared ownershipCopied-byte unit contract and Criterion regression suite per PR; raw allocation artifact retained when ownership changesownership decision, raw evidence, benchmark sources

The per-PR cluster and Sentinel commands intentionally include --ignored. In this repository, that flag marks an infrastructure boundary, not an optional quality tier. In the remaining tables, “per PR” also includes the same CI tier on pushes to main.

Client and protocol parity

The reusable command_tests! corpus exercises strings, keys, hashes, lists, sets, sorted sets, bitmaps, geospatial commands, HyperLogLog, and streams through the same typed-command API. It is instantiated along these paths:

TopologyClient pathProtocol policy exercised by the shared corpusAutomated cadence
StandaloneRedisConnectionForced RESP2Per PR on Redis 7.4.3 and 8.0.6; nightly version matrix
StandaloneRedisConnectionForced RESP3Per PR on Redis 7.4.3 and 8.0.6; nightly version matrix
ClusterClusterConnectionDefault Auto policy across routed connectionsPer PR on Redis 7.4.3 and 8.0.6
ClusterMultiplexedClusterClientDefault Auto policy across pipelined node connectionsPer PR on Redis 7.4.3 and 8.0.6
SentinelSentinelConnectionDefault Sentinel connection path; no forced-protocol parity claimPer PR on Redis 7.4.3 and 8.0.6
SentinelMultiplexedSentinelClientDefault Sentinel connection path; no forced-protocol parity claimPer PR on Redis 7.4.3 and 8.0.6

The generated parity total counts compiled expansions, not assertions that run unchanged on every server version. Six Vector Set cases probe capability and return early when the server does not support those commands.

Standalone also has focused negotiation coverage for forced RESP2, forced RESP3, Auto, and the default connection path in integration.rs. Cluster connection builders retain an explicit protocol policy across discovery, redirects, topology refresh, and reconnect, with focused coverage in connection.rs and multiplexed.rs. Sentinel does not yet expose equivalent explicit protocol selection, so its rows above must not be read as RESP2/RESP3 parity.

UniversalClient has focused public-entry tests for all three variants. The Cluster case proves percent-decoded authentication; the Sentinel case kills the original master and requires the already-created client to resume commands after election. These are topology/lifecycle probes, not a replay of the full shared command corpus, so this report does not claim full universal-client command parity.

On Unix CI, the live test_unix_url suite opens a percent-encoded socket path through every supported Unix URL alias, applies percent-decoded ACL credentials, selects a non-default database, negotiates RESP3, and repeats setup through a fresh factory connection. Unix sockets are not available on Windows, and the matrix does not claim otherwise.

Exclusive-session behavior is exercised through the constructors users call: transactions and blocking commands own a RedisConnection, while Pub/Sub converts an owned connection with PubSubConnection::from_connection and explicitly replays subscriptions through reconnect_with. Those session types do not promise transparent reconnect on their own; factory-backed multiplexed, Cluster, and Sentinel handles provide that lifecycle. The corresponding live tests run in the standalone, Cluster, and Sentinel per-PR suites, including transaction abort/commit, bounded blocking, subscription replay, and topology rediscovery. See the standalone integration suite and Cluster integration suite.

Server version matrix

ServerStandaloneClusterSentinelVersion-gated Redis 8 commandsModules
Redis 7.4.3Per PRPer PRPer PRNot applicableNo
Redis 8.0.6 source buildPer PRPer PRPer PRNot activated on the module-free source buildNo
Redis 8 module imageFocused Redis 8 command binary in the path-filtered gateNot covered by this image legNot covered by this image legPer PR when relevant; actual version discovered from INFOPer-PR capability preflight plus all module suites
Redis 7.2 and 7.4 container tagsNightlyNot in the nightly matrixNot in the nightly matrixNot applicableNo
Redis 8.0, 8.2, 8.4, 8.6, and 8.8 container tagsNightlyNot in the nightly matrixNot in the nightly matrixNightly, with the corresponding version inputRedis 8 module image in the separate nightly module workflow
Valkey 8.1 container tagNightlyNot coveredNot coveredNot applicableNo
Redis Stack 7.4.0-v3No core-client matrix claimNot coveredNot coveredNot applicableNightly JSON, Search, TimeSeries, and probabilistic suites

The exact per-PR patch versions and nightly image tags are defined in ci.yml, nightly.yml, and module-gate.yml, and nightly-modules.yml. Moving container tags intentionally test the current patch in their minor line; the capability log records the actual INFO redis_version and required command set, while the workflow log and image digest identify the image for a particular run. The nightly compatibility legs run the focused integration.rs suite, not every standalone test binary in the generated inventory.

Destructive and fault-injection inventory

Injected conditionRequired observationCadenceEvidence
Redis command exceeds its deadline (DEBUG SLEEP or a blocking command)A bounded CommandTimeout is surfaced before the slow command completesPer PRresilience tests, standalone integration
Connected TCP peer applies a write but withholds all or some pipeline repliesExplicit response timeout reports unknown execution, closes the old socket, and recovers on a replacement without replay or reply misalignmentPer PR, no Redis processauto-pipeline tests
Refused TCP connections, then a healthy endpointCircuit opens at the configured threshold, rejects immediately, admits a half-open probe, and closes after recoveryPer PRresilience tests
Managed standalone process stopped and restarted on the same portThe existing multiplexed client eventually resumes validated reads and writes against a fresh databasePer PR; deliberately bypassed in externally managed nightly service containersstandalone integration
Standalone REPLICAOF role changes and coordinated FAILOVERReplication completes, roles change, and acknowledged data survives promotionPer PRserver integration
Cluster slot migration held between importing/migrating and final ownershipClient follows ASK and then MOVED, while its redirect metrics record the expected orderPer PRcluster integration
Cached cluster data and invalidation connections killed on one masterCache fails closed, clears unsafe entries, returns current data, and restores complete tracking coveragePer PRcluster integration
Master killed in a dedicated cluster and its acknowledged replica promotedClient reaches a bounded first successful read, preserves data, and replaces the dead slot ownerPer PRcluster integration
One Sentinel killed, then the monitored master killedDiscovery survives one Sentinel loss while quorum remains, Sentinel elects a different master, the existing client rediscovers it, and a fresh connection finds itPer PR, in an isolated ordered test binarySentinel failover sequence
Pool contention with one available connectionConcurrent cluster and Sentinel callers serialize and recover rather than failing from exhaustionPer PRcluster suite, Sentinel suite
Cluster fixture startup expires, or all fixture nodes are frozenPartial processes and artifacts are reaped; the caller’s deadline stays hard; teardown resumes stopped nodes and removes only owned statePer PRfixture lifecycle tests
Real standalone SIGKILL/restart under continuous validated loadExact owned process dies, replacement starts on the same port, lifecycle event is observed, and useful work recovers within the boundPath-filtered PR smoke; long run by explicit publication protocolsoak workflow, soak harness
Real cluster master SIGKILL under continuous validated loadRedis promotes the replicated slot owner and the client completes a validated probe within the boundPath-filtered PR smoke; long run by explicit publication protocolsoak workflow, soak harness

Destructive Sentinel phases use a separate port range and a single orchestrating test because killing shared processes would make independent tests order-dependent. Dedicated cluster failover and reshard fixtures use their own topologies for the same reason.

Ignored-test policy

#[ignore] is reserved for tests that need infrastructure or intentionally mutate a shared topology: live cluster and Sentinel processes, module-enabled servers, Docker, destructive fault injection, or signal-cleanup checks. Each ignored test must state or inherit its prerequisite and must have one of these execution paths:

  • an explicit per-PR cargo test ... -- --ignored step;
  • a scheduled workflow matched to the required server image; or
  • a path-filtered/manual fault-harness workflow.

The mapping is visible in the CI, compatibility, per-PR module, nightly module, and soak workflows. A normal workspace test will therefore report ignored tests; that is expected. The conformance goal is no silently abandoned ignored tests, not a false claim that every environment can execute every test.

Reproduce the tiers

The process-backed commands require redis-server and redis-cli on PATH. Cluster, Sentinel, and destructive suites should be run single-threaded to match CI and avoid port/topology interference.

# Generated inventory and drift check
python3 scripts/generate_test_conformance.py
python3 scripts/generate_test_conformance.py --check

# Unit, rustdoc, standalone, and universal-client coverage
cargo test --lib --all-features
cargo test --doc --all-features
cargo test -p redis-tower --test '*' --all-features -- --test-threads=1
cargo test -p redis-tower-client --test '*' --all-features -- --test-threads=1

# Infrastructure-gated per-PR suites
cargo test -p redis-tower-client --test topologies --all-features -- \
  --ignored --test-threads=1
cargo test -p redis-tower-cluster --test cluster_integration \
  --all-features -- --ignored --test-threads=1
cargo test -p redis-tower-sentinel --test 'sentinel_*' \
  --all-features -- --ignored --test-threads=1
cargo test -p redis-tower-test --lib live_ -- --ignored --test-threads=1

# The upstream-backed circuit and timeout contract only
cargo test -p redis-tower --test test_resilience_integration \
  --all-features -- --test-threads=1

# Docker fixture used by nightly compatibility
REDIS_TEST_IMAGE=redis:8.8-alpine \
  cargo test -p redis-chaos-tests --test docker_smoke -- --ignored

# Module suites; preflight the actual server before behavior
REDIS_URL=redis://127.0.0.1:6399 REDIS_8X_REQUIRED=1 \
  cargo test -p redis-tower-modules --all-features --test capabilities \
  required_server_version_and_commands_are_present -- --ignored --exact --nocapture
REDIS_URL=redis://127.0.0.1:6399 REDIS_8X_REQUIRED=1 \
  cargo test -p redis-tower-modules --all-features -- --ignored

The exact short soak commands and the publication-grade long-run requirements live in the soak harness guide. Reproduce a nightly version leg by using the image, REDIS_URL, REDIS_8X_REQUIRED=1, and the expected REDIS_8X_VERSION shown in nightly.yml; that workflow is the canonical matrix definition.

Known limits

  • The old custom-vs-upstream circuit-breaker A/B from issue #416 is not runnable because the custom implementation was retired. The maintained adapter contract above is the replacement evidence.
  • There is no automated true network partition, Docker pause, or Toxiproxy latency/jitter leg today. Current fault tests cover refused connects, connection kills, process SIGKILL/restart, role changes, resharding, and command stalls. Deterministic connected-peer response-stall tests verify explicit timeout, quarantine and no replay, but do not simulate packet-level partitions or node-to-node quorum loss.
  • Valkey is exercised only through the standalone nightly suite. Cluster, Sentinel, module, and fault-injection parity are not claimed for Valkey.
  • The nightly version matrix is standalone-only; per-PR cluster and Sentinel coverage is limited to the two pinned Redis patch versions in CI.
  • Sentinel uses its default connection protocol path but does not expose the explicit protocol policy available to standalone and cluster builders.
  • The per-PR module gate is path-filtered and uses one Redis 8 image. The Redis Stack 7.x feature subset and broader image cadence remain nightly.
  • The short soak workflow proves harness and recovery behavior, not long-run performance, memory stability, or a publication result. Those claims require the sealed publication protocol and its retained artifacts.
  • Live Redis integration, module, and transport handshakes currently run on Linux. Publishable all-feature library tests also run on macOS arm64, Windows x64, and Linux arm64. Managed services still need separate validation; protocol compatibility alone is not provider certification.