HydraCache

HydraCache is a Rust-native cache runtime for applications that need cache behavior to be explicit: keys, tags, TTL, single-flight loading, invalidation, query-result caching, and local-first distributed coordination.

The goal is not to hide caching behind a magical map. The goal is to make cache semantics visible enough for production code review. A cache entry should answer three questions:

• what exact value is being reused?
• which writes can make it unsafe?
• what freshness policy applies while the backing source changes?

HydraCache starts with a local cache because local behavior is the foundation. From there it grows toward database query-result caching and local-first distributed invalidation without hiding the database or repository layer behind magic interception.

Why HydraCache

HydraCache is useful when a service needs more than HashMap-style reuse:

• Reuse typed values locally. Local cache with typed serialization boundaries.
• Avoid duplicate miss storms. Single-flight loading for same-key requests.
• Expire stale values eventually. TTL as a fallback freshness bound.
• Remove related values after writes. Tag invalidation by entity, collection, tenant, or query family.
• Cache repository/query results. Database-neutral query policies with explicit keys and tags.
• Grow toward multi-node behavior. Local-first invalidation that can be carried to peers.

The core idea is simple: a cache key identifies one value, and tags describe which writes can invalidate groups of values. Everything else builds on that contract.

Choose Your Path

First Path

If you are new to HydraCache, read these pages in order:

1. Getting Started
2. Architecture
3. Decision Guide
4. Production Checklist
5. Keys and Tags
6. Local Cache
7. Macros
8. Database Query Caching

Concepts vs Guides

The documentation is split deliberately:

• Concepts explain why the API is shaped this way.
• Guides show how to use the API in application code.
• Reference pages describe maintenance rules for the documentation itself.

If you need to make a design decision, start with concepts. If you already know what you want to build, start with guides.

Use Use Cases for concrete scenarios and API Links when you need exact Rust signatures on docs.rs.

Common search terms for this site include memoization, function caching, repository cache, query result cache, near cache, read-through cache, stale-if-error, stale while revalidate, single-flight, and distributed invalidation.

Documentation Shape

This site is intentionally separate from the longer Quarto book draft under docs/book/. The docs site is the practical public surface. The book can remain a deeper narrative track for design history and long-form explanation.

The public docs should be self-contained. The article series below is linked as background and publication history, but the material in this site should not require a reader to leave the docs.

Project Links

• GitHub repository
• crates.io package
• docs.rs API

Article Series

The Medium series is the narrative origin of several concepts in this site:

• Part 1: Why Rust Needs Cache Semantics, Not Just Another Cache Map
• Part 2: Single-flight Is Not an Optimization
• Part 3: TTL Is Not Enough
• Part 4: Local-first Distributed Invalidation
• Part 5: Typed Query Caching in Rust

Getting Started

The quickest way to understand HydraCache is to build one local cache flow:

1. create a cache;
2. store a typed value with tags and TTL;
3. load a missing value through get_or_load;
4. invalidate the related tag after a write.

For a published crate, add HydraCache to an application:

cargo add hydracache

Inside this repository, the documentation examples use path dependencies so they are checked against the current branch.

    let cache = HydraCache::local()
        .default_ttl(Duration::from_secs(300))
        .build();

    let user_42_options = CacheOptions::new()
        .ttl(Duration::from_secs(60))
        .tag("users")
        .tag("user:42");

    cache
        .put(
            "user:42",
            User {
                id: 42,
                name: "Ada".to_owned(),
            },
            user_42_options,
        )
        .await?;

    let cached: Option<User> = cache.get("user:42").await?;
    assert_eq!(cached.as_ref().map(|user| user.name.as_str()), Some("Ada"));

    let loads = Arc::new(AtomicUsize::new(0));
    let user_43_options = CacheOptions::new()
        .ttl(Duration::from_secs(60))
        .tag("users")
        .tag("user:43");

    let loaded = cache
        .get_or_load("user:43", user_43_options, {
            let loads = Arc::clone(&loads);
            move || async move {
                loads.fetch_add(1, Ordering::Relaxed);
                Ok::<_, std::io::Error>(User {
                    id: 43,
                    name: "Grace".to_owned(),
                })
            }
        })
        .await?;

    assert_eq!(loaded.name, "Grace");
    assert_eq!(loads.load(Ordering::Relaxed), 1);

    cache.invalidate_tag("user:42").await?;
    assert_eq!(cache.get::<User>("user:42").await?, None);

The important parts are:

• values are typed at the cache boundary;
• CacheOptions carries TTL and invalidation metadata;
• the loader runs only when the value is missing or expired;
• tag invalidation removes values related to a write;
• the example is compiled by the documentation examples crate.

Run the checked example from the repository root:

cargo run --manifest-path docs-site/examples/Cargo.toml --bin quick_start

Next, read Keys and Tags. Most HydraCache usage becomes straightforward once key identity and invalidation tags are clear.

Installation

Add HydraCache to an application crate:

[dependencies]
hydracache = "0.67"

For local development inside this repository, examples use path dependencies so the documentation is checked against the branch being edited:

hydracache = { path = "../../crates/hydracache" }
hydracache-db = { path = "../../crates/hydracache-db" }

Optional Crates

HydraCache keeps adapters in separate crates so applications can opt into only the integrations they need.

Local Verification

Build the public docs site:

mdbook build docs-site

Check the runnable documentation examples:

cargo check --manifest-path docs-site/examples/Cargo.toml --all-targets

Architecture

HydraCache is built as a set of explicit cache boundaries rather than one invisible cache layer.

The local runtime owns typed serialization, TTL, tags, single-flight loading, diagnostics, and event streams. Database adapters build query-result descriptors on top of that runtime. Distributed invalidation and cluster APIs carry invalidation intent and membership metadata without hiding the local cache.

Runtime Flow

On a hit, the runtime decodes and returns the cached value. On a miss, get_or_load runs a loader and stores the result under the chosen key and tags. Concurrent same-key misses share one in-flight loader.

Query Flow

HydraCache does not parse SQL or infer table dependencies. The application names the result and the writes that can invalidate it.

Invalidation Flow

The bus propagates intent, not values. This keeps distributed behavior local-first: each process remains responsible for its own cache contents and loader code.

Cluster Flow

Client/member cluster mode adds role, node id, generation, membership, ownership, and peer-fetch vocabulary. It does not turn HydraCache into a production data grid by itself.

Use cluster APIs when the application needs stable membership diagnostics or a future route toward owner-based peer reads. Keep local cache semantics visible even when multiple processes participate.

Use Cases

HydraCache fits services where cache correctness depends on domain knowledge.

Expensive Async Work

Use get_or_load or cacheable_loader! when a value is expensive to compute and the application can name the value with a stable key.

Good examples:

• HTTP API responses keyed by tenant, endpoint, and parameters;
• generated reports keyed by tenant, date range, and permission scope;
• authorization or feature-flag lookups keyed by subject and context.

Database Query Results

Use hydracache-db or an adapter crate when a repository result is reusable and the application knows which writes make it stale.

Good examples:

• one user by id;
• one page of a tenant-scoped search;
• a small collection used on many requests;
• a repository method whose SQL or ORM shape should remain in application code.

Write-side Invalidation

Use tags when a write affects more than one cached key.

Examples:

• user:42 for one entity;
• users for collection-level reads;
• tenant:7 for tenant-wide invalidation;
• permission:abc for permission-scoped query families.

Request Storm Control

Use single-flight loaders when many requests can miss the same key at the same time. HydraCache lets one loader run while other callers join the in-flight work.

Local Near-cache Before Coordination

Use the local cache first. Add the invalidation bus or cluster APIs when several local caches need to react to the same invalidation intent.

Related terms: near cache, read-through cache, memoization, function caching, repository cache, query result cache, stale-if-error, stale while revalidate.

Decision Guide

Use this page when you know the shape of the problem but not the HydraCache API to start with.

API Choices

Use put and get for simple explicit storage.

Use get_or_load when the loader can fail and you want the full local cache API.

Use get_or_insert_with when the loader cannot fail.

Use get_or_load_with_refresh when freshness behavior is a product decision and stale fallback must be visible.

Use cacheable_loader! only after the key, tags, TTL, and loader boundary are already obvious.

Use hydracache-db when cached values are repository or query results, not arbitrary function results.

Production Checklist

Use this checklist before a cached path moves beyond experimentation.

Cache Boundary

• The key names exactly one reusable value.
• Tenant, locale, permission, feature flag, pagination, and sort dimensions are present when they affect the value.
• The cached type is serializable and stable enough for the chosen cache lifetime.
• The loader boundary is small enough to review.

Invalidation

• Every write path that can make the value stale has an explicit invalidation plan.
• Entity reads have entity tags.
• Collection or search reads have collection/query-family tags.
• Transactional writes invalidate only after commit.
• Tag names are treated as domain API, not throwaway strings.

Freshness

• TTL is a fallback bound, not the only freshness model for mutable data.
• Refresh/stale behavior is explicitly accepted by the product path.
• Stale fallback windows are bounded.
• Loader errors are observable when stale fallback is allowed.

Operations

• Hits, misses, loads, invalidations, and stale load discards are observable.
• Diagnostics are checked in local smoke tests.
• High-volume access events are enabled only when needed.
• Distributed invalidation is tested with two cache instances before adding real transport.

Documentation

• The example lives under docs-site/examples when it appears in public docs.
• The Markdown includes the checked snippet instead of copying code by hand.
• Link checks and visual smoke checks pass before publishing.

Anti-patterns

These patterns usually mean the cache boundary is hiding correctness work.

Key Without Tenant

If tenant or account scope changes the value, it belongs in the key.

Bad:

profile:42

Better:

tenant:7:profile:42

TTL As The Only Freshness Model

TTL eventually removes stale values. It does not know which write made a value unsafe.

Use tags for write-side invalidation and TTL as the fallback bound.

Permission-scoped Result Without Permission Scope

Search results, reports, and dashboards often depend on the caller’s permissions. If the permission scope changes the rows or fields, include that scope in the key.

Hidden Repository Magic

Avoid a generic repository wrapper that caches everything behind one trait. A good HydraCache call site shows key, tags, TTL or refresh policy, and the loader boundary.

Collection Invalidation Without Entity Invalidation

Updating one entity may affect both user:42 and users or tenant:7:users. Invalidate all tags that describe stale readers.

Unbounded Stale Fallback

stale_while_revalidate and stale_on_loader_error are product choices. Keep windows bounded and observable.

Caching Arbitrary Remote Code

HydraCache cluster APIs do not ship closures to another process. Keep execution local and move only invalidation intent or encoded cached bytes through explicit transports.

Cache Semantics

HydraCache treats cache behavior as application semantics, not as a hidden map lookup.

A production cache entry needs a contract:

• key: the exact result identity;
• tags: invalidation handles for writes that can make the result stale;
• TTL: a fallback freshness bound;
• loader: the backing operation used on misses;
• events and stats: evidence that the cache avoids work and invalidates the right values.

The key mistake HydraCache tries to prevent is answering a different question than the caller asked. A key like users:active may be fine for a toy app, but it is unsafe when tenants, principals, filters, policy versions, locale, region, or feature flags change the result.

Use explicit cache semantics first. Add convenience only after the manual shape is clear.

Why a Plain Map Is Not Enough

A map answers one narrow question:

does this key currently have a value?

A production cache has to answer more:

is this value still safe for this caller, tenant, query shape, and write history?

That difference is why HydraCache exposes explicit options instead of treating every cache operation as a simple get and insert.

Review Shape

When reviewing a cached operation, prefer code that makes the decision visible:

• the key is named in domain terms;
• the tags correspond to write paths;
• the TTL is a fallback, not the only correctness mechanism;
• the loader boundary is clear;
• invalidation happens after successful writes.

The API can become more ergonomic over time, but the semantics should remain visible.

Keys and Tags

Keys and tags are the center of HydraCache’s cache contract.

A key identifies exactly one cached value. A tag identifies a set of cached values that a write can make stale.

Those two jobs should stay separate.

Key Identity

A good key includes every dimension that changes the result.

This key is usually too weak:

users:active

It does not say which tenant, caller, page, sort order, locale, feature variant, or permission policy shaped the result.

A safer key is explicit:

tenant:7:users:status=active:page=1:sort=name:principal=42:policy=3

That key is longer because the result has more meaning. The length is a signal, not a failure.

Tag Invalidation

Tags answer a different question: “Which writes might make this value unsafe?”

For the same cached user search, useful tags might be:

tenant:7
users
users:search

A write to one user might invalidate user:42. A bulk import might invalidate users. A tenant-level policy change might invalidate tenant:7.

Common Mistakes

Do not use a collection tag as the key:

key = users
tag = users

That caches one result under a name that sounds like every result. It cannot distinguish active users from disabled users, page 1 from page 2, or tenant 7 from tenant 8.

Do not omit visibility dimensions:

tenant:7:users:active

If the caller’s permissions shape the result, the caller, role, permission hash, or policy version belongs in the key.

Do not assume TTL fixes a bad key. TTL can limit the time window of a wrong answer, but it cannot make the answer correct.

Practical Rule

When reviewing a cached read, ask:

1. Does the key identify one exact value shape?
2. Do tags match real write paths?
3. Does the TTL describe fallback freshness rather than primary correctness?
4. Can a future maintainer see why the key and tags were chosen?

If the answer is unclear, keep the cache call explicit. Convenience wrappers should come after the semantics are obvious.

Single-flight

Single-flight means concurrent misses for the same key share one loader call.

It is not just an optimization. It protects the backing source during bursts, cold starts, retries, and partial outages. Without it, a cache miss can multiply load exactly when the system is already under pressure.

HydraCache keeps single-flight local to the cache runtime:

• the first caller starts the loader;
• same-key callers wait for the result;
• the loaded value is stored once;
• all joined callers receive the same value or error.

The cache key matters here too. If two different queries accidentally share a key, single-flight can join requests that should have remained separate.

Why It Matters

Without single-flight, a cache miss can multiply into many concurrent backing calls:

100 requests -> 100 misses -> 100 database calls

With same-key single-flight, the first caller runs the loader and the rest wait:

100 requests -> 1 miss load + 99 joins

That is not just faster. It changes failure behavior during cold starts, cache expiry, retry bursts, and upstream slowdowns.

Boundaries

Single-flight is only as correct as the key. If a key is missing tenant, permission, filter, or pagination dimensions, the runtime may join work that should be separate.

The sequence is:

1. design the key correctly;
2. use the cache API to coalesce same-key loads;
3. observe loads and joins to confirm the cache is doing useful work.

TTL and Invalidation

TTL is a fallback freshness bound. It is not a complete consistency model.

TTL-only caching is acceptable for values where temporary staleness is harmless. It is weaker for application data that changes through known write paths. In those cases, use invalidation tags so writes can remove related cached values immediately after commit.

HydraCache separates the two concerns:

• TTL limits how long a value may be reused without refresh.
• Tags say which writes can make a value unsafe.

The useful production question is not “what TTL did we pick?” It is “what makes this value stale, and does that write path invalidate the right tags?”

TTL-Only Caching

TTL-only caching can work when:

• the value changes rarely;
• stale reads are acceptable;
• no write path can cheaply identify affected entries;
• the value is defensive or advisory rather than authoritative.

Examples include feature metadata, low-risk catalog data, or values where the product already tolerates short staleness windows.

Invalidation-First Caching

Use invalidation when the application knows the write paths.

For example, a write to user:42 can invalidate:

user:42
users
tenant:7

Those tags may remove one entity entry, several collection entries, and broader tenant-scoped query results.

The Combined Model

The strongest practical model often uses both:

• invalidate after known writes;
• keep a TTL as a fallback if an invalidation is missed or delayed.

TTL bounds the damage window. Invalidation keeps the normal path fresh.

Local-first Invalidation

Distributed invalidation should start with a correct local invalidation model.

HydraCache treats local invalidation as the source of truth:

1. the local cache removes values by key or tag;
2. the invalidation event can be published to peers;
3. peers apply the same invalidation locally;
4. each node keeps read behavior simple and observable.

This avoids making the distributed layer responsible for interpreting database writes or query semantics. The application still decides which keys and tags describe a value. The distributed layer carries that decision to other nodes.

Why Local First

If a cache cannot invalidate correctly inside one process, distributing that invalidation only spreads uncertainty.

The local model should be clear before multi-node behavior enters the picture:

• writes know which tags they affect;
• cache entries attach those tags consistently;
• invalidation happens after commit;
• metrics show which keys and tags were removed.

After that, distributed invalidation becomes a transport problem: carry the same invalidation decision to other nodes.

Expected Semantics

Local-first distributed invalidation should be treated as a freshness improvement, not a replacement for the database’s consistency model.

Applications should still design for:

• delayed messages;
• duplicated invalidation messages;
• nodes that miss a message and rely on TTL fallback;
• external writers that need their own invalidation path.

Typed Query Caching

Typed query caching starts with the Rust value type, but the type is not enough.

Vec<User> can mean many different results:

• all active users in a tenant;
• users visible to a principal;
• one page of a filtered search;
• a result shaped by a feature flag or policy version.

HydraCache separates the value type from query-result identity. The database client, ORM, or repository remains responsible for SQL execution and row mapping. HydraCache owns the cache boundary around the result: key, tags, TTL, single-flight, serialization, stale behavior, diagnostics, and invalidation.

Use explicit query policies when a result is reused:

    let local = HydraCache::local().build();
    let queries = DbCache::new(local, "db");

    let user = queries
        .entity::<User>("user", 42)
        .collection_tag("users")
        .ttl(Duration::from_secs(60))
        .fetch_with(|| async {
            // Replace this with SQLx, Diesel, SeaORM, or repository code.
            Ok::<_, std::io::Error>(User {
                id: 42,
                name: "Ada".to_owned(),
            })
        })
        .await?;

    assert_eq!(user.name, "Ada");

What the Type Gives You

The Rust type gives the cache a serialization and deserialization boundary. A cached User comes back as a User. A cached Vec<User> comes back as a Vec<User>.

That is useful, but it is not the whole contract.

The cache still needs explicit identity:

• tenant;
• principal or permission policy;
• entity id;
• filters;
• pagination;
• sort order;
• locale or region;
• feature variant;
• time window.

What the Database Keeps Owning

HydraCache should not become the query authority.

SQLx, Diesel, SeaORM, or the repository layer still own:

• query construction;
• transactions;
• row mapping;
• retries;
• database errors;
• database-specific performance tuning.

HydraCache owns the cache boundary around that query result.

What HydraCache Is Not

HydraCache is intentionally narrow in several places. These boundaries are part of the design.

Not an ORM

HydraCache does not replace SQLx, Diesel, SeaORM, or a repository layer.

The database library remains responsible for SQL, query planning, transactions, row mapping, retries, and database errors. HydraCache wraps the result boundary: key, tags, TTL, loader execution, single-flight, serialization, and invalidation.

Not SQL Interception

HydraCache does not transparently intercept arbitrary SQL and decide what to cache.

Applications need to describe query-result identity explicitly. This keeps tenant, principal, permission, filter, page, sort, feature, and policy dimensions visible in code review.

Not Automatic CDC

HydraCache does not install database triggers or provide change data capture by default.

If writes happen outside the service, those writers need an invalidation path too. The cache cannot safely infer external changes by looking only at reads.

Not a Strongly Consistent Distributed Database

Distributed invalidation can reduce stale reads across nodes, but it does not turn a local cache into a strongly consistent replicated database.

HydraCache starts local-first: make the local invalidation contract correct, then carry that invalidation to peers.

Not Just a Map

HydraCache can store typed values, but its main purpose is not “a map with TTL.”

The runtime exists to keep cache semantics explicit:

• key identity;
• tag invalidation;
• TTL and stale policy;
• single-flight loading;
• query-result caching;
• observability around hits, misses, loads, and invalidations.

This makes the API more deliberate than a raw map, and that deliberateness is the point.

Local Cache

Use the local cache when the current process can safely reuse a value without contacting a backing source.

The local cache is the full-control API. You choose:

• the key;
• the TTL and refresh behavior;
• the invalidation tags;
• the loader boundary;
• the typed value stored at that boundary.

    let cache = HydraCache::local().build();
    let loads = Arc::new(AtomicUsize::new(0));

    let first = cache
        .get_or_load(
            "profile:42",
            CacheOptions::new()
                .ttl(Duration::from_secs(60))
                .tag("profiles")
                .tag("profile:42"),
            {
                let loads = Arc::clone(&loads);
                move || async move {
                    loads.fetch_add(1, Ordering::Relaxed);
                    Ok::<_, LoadError>(Profile {
                        id: 42,
                        display_name: "Ada".to_owned(),
                    })
                }
            },
        )
        .await?;

    let second: Option<Profile> = cache.get("profile:42").await?;
    assert_eq!(second, Some(first));
    assert_eq!(loads.load(Ordering::Relaxed), 1);

    cache.invalidate_tag("profile:42").await?;
    assert_eq!(cache.get::<Profile>("profile:42").await?, None);

This guide intentionally keeps the example small:

• put stores a typed value;
• get reads the same typed value;
• get_or_load avoids repeated loader calls;
• tag invalidation removes related entries after writes.

Production code should give keys and tags names that match the domain model. A key identifies one cached value. A tag identifies a group of values that a write can make stale.

Typed Namespaces

typed::<T>("namespace") creates a typed, namespaced view over the same cache.

Use it when several call sites work with the same value type and domain namespace. The view keeps shared storage, stats, single-flight, tags, and invalidation safety, but it removes repeated type annotations at call sites and prefixes keys with the namespace.

Refresh Behavior

TTL says when a value expires. Refresh behavior says what the cache may do around expiry.

Use explicit refresh options when a production path can tolerate a recently expired value while a background refresh runs. Keep this choice visible in code review because stale fallback is a product decision, not a storage detail.

Where To Go Next

• Use Cacheable Functions when ordinary async functions need the same explicit cache boundary with less boilerplate.
• Use Local Cache API as a compact reference for local runtime methods.

Cacheable Functions

Use cacheable function helpers when ordinary async work needs the same cache boundary with less boilerplate.

The macros are intentionally explicit. They do not discover a global cache, generate hidden keys from every function argument, or hide the loader. They build CacheOptions and call the same runtime methods you could call manually.

Loader Macro

cacheable_loader! is the compact form for fallible async loaders.

    let profile_id = 42_u64;
    let profile = cacheable_loader!(
        cache = cache,
        key = "profile:42",
        tags = ["profiles", "profile:42"],
        ttl_secs = 60,
        load = move || async move {
            Ok::<_, LoadError>(Profile {
                id: profile_id,
                name: "Ada".to_owned(),
            })
        },
    )
    .await?;

    assert_eq!(profile.id, 42);

Use this when you already have a cache instance and want the call site to show key, tags, TTL, and loader in one expression.

Infallible Loader

Use cacheable_infallible! when the loader cannot fail and Ok::<_, Error>(value) would only add ceremony.

    let total = cacheable_infallible!(
        cache = cache,
        key = "profiles:count",
        tags = ["profiles"],
        ttl_secs = 60,
        load = || async { 1_u64 },
    )
    .await?;

    assert_eq!(total, 1);

The cache operation can still fail because serialization, storage, or runtime boundaries can fail. The macro only removes the loader error wrapper.

Attribute Macro

Use #[cacheable] when the cached operation is naturally an async function.

#[cacheable(
    cache = cache,
    key_segments = ["profile", profile_id],
    tag_segments = [["profile", profile_id], ["profiles"]],
    ttl_secs = 60
)]
async fn load_profile(cache: &HydraCache, profile_id: u64) -> Result<Profile, LoadError> {
    Ok(Profile {
        id: profile_id,
        name: "Ada".to_owned(),
    })
}

The cache remains an explicit function argument. The generated wrapper returns hydracache::CacheResult<T> because cache errors can occur outside the loader.

Choosing A Form

Use the explicit local cache API first when designing a new cached operation. Move to a macro when the key, tags, TTL, and loader boundary are already obvious.

Prefer:

• cacheable_loader! for one-off fallible loaders;
• cacheable_infallible! for one-off loaders that cannot fail;
• #[cacheable] for reusable async functions with stable key/tag metadata.

Typed Cache

Use a typed cache view when several call sites share one value type and one domain namespace.

    let cache = HydraCache::local().build();
    let profiles = cache.typed::<Profile>("profiles");

    profiles
        .put(
            "42",
            Profile {
                id: 42,
                display_name: "Ada".to_owned(),
            },
            CacheOptions::new()
                .ttl(Duration::from_secs(60))
                .tag_set(TagSet::new().entity("profile", 42).tag("profiles")),
        )
        .await?;

    let profile = profiles.get("42").await?;
    assert_eq!(profile.as_ref().map(|profile| profile.id), Some(42));

    cache.invalidate_tag("profile:42").await?;
    assert_eq!(profiles.get("42").await?, None);

The typed view does not create separate storage. It is a namespaced view over the same HydraCache runtime, so stats, events, invalidation safety, and single-flight behavior remain shared.

The namespace prefixes keys. In the example, the typed key 42 is stored as profiles:42 in the underlying cache. Use namespaces that match domain boundaries rather than module names.

Typed views are most useful when they remove repeated annotations without hiding cache semantics. Keep keys, tags, and TTLs visible at the call site.

Refresh and Stale Reads

TTL says when a value expires. Refresh options say what the cache may do around expiry.

Use refresh behavior only when stale fallback is acceptable for the product path. It should be visible in code review.

    let cache = HydraCache::local().build();

    cache
        .put(
            "profile:42",
            Profile {
                id: 42,
                display_name: "Ada".to_owned(),
            },
            CacheOptions::new().ttl(Duration::from_millis(10)),
        )
        .await?;

    tokio::time::sleep(Duration::from_millis(20)).await;

    let value = cache
        .get_or_load_with_refresh(
            "profile:42",
            CacheOptions::new().ttl(Duration::from_secs(60)),
            RefreshOptions::new().stale_while_revalidate(Duration::from_secs(5)),
            || async {
                Ok::<_, LoadError>(Profile {
                    id: 42,
                    display_name: "Grace".to_owned(),
                })
            },
        )
        .await?;

    assert_eq!(value.display_name, "Ada");

stale_while_revalidate lets the cache return a recently expired value while a background refresh runs. Once the stale window expires, callers return to foreground loading.

Related choices:

• refresh_ahead refreshes a value before expiry while still serving the fresh cached value;
• stale_while_revalidate serves an expired value inside a bounded window and refreshes in the background;
• stale_on_loader_error can return an expired value when the refresh loader fails.

Use diagnostics to confirm which path is happening in practice.

Macros

HydraCache macros remove repetition around cache boundaries without making the boundary implicit.

They are useful after the key, tags, TTL, and loader shape are already clear. If those choices are still being designed, start with the explicit runtime APIs first.

Function Wrappers

Use cacheable_loader! when one call site has a fallible async loader and the cache metadata belongs next to the call.

Use cacheable_infallible! when the loader returns a value directly. The cache operation can still fail because serialization, storage, or runtime boundaries can fail.

    let user_id = 42_i64;
    let user = cacheable_loader!(
        cache = cache,
        key = "user:42",
        tags = ["user:42", "users"],
        ttl_secs = 60,
        load = move || async move {
            Ok::<_, LoadError>(User {
                id: user_id,
                name: "Ada".to_owned(),
            })
        },
    )
    .await?;

    assert_eq!(user.name, "Ada");

    let count = cacheable_infallible!(
        cache = cache,
        key = "users:count",
        tags = ["users"],
        ttl_secs = 30,
        load = || async { 1_u64 },
    )
    .await?;

    assert_eq!(count, 1);

Use #[cacheable] when the cached operation is naturally a reusable async function.

#[cacheable(
    cache = cache,
    key_segments = ["profile", user_id],
    tag_segments = [["user", user_id], ["users"]],
    ttl_secs = 60
)]
async fn load_profile(cache: &HydraCache, user_id: i64) -> Result<User, LoadError> {
    Ok(User {
        id: user_id,
        name: "Ada".to_owned(),
    })
}

The cache remains an explicit argument. HydraCache does not discover a global cache and does not derive keys from every function argument.

Entity Metadata

Use HydraCacheEntity when repository code repeatedly caches entity-shaped values.

#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize, HydraCacheEntity)]
#[hydracache(entity = "user", collection = "users")]
struct User {
    #[hydracache(id)]
    id: i64,
    name: String,
}

fn assert_user_entity_contract() {
    assert_eq!(User::cache_key_for(&42), "user:42");
    assert_eq!(User::entity_tag_for(&42), "user:42");
    assert_eq!(User::collection_tag(), Some("users".to_owned()));
}

The derive produces the CacheEntity metadata used by DbCache, QueryCachePolicy, prepared policies, and invalidation plans:

• entity key: user:42;
• entity tag: user:42;
• optional collection tag: users.

Use #[hydracache(id = Type)] on the struct when the id type is generated or not represented by one named field.

Query Policies

Use query_cache_policy! when one query call site should declare the whole key/tag/freshness contract in one expression.

    let policy = query_cache_policy!(
        preset = read_mostly,
        name = "load-user",
        entity = User,
        id = user_id,
        refresh_ahead_secs = 10,
        stale_while_revalidate_secs = 300,
    );

    assert_eq!(policy.key_value(), Some("user:42"));
    assert_eq!(
        policy.tags_value(),
        &["user:42".to_owned(), "users".to_owned()]
    );

The macro does not inspect SQL. It only builds the same QueryCachePolicy that could be written with builder calls.

Use prepared_query_policy! when most metadata is stable and only the entity id changes at call time.

    let queries = DbCache::new(cache.clone(), "db");
    let load_user = queries.prepare::<User>(prepared_query_policy!(
        per_entity = User,
        name = "load-user",
        ttl_secs = 300,
    ));

    let cached = load_user
        .load_id(42, || async {
            Ok::<_, std::io::Error>(User {
                id: 42,
                name: "Ada".to_owned(),
            })
        })
        .await?;

    assert_eq!(cached.id, 42);

Prepared policies keep hot repository methods compact while preserving an explicit dynamic id boundary.

Write-side Invalidation

The entity derive also keeps write-side invalidation small.

    let pending = InvalidationPlan::new().cache_entity::<User>(42);
    let report = pending.execute(&cache).await?;

    assert_eq!(report.tag_count, 2);

Stage invalidations while preparing repository work, then execute the plan only after the database transaction commits.

Choosing A Macro

Prefer macro inputs that read like the cache contract a reviewer should approve. If a macro invocation becomes hard to review, move some metadata into named builders or prepared policies.

Database Query Caching

Use database query caching when a repository result is expensive enough to reuse and the application can describe its identity and invalidation model.

The database layer remains visible. HydraCache does not parse SQL, infer table dependencies, install triggers, or replace SQLx, Diesel, SeaORM, or a repository layer.

    let local = HydraCache::local().build();
    let queries = DbCache::new(local, "db");

    let user = queries
        .entity::<User>("user", 42)
        .collection_tag("users")
        .ttl(Duration::from_secs(60))
        .fetch_with(|| async {
            // Replace this with SQLx, Diesel, SeaORM, or repository code.
            Ok::<_, std::io::Error>(User {
                id: 42,
                name: "Ada".to_owned(),
            })
        })
        .await?;

    assert_eq!(user.name, "Ada");

Before enabling a cached query, answer:

1. What exact result does this key identify?
2. Which write paths invalidate the tags?
3. What TTL or stale behavior is acceptable if invalidation is delayed?
4. How will hits, misses, loader calls, and invalidations be observed?

For SQLx, Diesel, and SeaORM integrations, keep the adapter helper small. Drop to fetch_with when a transaction, macro-shaped query, or repository method needs more control.

Policy Macro

Use query_cache_policy! when a query has stable metadata and repeated builder calls would obscure the key/tag contract.

    let user_id = 42_i64;
    let policy = query_cache_policy!(
        preset = read_mostly,
        name = "load-user",
        entity = User,
        id = user_id,
        refresh_ahead_secs = 10,
        stale_while_revalidate_secs = 300,
    );

    assert_eq!(policy.name(), Some("load-user"));
    assert_eq!(policy.key_value(), Some("user:42"));
    assert!(policy.refresh_policy_value().is_some());

    let search = query_cache_policy!(
        name = "search-users",
        key_segments = ["tenant", 7_u64, "q", "ada:lovelace", "page", 1_u32],
        tag_segments = [["tenant", 7_u64], ["users"]],
        ttl_secs = 30,
    );

    assert_eq!(search.key_value(), Some("tenant:7:q:ada%3Alovelace:page:1"));
    assert_eq!(
        search.tags_value(),
        &["tenant:7".to_owned(), "users".to_owned()]
    );

The macro is still explicit: it names the key source, tags, TTL or preset, and optional refresh behavior. It does not inspect SQL or infer invalidation.

SQLx Adapter

Use hydracache-sqlx when SQLx remains the query authority and HydraCache owns only the cache boundary.

The adapter re-exports DbCache, policy types, HydraCacheEntity, and query_cache_policy! for SQLx users. It does not replace SQLx macros, pools, transactions, or row mapping.

    let queries = DbCache::new(HydraCache::local().build(), "sqlx");

    let user = queries
        .for_entity::<User>(42)
        .fetch_with(|| async {
            // Replace this with SQLx query_as!, a transaction, or repository code.
            Ok::<_, hydracache_sqlx::sqlx::Error>(User {
                id: 42,
                name: "Ada".to_owned(),
            })
        })
        .await?;

    assert_eq!(user.id, 42);

Use fetch_with when SQLx macros, transactions, or repository functions need to stay in charge. Use sqlx_one, sqlx_optional, and sqlx_all for small pool-like helper calls.

Diesel Adapter

Use hydracache-diesel when Diesel remains the query authority and cached values should still use HydraCache keys, tags, TTL, and diagnostics.

Diesel is synchronous, so Diesel helpers run the supplied loader on tokio::task::spawn_blocking. The loader should own or acquire its connection inside the closure.

    let queries = DieselCache::new(HydraCache::local().build(), "diesel");

    let user_name = queries
        .entity::<String>("user", 42)
        .collection_tag("users")
        .diesel_one(move || {
            // Acquire or use a Diesel connection inside this blocking closure.
            Ok::<_, hydracache_diesel::diesel::result::Error>("Ada".to_owned())
        })
        .await?;

    assert_eq!(user_name, "Ada");

Use fetch_with when custom repository code, transactions, or an async Diesel wrapper needs more control.

SeaORM Adapter

Use hydracache-seaorm when SeaORM remains the query authority and HydraCache should wrap only the query-result boundary.

    let queries = SeaOrmCache::new(HydraCache::local().build(), "seaorm");

    let user_name = queries
        .entity::<String>("user", 42)
        .collection_tag("users")
        .sea_one(|| async {
            // Replace this with a SeaORM query or repository method.
            Ok::<_, hydracache_seaorm::sea_orm::DbErr>("Ada".to_owned())
        })
        .await?;

    assert_eq!(user_name, "Ada");

Use fetch_with when a SeaORM query, transaction, or repository function does not fit the convenience helper shape.

Redis API

HydraCache can expose an optional Redis-compatible RESP edge for clients that already speak Redis.

This surface is a compatibility facade over HydraCache client-surface commands. It is not a promise that HydraCache is a full Redis replacement, and it does not add SQL/query semantics. Use it when Redis client interoperability is useful but the application still wants HydraCache ownership of cache data, TTLs, tags, diagnostics, and invalidation intent.

Crates

Application crates that only use the embedded HydraCache runtime do not need these crates.

Enable The Listener

The server disables the Redis facade by default. Enable it explicitly and keep it on a separate address from HTTP, admin, and cluster listeners.

$env:HYDRACACHE_REDIS_API_ENABLED = "true"
$env:HYDRACACHE_REDIS_ADDR = "127.0.0.1:6379"

For authenticated local or staging use:

$env:HYDRACACHE_REDIS_AUTH_REQUIRED = "true"
$env:HYDRACACHE_REDIS_AUTH_USERNAME = "default"
$env:HYDRACACHE_REDIS_AUTH_TOKEN_FILE = "C:\secrets\hydracache-redis-token.txt"

For rediss://, enable server TLS and the Redis TLS facade together:

$env:HYDRACACHE_TLS_ENABLED = "true"
$env:HYDRACACHE_TLS_CERT_PATH = "C:\certs\server.crt"
$env:HYDRACACHE_TLS_KEY_PATH = "C:\certs\server.key"
$env:HYDRACACHE_TLS_CA_PATH = "C:\certs\ca.crt"
$env:HYDRACACHE_REDIS_REDISS_ENABLED = "true"

HydraCache rejects rediss startup without complete TLS material.

Keyspace notifications are a separate off-by-default 0.68 capability:

$env:HYDRACACHE_REDIS_KEYSPACE_EVENTS_ENABLED = "true"
$env:HYDRACACHE_REDIS_MAX_EVENT_SUBSCRIPTIONS_PER_CONNECTION = "64"
$env:HYDRACACHE_REDIS_MAX_EVENT_SUBSCRIPTION_BYTES_PER_CONNECTION = "65536"

Use SUBSCRIBE/PSUBSCRIBE for __keyspace@0__:* or __keyevent@0__:*; use the matching unsubscribe command to release the live connection state. Admission is bounded by both unique subscription count and retained channel/pattern bytes. The event mode supports only cache-mutation notifications, not arbitrary Pub/Sub or PUBLISH.

Observe a native backend write from a Redis client

The server also projects successful native writes into this notification stream. Use the native typed namespace named redis: its physical redis:<key> prefix is stripped before the Redis channel is rendered. This explicit namespace fence prevents unrelated native cache entries from being exposed to Redis subscribers. Values never enter the event message or the RESP client-surface store, so a native-write notification does not make Redis GET return that native value.

This stream is at-most-once. Removing the final subscription releases the native receiver, and writes made while no subscription exists are not replayed after resubscription. Slow subscribers are disconnected on a detectable gap; they must reconnect, resubscribe, and repair state with ordinary reads. RESP/HC2 and native events retain their own source order, but no global order is claimed between those independent sources.

The example below is compiled by the documentation build. It starts the real RESP TCP listener, subscribes with redis-rs, writes with the ordinary native HydraCache API, and verifies the received Redis key event.

    let cache = HydraCache::local().build();
    let resp = Arc::new(
        RedisRespServer::new(
            Arc::new(ClientSurfaceState::new(ClientSurfaceLimits::default())?),
            RedisListenerConfig {
                keyspace_events: RedisKeyspaceEventConfig {
                    enabled: true,
                    ..RedisKeyspaceEventConfig::default()
                },
                ..RedisListenerConfig::default()
            },
        )?
        .with_native_cache_events(cache.clone()),
    );
    let listener = TcpListener::bind("127.0.0.1:0").await?;
    let address = listener.local_addr()?;
    let accepting = tokio::spawn(async move {
        while let Ok((stream, _)) = listener.accept().await {
            let resp = Arc::clone(&resp);
            tokio::spawn(async move {
                let _ = resp.serve_connection(stream).await;
            });
        }
    });

    // This is a normal Redis client using a normal keyspace subscription.
    let client = redis::Client::open(format!("redis://{address}/"))?;
    let mut pubsub = client.get_async_pubsub().await?;
    pubsub.subscribe("__keyevent@0__:set").await?;

    // The backend writes through the native HydraCache API. The `redis`
    // typed namespace is the explicit, non-leaking bridge to Redis key bytes.
    cache
        .typed::<String>("redis")
        .put("user:42", "Ada".to_owned(), CacheOptions::new())
        .await?;

    let message = tokio::time::timeout(Duration::from_secs(2), pubsub.on_message().next())
        .await?
        .ok_or("Redis subscription closed before the event")?;
    assert_eq!(message.get_channel_name(), "__keyevent@0__:set");
    assert_eq!(message.get_payload::<String>()?, "user:42");

    drop(pubsub);
    accepting.abort();

Supported Shape

The facade targets Redis string/cache-client interoperability:

• connection and introspection basics such as PING, ECHO, HELLO, AUTH, CLIENT SETNAME, COMMAND, INFO, SELECT, and TYPE;
• string and key operations such as GET, SET, MGET, MSET, DEL, and EXISTS;
• TTL operations such as EXPIRE, PEXPIRE, PERSIST, TTL, and PTTL;
• lock-oriented EVAL, EVALSHA, SCRIPT LOAD, and SCRIPT EXISTS for the supported compare-value lock scripts;
• bounded SUBSCRIBE, UNSUBSCRIBE, PSUBSCRIBE, and PUNSUBSCRIBE for off-by-default keyspace notifications; and
• HydraCache extension commands such as HC.STATS, HC.DIAGNOSTICS, HC.INVALIDATE, HC.NAMESPACE, HC.TAG, HC.SETTAGS, and HC.INVALIDATE_TAG.

Unsupported Redis data structures or broad server features should be treated as outside the facade. For example, hash commands are not the goal of this surface.

Client Examples

Any Redis client can use the facade for the supported command subset.

PING
SET user:42 Ada PX 60000
GET user:42
MSET user:43 Grace user:44 Linus
MGET user:42 user:43 user:44
TTL user:42
DEL user:44

HydraCache-specific tags are available through extension commands:

HC.SETTAGS user:42 user:42 users
HC.INVALIDATE_TAG users
HC.STATS
HC.DIAGNOSTICS

Use tags when Redis-facing clients should participate in the same invalidation model as native HydraCache code.

Boundaries

The Redis API is best for interoperability, migration seams, smoke tests, and operational probes. Prefer the native Rust API when application code can call HydraCache directly because native calls preserve typed values, loader boundaries, single-flight behavior, and compile-time policy structure.

Review these boundaries before enabling the facade:

• RESP resource limits protect frame size, array length, bulk string size, read buffer size, and idle connections.
• AUTH is optional but should be required for any shared environment.
• The listener address must not conflict with HTTP, cluster, or admin addresses.
• Redis keys map into the Redis namespace of the HydraCache client surface.
• Keyspace notifications are node-local, metadata-only, at-most-once hints. A lagging client is disconnected and must reconnect, resubscribe, and repair state with ordinary reads.
• Redis clients send bytes, not typed Rust values; typed decoding remains a native API concern.

Use Redis compatibility to meet existing clients where they are. Use HydraCache semantics to decide what should be cached and invalidated.

Diagnostics and Events

Use diagnostics to prove cache behavior locally before adding more infrastructure. The simplest useful check is: call the same cached operation twice, then inspect hit, load, and request counters.

    let cache = HydraCache::local().build();

    let first = cacheable_infallible!(
        cache = cache,
        key = "expensive:42",
        tags = ["expensive"],
        ttl_secs = 60,
        load = || async { 42_u64 },
    )
    .await?;

    let second = cacheable_infallible!(
        cache = cache,
        key = "expensive:42",
        tags = ["expensive"],
        ttl_secs = 60,
        load = || async { 7_u64 },
    )
    .await?;

    let diagnostics = cache.diagnostics().await;

    assert_eq!((first, second), (42, 42));
    assert_eq!(diagnostics.stats.loads, 1);
    assert_eq!(diagnostics.stats.hits, 1);
    assert_eq!(diagnostics.total_requests(), 2);
    assert_eq!(diagnostics.hit_ratio(), Some(0.5));
    assert!(!diagnostics.is_empty());

The first call misses and runs the loader. The second call hits the cache and avoids the fallback loader value.

Event Streams

Use event streams when an application needs to observe cache behavior without wrapping every cache call manually.

    let cache = HydraCache::local().build();
    let mut events = cache.subscribe_tag("users");

    cache
        .put("user:42", 42_u64, CacheOptions::new().tag("users"))
        .await?;

    let event = events.recv().await.expect("stored event");
    assert_eq!(event.kind(), CacheEventKind::Stored);
    assert_eq!(event.key(), Some("user:42"));

    cache.invalidate_tag("users").await?;
    let invalidation = events.recv().await.expect("tag invalidation");
    assert_eq!(invalidation.kind(), CacheEventKind::TagInvalidated);

Mutation and invalidation events are published when subscribers exist. Hit, miss, and load events are higher volume, so they are opt-in with enable_access_events(true).

    let cache = HydraCache::local()
        .enable_access_events(true)
        .event_buffer_capacity(256)
        .build();
    let mut events = cache.subscribe_access();

    let answer = cache
        .get_or_insert_with("answer", CacheOptions::new(), || async { 42_u64 })
        .await?;

    assert_eq!(answer, 42);
    let event = events.next_event().await.expect("access event");
    assert_eq!(event.kind(), CacheEventKind::Miss);

Subscribers use a bounded buffer. Slow subscribers can lag, but cache operations do not wait for listeners.

Callback Listeners

Callback listeners are useful for lightweight integration points. Keep the returned handle alive for as long as the listener should be active.

    let cache = HydraCache::local().build();
    let listener = cache.on_mutation(|event| {
        println!("cache changed: {event:?}");
    });

    cache.put("user:42", 42_u64, CacheOptions::new()).await?;
    listener.unsubscribe();

HydraCache builds owned event payloads only when an event kind is enabled and at least one active subscriber can receive it.

HydraCache Management Center

HydraCache 0.57 introduced a read-only Management Center for operating a running daemon. HydraCache 0.72 replaces that single-page view with a routed, locally bundled TypeScript application backed by the versioned, bounded /management/v1 API. It is served from the internal admin surface at /console/; legacy /cluster/overview, /metrics, and granular /actuator/hydracache/* diagnostics remain available on the same internal listener.

The console is an observe-only surface. It does not call the authz-gated write API. Operational actions still flow through the Kubernetes operator or the admin write endpoints:

• POST /admin/drain
• POST /admin/reshard
• POST /admin/backup (request acceptance only; it does not create a durable backup artifact or restore point)

The separate POST /admin/diagnostics/reset route is not an operator data API. It is disabled by default and configuration accepts HYDRACACHE_DIAGNOSTIC_RESET_ENABLED=true only for role=local with a loopback admin listener. The route refuses active HC/2 connections, pending invocations, subscriptions and sessions; after reset it returns aggregate before/after owner counts and fails unless embedded and shared client data owners are logically zero. It preserves audit history and monotonic fencing/version counters. This endpoint exists only to synchronize non-ship local memory measurements and must never be enabled in member/client deployments or exposed on a network interface.

Trust Boundary

The admin listener defaults to 127.0.0.1:9091 and is intended for local operators, Kubernetes probes, Prometheus, and port-forwarded console sessions. Expose it as an internal service only. The public client surface uses /client/v1/*; /metrics, /cluster/overview, /actuator/hydracache/*, and /console/ are not mounted there.

When the console is served from /console/, browser reads to /cluster/overview and /metrics are same-origin and need no CORS policy. If an operator hosts the bundle elsewhere, allow only read-only GET requests from a narrow origin list. Never use browser CORS as the authorization boundary for admin writes.

Source Semantics

Every cluster view carries source:

• live means the daemon has a real grid/control-plane status source.
• modeled means the daemon is exposing a local model because the real grid host is not attached for that role.

Console readers must treat missing or unknown source as modeled. Modeled views are useful, but they are not evidence of a live cluster. In particular, /cluster/overview renders modeled leader as null, even if older operator status still has a local placeholder.

For role = "member", the daemon hosts the networked grid stack: durable RaftMetadataRuntime, chitchat discovery, and the cluster raft transport. It emits source:"live" from the same raft-backed membership authority used by the cache, so /cluster/overview can report a real elected leader and quorum from reachable raft voters. local and client roles stay modeled.

The historical W6b follow-up is closed as TD-0008. The HYDRACACHE_GRID_INPROC=1 path remains only as an explicit test/development fallback.

/cluster/overview

GET /cluster/overview returns one point-in-time JSON document:

• source
• members with role, reachability, and generation
• leader with node id, term, and epoch, or null while electing/unknown
• partitions with under_replicated and effective count
• consistency with configured_default plus op_counts_by_level
• backup_age_seconds, supplied only by an attached observability source and null when no authoritative backup-age observation exists. An accepted /admin/backup response does not populate this field.
• lifecycle with reshard and upgrade phases

It is a view, not a linearizable read. Consumers should poll it and replace the whole view. They should not infer hidden members, a current consistency level, or backup freshness from absent fields.

POST /admin/backup currently validates authorization, readiness, and backup configuration, then returns an explicit request-only response. In 0.66, outcome: "accepted" is paired with durable_artifact_created: false and restore_point_available: false; the console and operators must not translate that response into backup completion.

For the networked member grid, quorum_ok is live voter-majority state. The lifecycle.reshard_phase field remains an honest lifecycle label; it is idle unless a real reshard runtime has supplied a non-idle phase.

Actuator JSON

/cluster/overview is the aggregated console view. /actuator/hydracache/* is the granular per-cache read-only actuator mounted on the same admin listener:

• GET /actuator/hydracache/health
• GET /actuator/hydracache/caches
• GET /actuator/hydracache/caches/{name}/diagnostics
• GET /actuator/hydracache/caches/{name}/stats
• GET /actuator/hydracache/cluster/staging-health
• GET /actuator/hydracache/cluster/pilot-report
• GET /actuator/hydracache/correctness

The standalone daemon registers its cache as server. Unknown cache names return 404. These routes are read-only and remain available during drain, like /metrics.

Prometheus

Scrape /metrics on the same admin listener:

scrape_configs:
  - job_name: hydracache
    metrics_path: /metrics
    static_configs:
      - targets:
          - 127.0.0.1:9091

The metric catalog is registered in docs/COMPAT.md. Topology metrics carry a bounded source="live|modeled" label, and the exporter emits cache, admission, cluster-grid, topology, and backup-age series.

Grafana Dashboard

Import docs/observability/dashboards/hydracache-overview.json into Grafana with Prometheus as the datasource. The dashboard covers hit ratio, cache traffic, admission pressure, topology, replication/repair, and backup age. cargo xtask verify includes a drift guard that parses every PromQL expr in the dashboard and rejects references to metrics not emitted by registered_metric_names().

Day-2 Observe Flow

1. Port-forward the admin listener, for example kubectl port-forward statefulset/hydracache 9091:9091.
2. Open http://127.0.0.1:9091/console/.
3. Check the source badge first. Treat modeled as a constrained local view. Treat member-role live as the daemon’s raft-backed membership/status view; leader:null means an election is in progress or no leader is currently known.
4. Check degraded state. If the console cannot reach /cluster/overview, it must show an explicit unreachable state rather than a stale healthy view.
5. Correlate /cluster/overview lifecycle and partition data with /metrics counters before running any write action through the operator/admin API.
6. Use /actuator/hydracache/caches/server/diagnostics for per-cache stats when the aggregate overview is not detailed enough.

Persistence, Operations, and Audit

The 0.72 console reads three additional admin-only resources:

• GET /management/v1/persistence separates configured backup support, observed backup age, and verified artifact evidence. Missing verification is displayed as unavailable.
• GET /management/v1/operations?limit=... returns a cursor-bound, newest-first journal for the current process generation. Accepted backup and reshard requests stay accepted until a real owner publishes later transitions.
• GET /management/v1/audit?limit=... returns redacted metadata for those transitions only. It is not presented as a complete security audit stream.

The browser contains no controls for drain, backup, reshard, repair, restore, delete, or retry and never calls a write method. Tests assert that all management requests made by the console are GET, that accepted is not completed, terminal records are immutable, cursor snapshots invalidate on a new transition, bounded eviction is reported, recovery without a retained report is unknown, and destination/identity/credential strings do not enter JSON or the DOM. Four W10 falsifiability canaries deliberately violate those rules and must fail with their registered marker.

Local W10 verification:

cargo test -p hydracache-server --test management_operations_072 --locked
cargo test -p hydracache-server --lib --locked management_operations
npm --prefix console test

Management read security and accessibility

All /management/v1/** routes require a verified internal identity plus the dedicated management.read capability. Tenant-scoped read alone is insufficient and cannot enumerate the management surface. A write-admin identity implies management read, while a management reader cannot invoke /admin/* mutations. The management reader has its own 16-permit fail-fast budget; overload returns 429, cancellation releases the permit, and admin recovery admission remains independent.

The administration listener remains the default trust boundary. Normal access is loopback or an internal port-forward. Remote deployments must place a TLS (preferably mTLS) reverse proxy in front, reject public access to the raw listener, strip every client-supplied x-hydracache-* identity/capability header, and install only identity headers derived from authenticated proxy state. Forwarded headers are not a substitute for that boundary.

Console assets are same-origin and locally bundled. Responses set a restrictive CSP, deny framing, disable MIME sniffing and referrers, and disable caching. The browser issues GET only and inserts diagnostic values with text nodes. Static checks reject raw-HTML/eval/write markers, external or inline runtime dependencies and credential-like strings. The npm lock uses exact direct versions; the supply-chain gate validates registry provenance, integrity and reviewed licenses and writes a CycloneDX 1.5 SBOM to target/management-center-0.72-sbom.cdx.json.

Status is conveyed by labels as well as color. Semantic landmarks/tables, accessible names, focus-visible navigation, focusable overflow regions, reduced motion and forced-color behavior are tested at desktop, tablet/200%-zoom and narrow-mobile sizes. Automated axe checks run against the fully populated production page; forced-colors and keyboard workflows have separate behavioral oracles so OS color substitution cannot mask structural accessibility failures.

Local W11 verification:

cargo test -p hydracache-server --test management_security_072 --locked
cargo test -p hydracache-server --lib --locked management_security
npm --prefix console run build
npm --prefix console test
npm --prefix console run supply-chain
npm --prefix console audit --audit-level=high

Real-process and fault evidence

W12 composes existing storage/Raft source proofs with a dedicated management projection matrix at docs/testing/management-center/0.72/fault-matrix.toml. Its 13 rows cover clean reopen, missing derivatives, corruption, torn artifacts, ENOSPC, uncommitted WAL, authoritative snapshot failure, commit/apply separation, stale-peer deletion, foreign identity, interrupted reconciliation, concurrent aggregation and bounded pressure. Every source and projection reference is resolved to an actual test function by the fast suite.

The process gate starts the production server in one- and three-daemon shapes. It verifies the embedded console and every management section, then runs follower kill → explicit partial truth → 32-reader pressure → same-disk restart → recovered membership. The receipt binds the binary hash, fixed seed, length-framed schedule/event digests, endpoint p95 and process resources. Retry attempts are append-only and linked; failures cannot be overwritten. Long scheduled/candidate/ship receipts remain exact-candidate environment evidence and cannot be replaced by this short local proof.

Local W12 verification:

cargo test -p hydracache-server --test management_process_072 --locked
$env:HYDRACACHE_RUN_DAEMON_PROCESS_E2E='1'
cargo test -p hydracache-server --test management_process_072 --locked one_daemon_production_management_surface_is_typed_and_honest -- --nocapture
cargo test -p hydracache-server --test management_process_072 --locked three_daemon_fault_recovery_retains_partial_truth_and_resource_bounds -- --nocapture
Remove-Item Env:\HYDRACACHE_RUN_DAEMON_PROCESS_E2E

Verification

Local W5 verification:

npm --prefix console test
cargo test -p hydracache-server --locked deploy_smoke
$env:HYDRACACHE_RUN_NETWORKED_DAEMON_E2E='1'
cargo test -p hydracache-server --test grid_host multi_node_members_form_a_cluster_and_elect_one_leader --locked -- --nocapture
Remove-Item Env:\HYDRACACHE_RUN_NETWORKED_DAEMON_E2E -ErrorAction SilentlyContinue
cargo xtask verify

cargo xtask verify skips the console specs only when Node or npm is missing. If Node is available, the console static check and Playwright specs are part of the gate.

Memory accounting and repeatable measurements

HydraCache 0.71 distinguishes three different quantities. A logical owner is a live cache entry, tag membership, request/session, event, or other retained application object. The W1 counters record those owners and their conservative retained-byte estimates. Process resident set size (RSS) is the operating system’s resident pages for the process; it also includes allocator arenas, code, stacks, and mapped pages. Container memory may additionally include file cache and kernel slab. A logical counter returning to zero does not require RSS to return to the cold floor: a bounded allocator high-water mark is possible. Conversely, a low RSS sample cannot prove an owner was released.

Public API added in 0.71

Memory accounting is opt-in through CacheBuilder::memory_instrumentation_mode. The default MemoryInstrumentationMode::Disabled keeps the request path free from accounting updates; Production enables bounded counters, and Profiling adds the correlation data used by controlled measurement runs.

Use Cache::memory_footprint_snapshot for a bounded aggregate observation. When an exact snapshot is required, first stop mutations, obtain Cache::memory_snapshot_barrier, and pass the acknowledged epoch back through MemorySnapshotRequest::Exact. An epoch mismatch or a non-quiescent cache fails loudly instead of being reported as an exact result. Cache::reconcile_memory_footprint performs the more expensive map walk intended for tests and scheduled verification, not request-path diagnostics.

RetainedByteEstimate is deliberately reporting-only in 0.71. It estimates the logical ownership of encoded keys, values, cache entries, tag data and expiry metadata. It does not replace Moka’s legacy value-byte capacity, change eviction, or promise that the estimate equals RSS. Exact Rust signatures and error variants remain authoritative in hydracache Rustdoc.

Use the frozen statistics contract for five independent paired starts, alternating B1/C order, warmup, cadence, settling, and the predeclared slope/effect/regression budgets. The S7 host profile and admission receipts pin the host, CPU/IRQ policy, kernel, instrumentation overhead, binary identity, scenario digest, source and workflow SHA. Run only one long cell at a time. Reboot and re-admit after host drift; do not splice measurements from a partial 6- or 24-hour cell into an accepted result. Preserve partial output as diagnostic evidence and use a new immutable campaign ID for a corrected attempt.

For each sample, collect cold, fill, steady, expire/delete, reset, refill, post-idle and shutdown checkpoints with their exact command and timestamps. Check counts and retained bytes against the expected cardinality at every phase. Inspect smaps, cgroup anon/file/slab, CPU, throughput, latency, errors, thermal state, IRQ distribution, background daemons and disk activity together. Do not discard a slow sample merely because it is inconvenient: the frozen rejection policy only permits identity mismatch, telemetry gap, runner instability, or a non-zero workload error.

The accepted AX42 D4 chain is indexed here. It used the exact measured commit da8d6de409a657e0260e7fbfb4ab31d8d6ad5ca8; M3, M8, M9 and both serialized M10 24-hour cells completed. The original archives are retained outside the server; the public branch contains sanitized derivatives. To verify a materialized evidence checkout, run python verify.py at its root. To recheck admission from the extracted GitHub receipts, run:

cargo run --manifest-path crates/xtask/Cargo.toml --locked -- memory-campaign-check --release 0.71 --campaigns <extracted-campaign-directory> --require-ship --expected-source-sha da8d6de409a657e0260e7fbfb4ab31d8d6ad5ca8

The extracted directory must contain one subdirectory per immutable campaign ID, with each campaign-receipt.json, identity, state, admission, and compatibility receipt in their original relative locations. A green campaign admission validates identity and completion; numerical memory claims still require the separate S10 disposition and release-claims decision.

Memory sizing: measured scope and safe use

This page gives observed AX42 reference slopes, not a portable memory reservation formula. The D0 baseline used source 906aa24cc22ad6b50b824120ed6364208484203a and workflow 9c533d1de5a25b83bcb294e5b939064737b7d9fc; its complete archived grid contains three independent processes per shape. The D4 candidate has a different SHA and may not inherit a D0 numerical improvement claim. Keep the workload, allocator, protocol, TLS setting, instrumentation, host admission and exact phase fixed before comparing numbers.

For the D0 M1 shape grid, the table uses the median steady process RSS at 50,000 and 250,000 keys. The marginal figure is (median RSS at 250k - median RSS at 50k) / 200k; it includes the stored value plus key, metadata, indexes and allocator effects in this scenario.

The D0 M6 TLS-enabled HC/2 connection grid, with no slow consumers, had median steady RSS of 22.4 MiB at 100 connections and 62.0 MiB at 1,000 connections. The observed marginal slope over that interval was 46,171 RSS bytes per additional connection. The 100-connection/100-slow- consumer cell had 22.4 MiB median RSS; this does not establish that slow clients are free under other traffic or queue pressure. See the M6 raw evidence and redaction receipt.

Do not extrapolate either slope to a different value distribution, connection protocol, TLS profile, service mix, persistence mode, allocator or host. For a deployment estimate, start from the same-profile process floor, add a measured key/value marginal term and a measured connection term, then reserve separate headroom for allocator fragmentation, transient queues, file/page cache, kernel slab, background services and recovery. Validate the complete process and cgroup under the expected peak cardinality and pressure; the conservative W2a retained-byte estimate is an accounting signal, not a replacement for a process/cgroup memory limit. Do not turn these D0 observations into a universal per-key budget or a 0.71 candidate RSS claim.

The D4 evidence index proves exact-source qualification of the memory scenarios. It does not claim a numerical RSS win: the D3 foundation and safety proposals explicitly forbid that claim, and optional efficiency work is deferred under the release policy. A future profile or allocator change needs its own preregistered paired campaign before this guidance can be updated for that candidate.

Distributed Invalidation

Use a shared invalidation bus when several cache instances should share invalidation intent.

The bus propagates invalidations, not values. Each cache keeps its own local entries. When one cache invalidates a key, tag, or all entries, peers receive the same intent and remove their local copies.

    let bus = Arc::new(InMemoryInvalidationBus::default());
    let first = HydraCache::local()
        .shared_invalidation_bus(bus.clone())
        .invalidation_node_id("first")
        .build();
    let second = HydraCache::local()
        .shared_invalidation_bus(bus)
        .invalidation_node_id("second")
        .build();

    first
        .put("user:42", 42_u64, CacheOptions::new().tag("users"))
        .await?;
    second
        .put("user:42", 42_u64, CacheOptions::new().tag("users"))
        .await?;

    let mut events = second.subscribe_tag("users");
    first.invalidate_tag("users").await?;

    let event = tokio::time::timeout(Duration::from_millis(500), events.recv())
        .await
        .expect("remote invalidation event")
        .expect("subscription stays open");

    assert_eq!(event.origin(), CacheEventOrigin::DistributedBus);
    assert!(!second.contains_key("user:42").await);
    assert_eq!(first.stats().distributed_invalidations_published, 1);
    assert_eq!(second.stats().distributed_invalidations_applied, 1);

Important semantics:

• cached values are never replicated;
• the in-memory bus is best-effort and does not replay messages after restart;
• self-originated messages are ignored to avoid echo loops;
• remote invalidations emit normal events with CacheEventOrigin::DistributedBus;
• diagnostics expose published, received, applied, lagged, decode-error, and closed-receiver counters.

Framed Boundary

InMemoryFramedInvalidationBus serializes each message into a CacheInvalidationFrame before delivery. It is still in-process, but it exercises the binary boundary that external transports can use later.

    let bus = Arc::new(InMemoryFramedInvalidationBus::for_cluster("orders", 128));
    let first = HydraCache::local()
        .shared_invalidation_bus(bus.clone())
        .invalidation_node_id("first")
        .build();
    let second = HydraCache::local()
        .shared_invalidation_bus(bus)
        .invalidation_node_id("second")
        .build();

    let _ = (first, second);

Use this for transport experiments and compatibility tests. It is not a production Redis, NATS, Postgres, or TCP adapter.

Custom Transports

External transports implement CacheInvalidationBus and return a receiver.

#[derive(Debug, Clone)]
struct MyBus;

#[async_trait]
impl CacheInvalidationBus for MyBus {
    async fn publish(&self, message: CacheInvalidationMessage) -> CacheResult<()> {
        // Send `message` through Redis, NATS, Postgres LISTEN/NOTIFY, etc.
        let _ = message;
        Ok(())
    }

    fn subscribe(&self) -> Box<dyn CacheInvalidationReceiver> {
        Box::new(MyReceiver)
    }
}

struct MyReceiver;

#[async_trait]
impl CacheInvalidationReceiver for MyReceiver {
    async fn recv(&mut self) -> CacheInvalidationReceive {
        // Return Message(...) for normal delivery, Lagged(n) for skipped messages,
        // and Closed when the stream is no longer usable.
        CacheInvalidationReceive::Closed
    }
}

Return Message(...) for normal delivery, Lagged(n) when the transport reports skipped messages, and Closed when the stream is no longer usable.

Client and Member Cluster

HydraCache::client() and HydraCache::member() are the embedded cluster shape.

A client is an application-side near-cache. A member is a cluster participant. Both can join an InMemoryCluster, share invalidation, and expose role, node id, generation, bootstrap, lifecycle, and participant diagnostics.

The cluster vocabulary is useful today even before production remote-value distribution:

• roles distinguish application clients from member nodes;
• generations protect against stale processes reusing node ids;
• local-first invalidation keeps value ownership explicit;
• diagnostics make runtime state visible to health and actuator surfaces.

The current cluster surface intentionally does not replicate cached values. It gives applications a stable model for membership, invalidation, ownership, and future peer-fetch routing while keeping local cache semantics intact.

Support Boundary

Current support includes:

• local, client, and member cache roles;
• generation-safe admission, leave, and invalidation publishing;
• in-memory cluster control plane for tests, demos, and local embedding;
• chitchat-backed discovery candidate adapter;
• raft-rs-backed metadata/control-plane adapter;
• deterministic ownership resolution over admitted members;
• transport-neutral peer-fetch over encoded cache bytes;
• read-only diagnostics and observability surfaces.

It intentionally does not yet include:

• production multi-node Raft networking or full durable Raft log storage;
• transparent remote closures or arbitrary executable code;
• value replication, backup ownership, or failover repair;
• external invalidation transports such as Redis, NATS, or Postgres LISTEN/NOTIFY;
• TLS/certificate management, external identity, or write-enabled admin APIs.

See the repository guide docs/PRODUCTION_CLUSTER_READINESS.md for the current staging checklist and non-goals.

Optional Adapters

The cluster APIs are split into optional crates so applications can adopt only the pieces they need:

• use hydracache-cluster-chitchat for chitchat-backed candidate discovery;
• use hydracache-cluster-raft for the raft-rs metadata control-plane runtime;
• use hydracache-cluster for the standard chitchat plus raft composition;
• use hydracache-cluster-transport-axum when members expose HTTP peer-fetch over encoded cache bytes.

Start with InMemoryCluster for tests and demos. Add real adapters only when the deployment has a real discovery and metadata story.

Releases

Release pages are generated from the canonical notes under docs/releases. The site wrappers do not copy their text: mdBook includes the canonical Markdown during every build. This keeps the repository notes, GitHub release source, search index and public site on the same content.

Published releases

• HydraCache 0.71.0 — memory accounting, retained-byte evidence and active expiry.
• HydraCache 0.72.0 — Management Center 2.0, bounded operational visibility and exact-candidate long-run evidence.

Release candidates

• HydraCache 0.73.0 — evidence-driven allocation/copy reductions, bounded HC/2 backpressure and immutable six-/24-hour measurement archives. Publication remains conditional on green corrected exact-SHA gates.

Verification records

• HydraCache 0.72 verification — executable gates, campaign findings and accepted evidence rules.

Release notes before 0.71 predate the public mdBook release archive and remain available in the docs/releases repository directory.

HydraCache 0.71.0 — memory accounting and retention evidence

0.71 closes the memory-ownership work begun in 0.70. It adds coherent production counters, reporting-only retained-byte accounting, bounded-owner proofs, and active-expiry cleanup. The release does not claim a numerical memory-footprint improvement, a Redis advantage, or portable sizing from the AX42 measurements.

The accepted D4 candidate is the exact source and workflow commit da8d6de409a657e0260e7fbfb4ab31d8d6ad5ca8. Its immutable M3, M8, M9 and M10 campaigns completed 10, 8, 2 and 2 jobs respectively. M10 measured the B1 baseline and C candidate for 24 hours each. The campaign checker accepted the complete chain as ship-eligible. Results are scoped to that source, workload, toolchain and admitted bare-metal host; later documentation and release-governance commits do not turn into a newly measured runtime candidate.

The D3 decision admits W2a retained-byte accounting as foundation, and W4 active expiry as a correctness fix. W2b, W5–W7, W8 and W9–W11 are explicitly deferred because the current owners remain bounded and correct and no optional optimization cleared the preregistered evidence bar. In particular, the post-idle B1 and C TTL samples have different live-key cardinalities and are not a like-for-like RSS comparison. The release policy requires this negative result to remain visible instead of presenting a no-win candidate as a performance win.

Publication requires all W0–W13 canaries, exact-commit fast and gated receipts, compatibility checks, full CI, and release-evidence --release 0.71 --require-ship to pass. The 0.67.1 prerequisite is an evidence-only milestone with a pinned accepted closure receipt, not a v0.67.1 package tag. The normal 0.70.0 tagged dependency is still checked. The measured source must have a complete green main CI run and its tag must have one green designated 0.71 admission. Legacy or nightly jobs triggered by the tag are diagnostic for this release; their result cannot substitute for either of those two required proofs.

The D4 decision and claim limits, measurement index, memory accounting guide, and D0-only sizing observations describe the method, evidence and limitations. Sanitized archives live on the separate immutable evidence/0.71/ax42/d4 Git branch; the complete raw archives and GitHub artifacts were SHA-256-verified and retained off the rented host. Raw archives are not in the release branch or main.

The v0.71.0 tag must point to the exact measured commit above. Release documentation and governance changes may be merged later, but a runtime or scenario change requires a new immutable measurement chain.

HydraCache 0.72.0 — Management Center 2.0 and operational visibility

Status: shipped 2026-09-22 from exact qualified candidate 24927c28c279c6c34ad90111ee6470b4065e0815.

0.72 replaces the single-page observe-only console with a locally bundled TypeScript/component application and a versioned, read-only /management/v1 API. Operators now have routed dashboard, member and partition, HC/1-HC/2-RESP client, namespace/cache, health, persistence, operation, metrics and audit views. Cluster aggregation is authenticated and bounded. Every observation states whether it is live, modeled or unavailable; fresh or stale; complete or partial.

The browser remains read-only. This release does not add arbitrary proxying, browser-to-member fan-out, state-changing controls, unbounded history, sensitive identifiers, a new storage engine or a numerical performance claim. Optional Prometheus history uses only fixed reviewed queries. Missing or mixed-epoch evidence is shown as unknown/partial and cannot become a green health state.

Release evidence

The candidate was frozen before final qualification. Full exact-SHA campaign 35537203094 completed 50 jobs with zero failures, including the six-hour Management Candidate Soak and real v0.71.0 to 0.72 upgrade, leadership-change, restart and rollback proof. Separate ship campaign 35556487047 completed the 24-hour confirmation. Durations, one-request-per-second protocol load, 64-key HC/1 ring, TTL, hourly fault schedule and FD/RSS/latency ceilings were not weakened.

The campaign exposed a real unbounded owner before acceptance: ClusterAdmissionBridge retained every 50 ms admission decision. The candidate bounds detailed diagnostics to a 1,024-entry FIFO while preserving monotonic lifetime and eviction counters. Unit tests overflow that bound and verify newest-entry retention; Linux one- and three-daemon isolation profiles changed the idle behavior from linear growth to a stable RSS plateau; the complete six-hour and 24-hour contours then passed.

External baseline run 35723493297 measured clean pre-feature commit 8d205fa302d81a07c19147cb4431e16390d256c3, published v0.71.0 commit da8d6de409a657e0260e7fbfb4ab31d8d6ad5ca8, and the candidate independently. Both legacy baselines retained zero origin owners after quiescence, a 20,915-byte bundle and 93.1142% executed JavaScript byte-range coverage; the 0.71 baseline also passed all 12 legacy Playwright instances. These values establish provenance and regression boundaries, not a portable speed or memory claim.

After materializing the immutable artifacts, the unchanged candidate passed:

cargo xtask management-center-check --release 0.72 --require-evidence
cargo xtask release-evidence --release 0.72 --receipts-dir target/release-evidence/receipts --require-ship

The final aggregate contains 15 ship-ready work items, zero lower-stage items and no rejection reason. The accepted normalized evidence and byte-exact key Actions ZIPs are retained on branch evidence/0.72/management-center-ship at commit 351766af9241399306a385298f0cc252aca6036c; SHA256SUMS and the artifact manifest bind the archive to the source runs. Baseline capture tooling and its accepted decision remain on evidence/0.72-baseline-capture.

The Management Center guide describes the supported operator surface. The 0.72 verification page describes the test layers, fault schedule, evidence identity and release-admission rules.

The v0.72.0 tag points to the exact measured runtime candidate above. Later release-documentation and evidence-archive commits do not become a different runtime candidate.

HydraCache 0.73.0 — Evidence-driven performance efficiency

Status: shipped 2026-10-03 from exact candidate d1db9937e61295341ac95f289bace275641b1650, tagged v0.73.0.

0.73 applies six narrowly scoped runtime improvements only after owner attribution, local falsification, and focused paired qualification. It reduces expiry-scan key copying, shares immutable event bytes and tags, transfers RESP output buffers without a redundant copy, bounds HC/2 queued outbound bytes, and caches the durable-store budget total while preserving its validating public scan. Management overhead produced a measured no-win; allocator work remains deferred behind an external profiling blocker; retained-byte admission was not applicable.

The Moka removal observer needed by production accounting is supplied by the independently published, exact-pinned hydra-moka 0.12.15-hydra.1 package. Its runtime source is byte-equivalent to the reviewed Git revision, but the registry package is deliberately treated as a new build identity. HydraCache does not depend on acceptance of the upstream Moka proposal to publish 0.73.

Compatibility and claim boundary

The runtime changes preserve public APIs, HC/1, HC/2 and RESP bytes, tagged-key and TTL semantics, tenant isolation, durable record/checksum/fsync formats, Management Center truth semantics, and same-disk rollback. The final exact-tree admission checked 44 public package/profile rows against the published compatibility baseline with cargo-semver-checks 0.50.0 and Rust 1.94.0; all 44 were classified compatible.

No portable numerical performance improvement is claimed. Five-pair focused host comparisons passed the frozen goodput, CPU/op, p99, correctness and resource guards for the earlier exact candidate. Six-hour run 36532416869 and 24-hour replacement confirmation 36839197349 passed every frozen guard for the exact final registry candidate. In the confirmation, each role completed 1.0368 billion operations and 1,442 checkpoints with zero errors, timeouts or rejections. Goodput was effectively unchanged, CPU/operation increased 0.260%, p99 improved 0.328%, and the decisive RSS/anonymous-PSS 95% upper slope bound fell 32.60%. These results admit final C73 but do not claim portable capacity, allocation savings, or Redis/Hazelcast superiority.

What the measurements actually showed

The figures below are deliberately split by evidence class. Local allocation and process screens identify owners and can reject a change, but they are not portable release benchmarks. Only the dedicated-host rows are release admission evidence, and their fixed 12,000 operations/second load must not be presented as HydraCache’s maximum throughput.

The machine-readable sources for every row live under docs/testing/performance/0.73. The byte-exact long-run packets are not stored only in Actions: the immutable branch and commit-pinned archive below are the durable source of record.

Evidence archive

The permanent W11 archive is assigned to branch evidence/0.73/w10-confirmation. Its immutable archive is commit 570a5bcb6959ecc7f01f8c80d0fc32b719832ad9. The artifact manifest and SHA256SUMS pin every accepted, rejected and interrupted attempt. Outer and nested SHA-256 values, I73-to-C73 continuation handoff, identities, workload, duration, estimator, thresholds and all five guards were independently verified. These commit-pinned links remain the release record after provider retention expires.

The branch is append-only; expiring GitHub Actions artifact links are not the release archive.

Release closure

The exact tagged tree passed contract, compatibility, packaging, supported-target and release-evidence admission. CI run 37140860241, documentation run 37140860224, and HC/2 run 37140860295 all completed successfully. The PR synthetic merge tree and candidate tree were both 5bea41dd43278a9405648939666de5024d2164cc; the product-runtime diff from measured candidate 16d2e98b6cc9e22d9ccf95eb26fe28bbbcf80f2b remained empty. Local final admission ran 3,472 tests with 3,472 passing and 82 skipped by the reviewed profile; performance-contract-check --require-ship passed and release-evidence --require-ship reported all 12 work items ship-ready. The machine-readable record, including provider and nested artifact hashes, is w11-exact-sha-ci-passed-37140860241.toml.

Failed and interrupted attempts remain append-only. Three local final-gate attempts were also kept in the W11 record: two exhausted local disk during a cold isolated build/link, and one hit the unchanged 360-second per-test limit while warming the compile-test cache. The same command then passed on the unchanged exact SHA and unchanged limits. None of these infrastructure attempts is reclassified as product evidence, and neither duration run authorizes a portable RSS, allocation, capacity, Redis or Hazelcast comparison.

The first final exact-tree admission was PR #214. Its head was the measured candidate 16d2e98b6cc9e22d9ccf95eb26fe28bbbcf80f2b; GitHub’s synthetic merge commit 269f7fafccfd8b912fb6ad4957d905c65a97044a had the same tree 92336607f21a68f563e65dc0fccccd8efaa14f7b. HC/2 run 37126998936 passed, but CI run 37126998956 and documentation run 37126998896 failed. The root failures were a 0.72-versioned docs example lockfile in the 0.73 workspace, incomplete CI-topology, memory-owner and gated-test registries, one retained compatibility test that required the workspace to remain exactly 0.72, and an instruction tripwire that emitted no benchmark summaries. Downstream admission failures were fail-closed consequences, not independent green evidence.

The failed attempt remains immutable historical evidence at w11-exact-sha-ci-rejected-37126998956.toml. The corrected candidate was authorized to repair publication inputs and CI orchestration only. Its product runtime paths are unchanged from measured candidate 16d2e98b6cc9e22d9ccf95eb26fe28bbbcf80f2b; therefore the accepted six- and 24-hour product measurements remain attributable. Documentation, governance, packaging, compatibility and instruction-tripwire gates were repeated on the corrected exact SHA and passed. The corrected local release tooling also executes the workspace format check package-by-package on Windows, avoiding process command-line error 206 without weakening rustfmt --check; Linux CI retains the canonical cargo fmt --all invocation. Its Windows Clippy path now matches the supported allocator matrix used by cargo xtask verify: all features for non-runtime workspace crates, plus explicit common, system-allocator and mimalloc runtime lanes. The unsupported Unix-only jemalloc build is not invoked on Windows; Linux retains the all-features gate that covers jemalloc. This repairs release orchestration without changing the measured product candidate or suppressing warnings.

The second exact-tree attempt used corrected SHA 58419e64627d27f8a0b2c61d24959638479ff2f4; its PR merge tree was byte-identical to candidate tree f84d209c5d779b9584af78ffdabf139fcc01e920, and the product-runtime diff from measured SHA 16d2e98b6cc9e22d9ccf95eb26fe28bbbcf80f2b was empty. Documentation run 37138395690 and HC/2 run 37138395655 passed. CI run 37138395653 still failed for two release-tooling reasons rather than a measured product regression: cargo-semver-checks 0.49.0 could not analyze rustdoc JSON v61, and the primary Rust job’s floating stable channel resolved to Rust/Clippy 1.99.0, introducing new fetch_update deprecations and double_must_use findings, including generated-code findings, under the unchanged -D warnings policy. The exact record is w11-exact-sha-ci-rejected-37138395653.toml. The final candidate therefore pins the reviewed Rust 1.94.0 toolchain in both release-critical jobs and updates only the analyzer to cargo-semver-checks 0.50.0, which supports v61. It keeps Clippy’s warning policy and feature/allocator matrix intact and does not change product runtime to satisfy a newer compiler. This second failed attempt remains immutable evidence and was not rerun. The new exact candidate then passed all three normal PR workflows and became tag v0.73.0.

The tag push exposed one more release-orchestration defect. Tag run 37149833980 completed 35 jobs successfully, skipped 14 by their declared conditions, and failed exactly one of 50 jobs: Redis Compatibility Release Proof. That job alone still selected dtolnay/rust-toolchain@stable, which resolved to Rust/Clippy 1.99.0; its unchanged -D warnings Clippy command stopped on the same new fetch_update deprecations and generated double_must_use findings before the Docker/client/oracle and resource-smoke steps could run. The rejected attempt is retained in w11-tag-ci-rejected-37149833980.toml.

Tooling commit a60fd380c90bf2c1c0445f0baedcbf46eebbdc8f removes all 21 remaining floating Rust action refs, pins the reviewed 1.94.0 toolchain, adds a regression contract, and adds a focused manual repair path that requires a full candidate SHA and verifies git rev-parse HEAD. Focused run 37150976176 checked out the unchanged tag commit d1db9937e61295341ac95f289bace275641b1650: fast checks, formatting, Redis contracts/tests, server lifecycle checks and Clippy passed; the real Docker/client/oracle matrix reported 21 passed and zero failed, and resource smoke reported two passed and zero failed. The complete repair workflow then closed with 14 successful jobs, 27 conditionally skipped jobs and zero failures or cancellations. The accepted repair is retained in w11-redis-proof-repair-passed-37150976176.toml. The failed tag run is not rewritten, the tag is not moved, and neither the product candidate nor the six- and 24-hour measurement inputs are changed or repeated. This closes a compatibility proof; it does not add a Redis throughput, capacity, latency or superiority claim.

Local Cache API

This page summarizes the main local cache methods. Use it as a map, not as a replacement for Rustdoc.

Reads And Writes

Loaders

Concurrent same-key loader calls share one in-flight load. Cache hits bypass single-flight entirely.

Keys And Tags

If a tag is invalidated while a tagged loader is still running, HydraCache skips storing that stale loader result. Callers after the invalidation start or join a fresh load instead of joining the stale one.

Typed Views

typed::<T>("namespace") creates a TypedCache<T> namespaced view over the same runtime.

It keeps shared storage, stats, single-flight, tags, and invalidation behavior, while making repeated typed operations less noisy.

Diagnostics

Use diagnostics to prove basic cache behavior locally: the first call should miss and load, and the second same-key call should hit.

See Memory Accounting for instrumentation modes, consistency requirements, retained-byte semantics and the difference between logical ownership and RSS.

For exact signatures, see API Links.

Crate Map

Use this page to choose the right crate for an application or integration.

Most application code should start with hydracache. Add adapter crates only when the application needs that integration surface.

Quality Gate

Use these checks before publishing a documentation or release branch.

cargo fmt --all -- --check
cargo check --workspace --all-targets --locked
cargo test --workspace --all-targets --locked
cargo clippy --workspace --all-targets --all-features --exclude hydracache --locked -- -D warnings
cargo clippy -p hydracache --all-targets --locked -- -D warnings
cargo test --doc --workspace --locked
cargo llvm-cov --workspace --all-targets --locked --summary-only

For the public docs site specifically:

cargo fmt --manifest-path docs-site/examples/Cargo.toml --check
cargo check --manifest-path docs-site/examples/Cargo.toml --all-targets --locked
mdbook build docs-site
node scripts/docs-link-check.mjs
node scripts/docs-visual-smoke.mjs

Cluster load stability checks live in a separate integration target. The small smoke test runs in the normal suite, and the heavier manual workload is ignored by default.

cargo test -p hydracache --test cluster_load_stability --locked -- --nocapture
cargo test -p hydracache --test cluster_load_stability --locked -- --ignored --nocapture

Coverage is tracked with cargo-llvm-cov. The current target is 95%+ line coverage for reusable library crates and a workspace trend toward 95%+, including the manual sandbox.

Release 0.72 adds exact-candidate process, compatibility, soak, resource, fuzz and canary admission on top of this general development gate. See Release 0.72 Verification for the complete public contract. Passing the commands on this page alone does not satisfy the 0.72 ship gate.

Release 0.72 Verification

HydraCache 0.72 is admitted from executable evidence, not from a checklist marked by hand. This page describes the public verification contract for the Management Center 2 release. The machine-readable source of truth remains docs/testing/release-evidence/0.72.toml together with docs/testing/gated-test-registry.toml in the repository.

What is verified

The release plan is split into work items W0-W14. Admission covers all of them:

Test layers

No single layer can replace another:

1. Fast unit, schema and registry tests validate deterministic transformations and ownership.
2. API and integration tests validate authorization, typed envelopes, partial truth and recovery.
3. Browser tests validate desktop/mobile rendering, filtering, accessibility and unavailable states.
4. Production-process tests launch the real daemon and read the exact embedded UI asset.
5. Linux-only resource tests own numerical FD and RSS claims.
6. Mixed-binary tests own 0.71/0.72 upgrade and rollback claims.
7. Candidate and ship soaks own wall-clock stability claims.
8. Fuzz, coverage and expected-red canaries prove decoder and admission boundaries.

An ignored test, an empty feature-gated test binary, a skipped environment or a prose report is not release evidence. Missing capabilities produce a loud blocking result.

Candidate and ship gates

The candidate gate runs for exactly six hours. The ship gate first repeats the candidate gate and then runs for exactly 24 hours on the same labelled self-hosted Linux runner. Both use three production daemons and a fixed seed. They poll the typed dashboard every second, send HC/1 writes over a declared 64-key ring and RESP traffic every second, restart a follower hourly, require visible partial truth followed by full recovery, and enforce latency, file-descriptor and RSS ceilings. The receipt is persisted before terminal budget assertions so a failed ceiling retains the measured baseline and final values. Receipt schema v3 additionally records every daemon’s node index/id, pid, RSS, process-lifetime high-water mark and open file descriptors every 60 seconds, so a cluster total can always be traced to one process and follower restarts remain visible.

The protocol-coexistence gate separately proves HC/1 and mTLS HC/2 against the same exact candidate. The compatibility gate uses the full-history v0.71.0 predecessor and exercises all five registered mixed-version scenarios, leader change, old-peer restart and rollback. None of these durations or binary identities can be overridden by an ordinary CI runner.

Readiness and recovery rules learned from the campaign

Dedicated-host execution hardened the harness in four places:

• Dashboard, HC/1 and RESP readiness are checked independently. Each bounded probe preserves its last protocol-specific error and failure of any surface fails the gate.
• The soak topology explicitly enables the production HC/1 listener; a TCP connection alone is not accepted as HC/1 evidence.
• Sustained HC/1 traffic uses one bounded, reusable HTTP client and consumes complete responses.
• Chaos Mesh IOChaos deletion has a bounded 120-second controller-reconciliation window. The object must still be deleted and the replacement pod must become ready; an uncleared finalizer, stale target or timeout remains a hard failure.
• Hourly fault counts use the soak’s exclusive deadline. A six-hour run therefore requires the five scheduled recoveries at hours 1-5, and a 24-hour run requires the 23 recoveries at hours 1-23. The endpoint at hour 6 or 24 is the completed duration, not another event inside the run.
• HC/1 writes use a fixed 64-key ring. This preserves one real write per second while keeping the Management Center endurance workload cardinality-bounded; dedicated retention campaigns own unbounded-cardinality and expiry-churn claims.
• The bounded-key campaign still exposed linear RSS growth. Linux isolation profiles reproduced it with an entirely idle cluster, excluding dashboard, HC/1, RESP and restart traffic. The cause was an unbounded admission-decision journal updated by the 50 ms grid loop. Detailed history is now a 1,024-entry FIFO; lifetime counters remain monotonic and report evictions. Single-node and three-node idle profiles verify that RSS reaches a plateau before the full soak is admitted.

These corrections improve fidelity and diagnostics. They do not reduce traffic, recovery or resource thresholds, and failed earlier attempts are preserved rather than overwritten.

Evidence identity

Every receipt is bound to one clean 40-character source commit. It records the gate and command digests, registry digest, binary/UI/schema/SBOM digests where applicable, fixed seed, host fingerprint, timestamps, samples, observations, resource series and artifact digests. A retry links to its predecessor and cannot overwrite it.

This has an important consequence: a documentation-only commit is still a new source commit. A receipt produced for an earlier SHA may remain useful historical evidence, but it cannot be relabeled as evidence for the new release candidate. The exact-SHA gates must be regenerated for the commit that is actually promoted.

Admission decision

The final release-evidence command validates every required receipt before aggregation. It rejects missing mixed-version, soak, resource, fuzz or coverage evidence; dirty or stale commits; changed commands or registries; mismatched artifacts; overwritten retries; and expected-red canaries that do not fail for the registered reason.

Documentation explains the contract but never grants a pass. Only validated artifacts from the frozen candidate can move 0.72 to ship status. Candidate 24927c28c279c6c34ad90111ee6470b4065e0815 passed the 50-job full campaign, six-hour candidate soak, mixed 0.71/0.72 compatibility proof and separate 24-hour ship confirmation. After the two external baselines were supplied, final admission reported 15 of 15 work items ship-ready.

The compact accepted archive, original key Actions ZIPs and SHA256SUMS are retained on the separate evidence/0.72/management-center-ship branch. Release documentation may advance after qualification, but the runtime tag remains bound to the exact measured candidate.

Documentation-site checks

Changes to this page are verified with the same site pipeline used for GitHub Pages:

cargo fmt --manifest-path docs-site/examples/Cargo.toml --check
cargo check --manifest-path docs-site/examples/Cargo.toml --all-targets --locked
mdbook build docs-site
node scripts/docs-link-check.mjs
node scripts/docs-visual-smoke.mjs

Workspace Layout

HydraCache is split into focused crates so applications can depend only on the surfaces they use.

The hydracache crate keeps public API re-exports in src/lib.rs and splits runtime code into focused modules:

API Links

Use docs.rs for exact Rust API signatures. This mdBook explains how the pieces fit together.

Core Runtime

Function Caching

Database Caching

Redis Compatibility

Adapter Crates

• https://docs.rs/hydracache-sqlx/latest/hydracache_sqlx/
• https://docs.rs/hydracache-diesel/latest/hydracache_diesel/
• https://docs.rs/hydracache-seaorm/latest/hydracache_seaorm/
• https://docs.rs/hydracache-observability/latest/hydracache_observability/
• https://docs.rs/hydracache-actuator-axum/latest/hydracache_actuator_axum/
• https://docs.rs/hydracache-redis-compat/latest/hydracache_redis_compat/
• https://docs.rs/hydracache-server/latest/hydracache_server/

Publishing Docs

The public documentation site lives under docs-site.

Local Preview

From the repository root:

mdbook serve docs-site --hostname 127.0.0.1 --port 3000

Open:

http://127.0.0.1:3000

Build

mdbook build docs-site

The generated site is written to:

docs-site/book

Example Checks

Documentation snippets are included from checked Rust files:

cargo fmt --manifest-path docs-site/examples/Cargo.toml --check
cargo check --manifest-path docs-site/examples/Cargo.toml --all-targets --locked
node scripts/docs-link-check.mjs
node --test scripts/docs-release-check.test.mjs
node scripts/docs-release-check.mjs

When an API changes, update the Rust example and the Markdown page in the same branch.

Release notes: one source of truth

Release notes are authored once under docs/releases/<version>.md. For releases from 0.71 onward, the public site adds a one-line wrapper at docs-site/src/releases/<version>.md:

{{#include ../../../docs/releases/<version>.md}}

Add the wrapper to SUMMARY.md and the release archive. docs-release-check.mjs rejects missing, copied or incorrectly wired pages. Because the canonical file is expanded by mdBook, its content is also indexed by site search. Use absolute URLs inside an externally included canonical file so that links mean the same thing on GitHub, in GitHub release notes and on the generated site.

Rust Playground

The Rust Playground run button is disabled in book.toml:

[output.html.playground]
runnable = false
copyable = true

This is deliberate. The public docs include snippets from checked examples that use local path dependencies, while Rust Playground can only run against published crates. Keep the run button disabled until the runnable examples target a published crate version and each runnable block is a complete standalone program.

Visual Smoke

With mdbook serve running:

node scripts/docs-visual-smoke.mjs

The smoke test opens the home page, architecture page, production checklist, database guide, and API links page on desktop and mobile viewports. It checks that content is present, the home logo is visible, static architecture diagrams render, and pages do not introduce horizontal overflow.

Assets

Brand assets live under:

docs-site/src/assets/brand

Use the *-256.png asset in pages and README content. Keep the original PNG as the source asset.

Publishing Target

The intended production target is GitHub Pages, optionally behind a custom domain later. The book.toml site-url is set to /hydracache/, which matches the repository GitHub Pages path.

The Documentation Site workflow builds docs-site/book, runs checks, and uploads that directory as the GitHub Pages artifact on main pushes that touch docs content. Treat docs-site as the production Pages source; other browser demos should be linked from the docs or published under their own path without replacing the docs artifact.

Versioning

The published docs should describe the released crate line that readers can install from crates.io.

Branches

main is the source of truth for the next published docs site. Documentation branches should compile examples against the code in the same branch.

Releases

When a crate release is published:

• update release notes under docs/releases;
• make sure docs.rs links point at latest unless documenting a version-specific behavior;
• build docs-site from the same commit that passed release checks;
• publish the GitHub Pages site from that reviewed state.

URLs

The repository GitHub Pages path is expected to be:

https://javaquasar.github.io/hydracache/

If a custom domain is added later, keep /hydracache/ compatibility or add redirects so existing article and README links do not break.

Examples

Public examples in docs-site/examples are branch-coupled. A documentation PR should fail if examples no longer compile against that branch.

Rust Playground examples are different: they run against published crates, not the current branch. Only enable runnable playground blocks for APIs that exist in the latest published crate version, or pin the docs to a version-specific target.

Docs Ownership

The public docs site is the primary reader-facing documentation surface.

Rules

• README.md stays short and points readers into docs-site.
• Public docs should be self-contained; Medium articles are linked only from the home page as background.
• Runnable snippets belong in docs-site/examples and are included into Markdown.
• Branch docs compile against branch code.
• Playground run buttons stay disabled until snippets target published crates.io versions.
• Brand assets live under docs-site/src/assets/brand.
• Architecture diagrams live under docs-site/src/assets/diagrams as checked static assets, not CDN-rendered runtime dependencies.
• Canonical release notes live under docs/releases; docs-site/src/releases contains only mdBook include wrappers and the release index.
• Public long-form material that already has a canonical file under docs/ is exposed through an mdBook include wrapper instead of being copied and edited independently.
• Files included from outside docs-site/src use absolute public links. Relative links would be interpreted from the generated page rather than from the canonical source file.

Review

A documentation PR should be reviewed for:

• correctness of keys, tags, TTL, refresh, and invalidation examples;
• whether new examples compile or are intentionally marked as non-runnable;
• local link integrity;
• mobile readability and lack of horizontal overflow;
• whether content duplicates README instead of linking to the right docs page.
• whether every release from 0.71 onward has an exact include wrapper and navigation entry.

Docs Release Checklist

Use this checklist before publishing a crate release or updating the public docs site.

Content

• README still reads as a short project entry point.
• Crate Map reflects all public crates and adapter names.
• API Links point to the correct docs.rs crates.
• Versioning page matches the release branch and GitHub Pages target.
• Adapter pages match current SQLx, Diesel, and SeaORM helper names.
• Production Checklist and Anti-patterns still describe current behavior.
• Release 0.72 Verification matches the executable gate registry and preserves campaign findings.
• Every docs/releases entry from 0.71 onward has an exact include wrapper, release-index entry and SUMMARY.md navigation entry.
• Canonical release notes use links that remain valid both in GitHub release notes and after mdBook inclusion.

Checks

cargo fmt --manifest-path docs-site/examples/Cargo.toml --check
cargo check --manifest-path docs-site/examples/Cargo.toml --all-targets --locked
mdbook build docs-site
node scripts/docs-link-check.mjs
node --test scripts/docs-release-check.test.mjs
node scripts/docs-release-check.mjs
node scripts/docs-visual-smoke.mjs

Publishing

• Build docs from the reviewed commit.
• Publish docs-site/book through the GitHub Pages workflow.
• Confirm no other workflow replaces the docs Pages artifact for the same release.
• Confirm favicon, logo, static diagrams, search, and mobile navigation after deploy.
• Keep article links on the home page only.
• Confirm the Release 0.72 Verification page appears in navigation and search.
• Confirm the published release page appears in the release archive, navigation and search without maintaining a second copy of its text.
• Do not describe an in-progress or historical exact-SHA attempt as evidence for a newer commit.

Legacy README Map

The root README used to carry most public documentation. It is now a short entry point. Use this map when looking for content that moved into the docs site.

Older release notes, implementation plans, and operational evidence remain under the repository docs/ directory.

Examples Contract

Documentation examples should stay tied to real code.

The public docs use checked Rust files under:

docs-site/examples/src/bin/

Markdown pages include snippets from those files with mdBook include directives. This keeps the displayed code and the compiled code in one place.

Local Checks

Build the documentation site:

mdbook build docs-site

Compile the documentation examples:

cargo check --manifest-path docs-site/examples/Cargo.toml --all-targets

For a complete docs publishing workflow, see Publishing Docs.

Rules

• Runnable examples belong in docs-site/examples.
• Markdown should include snippets instead of copying them by hand.
• Use rust,ignore only for intentionally incomplete sketches.
• When an API changes, update the example and the page in the same branch.

This means a documentation branch is checked against the code in that branch, and the published site from main reflects code that has already passed review.