Skip to main content

Persisting JWT Signing Keys in Production

Overview​

Abblix OIDC Server signs JWT tokens using keys supplied through the IAuthServiceKeysProvider contract. The default implementation, OidcOptionsKeysProvider, reads keys from OidcOptions.SigningKeys, a property whose default value is an empty collection. In any multi-instance deployment or across process restarts, this default is unsafe: unless the host provisions persistent, shared signing keys, instances will issue tokens that other instances or future restarts cannot validate, producing intermittent invalid_token errors and mass logouts under rolling deploys.

This guide covers the IAuthServiceKeysProvider contract, the three production failure modes caused by ephemeral or per-instance keys, three storage recipes (static secret, Redis-backed, database-backed), the key rotation discipline, and a verification test suitable for CI.

Scope: Signing Keys, Not Data Protection​

Before anything else, a disambiguation worth a minute of your time.

ASP.NET Core Data Protection is the framework service that protects cookies, anti-forgery tokens, and TempData. Its key storage is a separate concern, configured via services.AddDataProtection().PersistKeysTo...(). If you've read that .NET teams routinely lose production Data Protection keys on restart and wonder whether that applies to Abblix, the answer is: it does not, because the library does not route signing through Data Protection.

Abblix OIDC Server signing keys are the JSON Web Keys (JWKs) used to sign ID tokens, access tokens, and refresh tokens. They live behind a dedicated library contract, IAuthServiceKeysProvider, and the default implementation reads from OidcOptions.SigningKeys, a strongly-typed options property.

Both problems have the same shape (a key store that must survive restarts and be shared across pods) and both failure modes look identical from the outside (random authentication errors, phantom logouts). But the remedies are unrelated. Configuring Data Protection persistence does not persist your signing keys. You have to do both.

The rest of this article is about the signing-key side exclusively.

The Library Contract​

Abblix OIDC Server exposes signing-key access through a small interface:

public interface IAuthServiceKeysProvider
{
    IAsyncEnumerable<JsonWebKey> GetEncryptionKeys(bool includePrivateKeys = false);
    IAsyncEnumerable<JsonWebKey> GetSigningKeys(bool includePrivateKeys = false);
}

IAsyncEnumerable is deliberate. Retrieval may hit a cache, a Redis hash, a database table, or an external KMS. The library does not assume synchronous in-memory access.

The default implementation is OidcOptionsKeysProvider, which reads from IOptions<OidcOptions>:

public IReadOnlyCollection<JsonWebKey> SigningKeys { get; set; } = [];
public IReadOnlyCollection<JsonWebKey> EncryptionKeys { get; set; } = [];

The default value of both properties is an empty collection. That is not a sensible production default, and it is not meant to be. The library ships an escape hatch (the options) and an extension point (the interface), and expects the host to choose which one to use before going live.

The XML documentation on OidcOptionsKeysProvider says this out loud:

It is recommended to implement a dynamic resolution mechanism in production environments to enable seamless certificate replacement without the need for service reloading.

Production Failure Modes​

Three concrete scenarios motivate the recipes in the rest of this guide. Each produces authentication errors that are hard to diagnose without knowing the underlying cause.

Multi-Pod Key Divergence​

A Kubernetes Deployment runs three pods of your auth-service. Each pod's startup code calls something like JsonWebKeyFactory.CreateRsa() to populate OidcOptions.SigningKeys. Each pod independently generates a different key pair. The three pods now have three different jwks_uri responses, three different sets of kid values, and three different private keys signing tokens that are all presented to the same load balancer.

The user impact is intermittent 401 errors that correlate with nothing the user can control. Sticky sessions partially mask the problem; a rolling deploy exposes it. Your tracing shows nothing in application code. The only clue is that failures correlate with the kid claim on the token not matching the verifying pod's published JWKS.

Diagnosis is slow because the usual suspects (token expiry, clock skew, incorrect audience) all check out.

Restart Invalidates Every Token​

A single-pod deployment has keys generated at process start. Every refresh token in circulation is signed by the current process's in-memory private key. A rolling restart, a liveness-probe kill, or a Kubernetes eviction replaces the process. New keys, new kid values, and every refresh token that was valid a minute ago is now unrecognized.

Users experience a mass logout. Mobile apps start retrying refresh, hit the auth endpoint harder than steady state, and some teams accidentally DoS themselves during a routine deploy.

Keys in Git​

The third failure mode is avoided by engineers who know not to commit .pem files, but it keeps appearing in leaked-secrets audits anyway. A developer pastes a JWK into appsettings.Development.json, copies that file into appsettings.Production.json to "unblock staging", and a month later somebody greps the repo history and finds a live production private key in commit.

The code works. The deploy works. The security review three quarters later finds the key.

Recipe A: Static Keys from a Shared Secret​

This is the simplest production-grade setup, appropriate when you have a small number of pods, a stable key set, and no strict rotation SLA.

Keys live in a Kubernetes Secret (or Vault, or AWS Secrets Manager) and are mounted into every pod as an environment variable or file. All pods read the same keys at startup and never change them until a planned rotation.

Generate a JWK once, store it in your secret manager, and load it through configuration:

services.AddOidcServices(options =>
{
    var jwkJson = Environment.GetEnvironmentVariable("ABBLIX_SIGNING_JWK")
        ?? throw new InvalidOperationException(
            "ABBLIX_SIGNING_JWK is required. See docs/signing-key-persistence.");

    var jwk = JsonSerializer.Deserialize<JsonWebKey>(jwkJson)
        ?? throw new InvalidOperationException("ABBLIX_SIGNING_JWK is not valid JSON.");

    options.SigningKeys = new[] { jwk };
});

The corresponding secret (Kubernetes example):

kubectl create secret generic abblix-signing-key \
  --from-literal=ABBLIX_SIGNING_JWK="$(cat signing-key.jwk.json)" \
  --namespace=production

And the Deployment references it:

env:
  - name: ABBLIX_SIGNING_JWK
    valueFrom:
      secretKeyRef:
        name: abblix-signing-key
        key: ABBLIX_SIGNING_JWK

What this buys you:

  • Every pod sees the same keys at startup. No multi-pod divergence.
  • A pod restart reads the same secret and produces the same jwks_uri. Existing tokens remain valid.
  • The key never appears in git, in a built image, or in a runtime memory dump anyone can trivially grab.

What this does not buy you:

  • Automatic rotation. A new key requires a manual secret update and a restart.
  • Graceful overlap during rotation. Unless you deliberately publish both the old and the new key for a window, clients will see JWKS churn and existing tokens may stop validating.

For teams with low-volume auth traffic and rotations measured in months, Recipe A is often the right answer. For high-volume services with compliance-driven rotation policies, read on.

Why not just put the JWK in appsettings.Production.json?

Two reasons. First, application settings files typically end up in your build artifact, Docker image, or deployment manifest, all of which are easier to accidentally leak than a secret manager entry. Second, appsettings.*.json is not designed for secrets and most organizations' scanning tooling does not treat it as such. Keep private keys in a dedicated secret store, always.

Recipe B: Redis-Backed Provider​

When rotation is frequent, or when you want the ability to swap keys without a deploy, a Redis-backed provider gives you a shared, out-of-process key store that every pod reads from on demand.

The library deliberately does not depend on any particular Redis client, so the host owns the dependency. That keeps Abblix OIDC Server free of transitive weight for teams who use a different backend, and it keeps version conflicts from forcing your hand.

Pseudocode (requires the host to have StackExchange.Redis available and an IConnectionMultiplexer registered):

public sealed class RedisAuthServiceKeysProvider(
    IConnectionMultiplexer redis,
    ILogger<RedisAuthServiceKeysProvider> logger) : IAuthServiceKeysProvider
{
    private const string SigningKeysHash = "abblix:signing-keys:active";
    private const string EncryptionKeysHash = "abblix:encryption-keys:active";

    public IAsyncEnumerable<JsonWebKey> GetSigningKeys(bool includePrivateKeys = false)
        => ReadKeysAsync(SigningKeysHash, includePrivateKeys);

    public IAsyncEnumerable<JsonWebKey> GetEncryptionKeys(bool includePrivateKeys = false)
        => ReadKeysAsync(EncryptionKeysHash, includePrivateKeys);

    private async IAsyncEnumerable<JsonWebKey> ReadKeysAsync(
        string hashKey, bool includePrivateKeys)
    {
        var entries = await redis.GetDatabase().HashGetAllAsync(hashKey);

        foreach (var entry in entries)
        {
            if (entry.Value.IsNullOrEmpty) continue;

            JsonWebKey? jwk;
            try { jwk = JsonSerializer.Deserialize<JsonWebKey>(entry.Value!); }
            catch (JsonException ex)
            {
                logger.LogWarning(ex, "Malformed JWK at {HashKey}/{Field}", hashKey, entry.Name);
                continue;
            }

            if (jwk is null) continue;
            yield return jwk.Sanitize(includePrivateKeys);
        }
    }
}

Host registration — register your provider after AddOidcServices() so it overrides the library default:

services.AddOidcServices(options => { /* leave SigningKeys empty; Redis provider wins */ });
services.AddSingleton<IAuthServiceKeysProvider, RedisAuthServiceKeysProvider>();

What to put in Redis:

  • A hash per key-usage (one for signing, one for encryption).
  • Each field is a kid, each value is the full JWK serialized as JSON.
  • A separate small wrapper job (or an admin UI) adds new keys and retires old ones on schedule.

Performance note: a production-quality Redis provider caches the result of ReadKeysAsync in-process for a short TTL (five to thirty seconds is typical). Reading Redis for every token-sign operation is unnecessary overhead, because keys rotate on human timescales, not per-request. Implement the cache with IMemoryCache or equivalent, and invalidate on a simple Redis pub/sub signal if you need sub-minute propagation.

A compilable sample project for Recipe B lives separately under the Oidc.Server.GettingStarted repository, where it can carry its StackExchange.Redis dependency without burdening the library.

Recipe C: Database-Backed Provider​

The library does not prescribe an ORM, and neither does this recipe. If you already have EF Core, use EF Core. If you prefer Dapper, use Dapper. If you're writing raw ADO.NET against a legacy stored-procedure layer, that works too. What matters is the table shape and the query.

Suggested table:

CREATE TABLE abblix_signing_keys (
    kid          TEXT PRIMARY KEY,
    key_usage    TEXT NOT NULL,          -- 'sig' or 'enc'
    algorithm    TEXT NOT NULL,          -- e.g. 'RS256', 'ES384'
    jwk_json     TEXT NOT NULL,          -- full JWK, including private material
    not_before   TIMESTAMPTZ,            -- NULL means effective immediately
    not_after    TIMESTAMPTZ,            -- NULL means indefinite
    is_active    BOOLEAN NOT NULL DEFAULT TRUE,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX abblix_signing_keys_active
    ON abblix_signing_keys (key_usage, not_after)
    WHERE is_active = TRUE;

The effective query for GetSigningKeys:

SELECT jwk_json
  FROM abblix_signing_keys
 WHERE key_usage = 'sig'
   AND is_active = TRUE
   AND (not_before IS NULL OR not_before <= NOW())
   AND (not_after IS NULL OR not_after > NOW())
 ORDER BY created_at DESC;

Wrap that in whichever data-access technology you already use. Cache the result in-process with the same short TTL you'd use for the Redis recipe. Implement IAuthServiceKeysProvider on top of the query-plus-cache, register it in DI, done.

Private keys at rest deserve their own conversation. If your database is unencrypted and unrotated, storing RSA private keys inside it is strictly worse than a Kubernetes Secret. Prefer column-level encryption (pgcrypto, Always Encrypted, TDE with additional protections) or store only key IDs in the database and fetch private material from a KMS per use.

Key Rotation Discipline​

Whichever storage you pick, rotation needs the same four moves.

  • Generate a new key ahead of time. Give it a unique kid, a not_before set to about ten minutes from now, and insert it as is_active = TRUE. Your JWKS endpoint will start publishing the new key immediately, but the signing pipeline will keep using the previous one until the not_before window opens.
  • Start signing with the new key. At the configured not_before, your signing code should prefer the newest active key. Relying parties who fetched JWKS in the last ten minutes already have the new kid in their cache. Freshly issued tokens validate.
  • Retain the old key for the token lifetime plus a buffer. Set the old key's not_after to now plus your longest refresh-token lifetime plus a safety margin of at least an hour. During this overlap window, both keys are published in JWKS, old tokens validate against the old key, and new tokens validate against the new key. Nobody is locked out.
  • Retire the old key. Once no extant token could still be signed by it, set is_active = FALSE and drop it from JWKS. Keep the row around for an audit trail, but stop publishing the public half.

A common beginner mistake is to rotate the key and immediately remove the old one, which invalidates every outstanding refresh token. Another is to skip not_before and start signing with a key the JWKS endpoint hasn't published yet, which causes a brief window of 401s for every client that cached JWKS too recently. The overlap window is not a nicety, it is the whole point.

How long should the overlap window be?

At minimum, the longest refresh-token lifetime you issue. If your refresh tokens live for thirty days, the old key must remain verifiable for thirty days after you stop signing with it. For most deployments, forty-five days is a reasonable default. For short-lived tokens in higher-risk environments, shorter overlap windows reduce the blast radius if a key is compromised.

A Multi-Pod Consistency Test Worth Writing​

An integration test you can run in CI, using ephemeral keys but a shared store:

  • Start two auth-service instances pointed at the same Redis or database.
  • Issue a token from instance A.
  • Present that token to instance B's token introspection endpoint.
  • Assert the introspection result is active: true.

Repeat with a forced key rotation between issue and validate. Assert the old-signed token still validates during the overlap window. Assert it fails after the window elapses.

This test catches almost every real configuration mistake in this area before it reaches production. It runs in CI alongside normal integration tests.

Registering Your Provider​

To override the library's default IAuthServiceKeysProvider, register your implementation after AddOidcServices():

services.AddOidcServices(options => { /* ... */ });
services.AddSingleton<IAuthServiceKeysProvider, MyProvider>();

Order matters. Microsoft's default DI container returns the most recently registered implementation when resolving a single service, so your AddSingleton must come after AddOidcServices() to win. Registering in the opposite order silently leaves the library default in place.

Summary​

Signing keys need three properties to work in production: they must be shared across all instances, they must survive restarts, and they must not live in source control. Abblix OIDC Server provides the IAuthServiceKeysProvider extension point exactly so you can meet all three without fighting the library.

Pick the simplest recipe that meets your rotation needs. Static keys from a secret store handle small deployments with planned rotations. A Redis-backed provider gives you dynamic rotation without a deploy. A database-backed provider integrates naturally into teams that already have durable state in Postgres or SQL Server.

Whatever you pick, back the configuration with the multi-pod consistency test. It is the practical difference between a setup you can trust under rolling-deploy conditions and one whose failures only surface under production load.