Skip to main content

Persisting JWT Signing Keys in Production

Overview​

Abblix OIDC Server signs JWT tokens using keys supplied through the IAuthServiceKeysProvider contract, empty by default. In any multi-instance deployment or across process restarts, this default is unsafe: unless the host provisions persistent, shared signing keys, instances will issue tokens that other instances or future restarts cannot validate. The result is intermittent invalid_token errors and mass logouts under rolling deploys.

This guide covers the IAuthServiceKeysProvider contract, the three production failure modes caused by ephemeral or per-instance keys, three storage recipes (static secret, Redis-backed, database-backed), encrypting the stored key at rest, the key rotation discipline, and a verification test suitable for CI.

Scope: signing keys, not Data Protection​

Before anything else, a disambiguation worth a minute of your time.

ASP.NET Core Data Protection is the framework service that protects cookies, anti-forgery tokens, and TempData. Its key storage is a separate concern, configured via services.AddDataProtection().PersistKeysTo...(). If you've read that .NET teams routinely lose production Data Protection keys on restart and wonder whether that applies to Abblix, the answer is: it does not, because the library does not route signing through Data Protection.

Abblix OIDC Server signing keys are the JSON Web Keys (JWKs) used to sign ID tokens, access tokens, and refresh tokens. They live behind a dedicated library contract, IAuthServiceKeysProvider.

Both problems have the same shape (a key store that must survive restarts and be shared across pods) and both failure modes look identical from the outside (random authentication errors, phantom logouts). But the remedies are unrelated. Configuring Data Protection persistence does not persist your signing keys. You have to do both.

The rest of this article is about the signing-key side; Data Protection returns only later, as a tool for encrypting that key at rest, never for its own cookie and anti-forgery duties.

The library contract​

Abblix OIDC Server exposes signing-key access through a small interface:

public interface IAuthServiceKeysProvider
{
    IAsyncEnumerable<JsonWebKey> GetEncryptionKeys(bool includePrivateKeys = false);
    IAsyncEnumerable<JsonWebKey> GetSigningKeys(bool includePrivateKeys = false);
}

IAsyncEnumerable is deliberate. Retrieval may hit a cache, a Redis hash, a database table, or an external KMS. The library does not assume synchronous in-memory access.

The default implementation is OidcOptionsKeysProvider, which reads from IOptions<OidcOptions>:

public IReadOnlyCollection<JsonWebKey> SigningKeys { get; set; } = [];
public IReadOnlyCollection<JsonWebKey> EncryptionKeys { get; set; } = [];

The default value of both properties is an empty collection. That is not a sensible production default, and it is not meant to be. The library ships an escape hatch (the options) and an extension point (the interface), and expects the host to choose which one to use before going live.

The XML documentation on OidcOptionsKeysProvider says this out loud:

It is recommended to implement a dynamic resolution mechanism in production environments to enable seamless certificate replacement without the need for service reloading.

Production failure modes​

Three concrete scenarios motivate the recipes in the rest of this guide. Each produces authentication errors that look random until you know the cause.

Multi-pod key divergence​

A Kubernetes Deployment runs three pods of your auth-service. Each pod's startup code calls something like JsonWebKeyFactory.CreateRsa() to populate OidcOptions.SigningKeys. Each pod independently generates a different key pair. The three pods now have three different jwks_uri responses, three different sets of kid values, and three different private keys signing tokens that are all presented to the same load balancer.

The user impact is intermittent 401 errors that correlate with nothing the user can control. Sticky sessions partially mask the problem; a rolling deploy exposes it. Your tracing shows nothing in application code. The only clue is that failures correlate with the kid claim on the token not matching the verifying pod's published JWKS.

Diagnosis is slow because the usual suspects (token expiry, clock skew, incorrect audience) all check out.

Restart invalidates every token​

A single-pod deployment has keys generated at process start. Every refresh token in circulation is signed by the current process's in-memory private key. A rolling restart, a liveness-probe kill, or a Kubernetes eviction replaces the process. New keys, new kid values, and every refresh token that was valid a minute ago is now unrecognized.

Users experience a mass logout. Mobile apps start retrying refresh, hit the auth endpoint harder than steady state, and some teams accidentally DoS themselves during a routine deploy.

Keys in Git​

The third failure mode is avoided by engineers who know not to commit .pem files, but it keeps appearing in leaked-secrets audits anyway. A developer pastes a JWK into appsettings.Development.json, copies that file into appsettings.Production.json to "unblock staging", and a month later somebody greps the repo history and finds a live production private key in a commit.

The code works. The deploy works. The security review three quarters later finds the key.

Recipe A: static keys from a shared secret​

This is the simplest production-grade setup, appropriate when you have a small number of pods, a stable key set, and no strict rotation SLA.

Keys live in a Kubernetes Secret (or Vault, or AWS Secrets Manager) and are mounted into every pod as an environment variable or file. All pods read the same keys at startup and never change them until a planned rotation.

Generate a JWK once, store it in your secret manager, and load it through configuration:

services.AddOidcServices(options =>
{
    var jwkJson = Environment.GetEnvironmentVariable("ABBLIX_SIGNING_JWK")
        ?? throw new InvalidOperationException(
            "ABBLIX_SIGNING_JWK is required. See docs/signing-key-persistence.");

    var jwk = JsonSerializer.Deserialize<JsonWebKey>(jwkJson)
        ?? throw new InvalidOperationException("ABBLIX_SIGNING_JWK is not valid JSON.");

    options.SigningKeys = new[] { jwk };
});

The corresponding secret (Kubernetes example):

kubectl create secret generic abblix-signing-key \
  --from-literal=ABBLIX_SIGNING_JWK="$(cat signing-key.jwk.json)" \
  --namespace=production

And the Deployment references it:

env:
  - name: ABBLIX_SIGNING_JWK
    valueFrom:
      secretKeyRef:
        name: abblix-signing-key
        key: ABBLIX_SIGNING_JWK

What this buys you:

  • Every pod sees the same keys at startup. No multi-pod divergence.
  • A pod restart reads the same secret and produces the same jwks_uri. Existing tokens remain valid.
  • The key never appears in git, in a built image, or in a runtime memory dump anyone can trivially grab.

What this does not buy you:

  • Automatic rotation. A new key requires a manual secret update and a restart.
  • Graceful overlap during rotation. Unless you deliberately publish both the old and the new key for a window, clients will see JWKS churn and existing tokens may stop validating.

For teams with low-volume auth traffic and rotations measured in months, Recipe A is often the right answer. For high-volume services with compliance-driven rotation policies, read on.

Why not just put the JWK in appsettings.Production.json?

Two reasons. First, application settings files typically end up in your build artifact, Docker image, or deployment manifest, all of which are easier to accidentally leak than a secret manager entry. Second, appsettings.*.json is not designed for secrets and most organizations' scanning tooling does not treat it as such. Keep private keys in a dedicated secret store, always.

Recipe B: Redis-backed provider​

When rotation is frequent, or when you want the ability to swap keys without a deploy, a Redis-backed provider gives you a shared, out-of-process key store that every pod reads from on demand.

The library deliberately does not depend on any particular Redis client, so the host owns the dependency. That keeps Abblix OIDC Server free of transitive weight for teams who use a different backend, and it keeps version conflicts from forcing your hand.

Pseudocode (requires the host to have StackExchange.Redis available and an IConnectionMultiplexer registered):

public sealed class RedisAuthServiceKeysProvider(
    IConnectionMultiplexer redis,
    ILogger<RedisAuthServiceKeysProvider> logger) : IAuthServiceKeysProvider
{
    private const string SigningKeysHash = "abblix:signing-keys:active";
    private const string EncryptionKeysHash = "abblix:encryption-keys:active";

    public IAsyncEnumerable<JsonWebKey> GetSigningKeys(bool includePrivateKeys = false)
        => ReadKeysAsync(SigningKeysHash, includePrivateKeys);

    public IAsyncEnumerable<JsonWebKey> GetEncryptionKeys(bool includePrivateKeys = false)
        => ReadKeysAsync(EncryptionKeysHash, includePrivateKeys);

    private async IAsyncEnumerable<JsonWebKey> ReadKeysAsync(
        string hashKey, bool includePrivateKeys)
    {
        var entries = await redis.GetDatabase().HashGetAllAsync(hashKey);

        foreach (var entry in entries)
        {
            if (entry.Value.IsNullOrEmpty) continue;

            JsonWebKey? jwk;
            try { jwk = JsonSerializer.Deserialize<JsonWebKey>(entry.Value!); }
            catch (JsonException ex)
            {
                logger.LogWarning(ex, "Malformed JWK at {HashKey}/{Field}", hashKey, entry.Name);
                continue;
            }

            if (jwk is null) continue;
            yield return jwk.Sanitize(includePrivateKeys);
        }
    }
}

Host registration: register your provider after AddOidcServices() so it overrides the library default:

services.AddOidcServices(options => { /* leave SigningKeys empty; Redis provider wins */ });
services.AddSingleton<IAuthServiceKeysProvider, RedisAuthServiceKeysProvider>();

What to put in Redis:

  • A hash per key-usage (one for signing, one for encryption).
  • Each field is a kid, each value is the full JWK serialized as JSON.
  • A separate small wrapper job (or an admin UI) adds new keys and retires old ones on schedule.

Performance note: a production-quality Redis provider caches the result of ReadKeysAsync in-process for a short TTL (five to thirty seconds is typical). Reading Redis for every token-sign operation is unnecessary overhead, because keys rotate on human timescales, not per-request. Implement the cache with IMemoryCache or equivalent, and invalidate on a simple Redis pub/sub signal if you need sub-minute propagation.

The library carries no Redis client dependency, so a Redis provider is host-owned code that pulls in StackExchange.Redis itself. The runnable sample in the repository is the database-backed recipe below, not this one; the Redis shape mirrors it, swapping the table query for a hash read.

Recipe C: database-backed provider​

The library does not prescribe an ORM, and neither does this recipe. If you already have EF Core, use EF Core. If you prefer Dapper, use Dapper. If you're writing raw ADO.NET against a legacy stored-procedure layer, that works too. What matters is the table shape and the query.

Suggested table:

CREATE TABLE abblix_signing_keys (
    kid          TEXT PRIMARY KEY,
    key_usage    TEXT NOT NULL,          -- 'sig' or 'enc'
    algorithm    TEXT NOT NULL,          -- e.g. 'RS256', 'ES384'
    jwk_json     TEXT NOT NULL,          -- full JWK, including private material
    not_before   TIMESTAMPTZ,            -- NULL means effective immediately
    not_after    TIMESTAMPTZ,            -- NULL means indefinite
    is_active    BOOLEAN NOT NULL DEFAULT TRUE,
    created_at   TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX abblix_signing_keys_active
    ON abblix_signing_keys (key_usage, not_after)
    WHERE is_active = TRUE;

The effective query for GetSigningKeys:

SELECT jwk_json
  FROM abblix_signing_keys
 WHERE key_usage = 'sig'
   AND is_active = TRUE
   AND (not_before IS NULL OR not_before <= NOW())
   AND (not_after IS NULL OR not_after > NOW())
 ORDER BY created_at DESC;

Wrap that in whichever data-access technology you already use. Cache the result in-process with the same short TTL you'd use for the Redis recipe. Implement IAuthServiceKeysProvider on top of the query-plus-cache, register it in DI, done.

Private keys at rest deserve their own conversation, and the honest answer is that an application database is not their real home. In a high-assurance system signing keys live in an HSM or a managed KMS (AWS KMS, Azure Key Vault, GCP KMS, HashiCorp Vault), where the private half ideally never leaves the boundary in plaintext. This library imposes a ceiling here worth stating plainly: IAuthServiceKeysProvider hands a JsonWebKey with private material to an in-process signer, so vault-resident signing (the key never leaving the HSM) is not reachable through that seam. What the keys provider can do is fetch the private material from a KMS per use, which brings the plaintext private half into process memory to sign. The database-backed recipe here is a documented, deliberate tradeoff for teams that accept DB key custody. If you take it, encrypt the key at rest at a minimum, either at the database layer (pgcrypto, Always Encrypted, TDE with additional protections) or at the application layer with ASP.NET Data Protection (see Encrypting the stored key at rest). Whichever layer you pick, the encryption relocates key custody to that layer's own master key rather than removing it, as the next section traces for Data Protection. Prefer keeping only key IDs in the database while fetching private material from a KMS per use. Left unencrypted and unrotated, a database holding raw RSA private keys is worse than a Kubernetes Secret.

A runnable version of this recipe, backed by EF Core and SQLite, lives in the AspNetIdentitySample project of the Getting Started repository. It generates and persists a signing key on first run, encrypts it at rest with ASP.NET Data Protection (see the next section), and reloads the same key on every restart, so tokens issued before a restart keep validating. The residual custody concerns are marked with #warning directives in the sample, so a developer copying it toward production trips over them at build time.

Encrypting the stored key at rest​

Storing a private key as plaintext in any database is the weakest form of this recipe. One step up, without reaching for a KMS, is to encrypt the JWK before it is written and decrypt it on read. ASP.NET Core Data Protection is the built-in tool for that: create a protector with a fixed purpose string, Protect the serialized JWK on the way in, Unprotect it on the way out.

// on write (seeding or rotation)
var protector = dataProtectionProvider.CreateProtector("MyApp.SigningKeys.v1");
record.JwkJson = protector.Protect(JsonSerializer.Serialize<JsonWebKey>(key));

// on read, inside your IAuthServiceKeysProvider
var json = protector.Unprotect(record.JwkJson);
var key = JsonSerializer.Deserialize<JsonWebKey>(json);

The AspNetIdentitySample sample does this, so a raw dump of the SQLite file yields ciphertext, not a signing key. Be precise, though, about what this buys and what it does not.

There is an irony worth naming. The scope note at the top of this article warned that Data Protection keys and signing keys are two separate stores with the same failure modes. Encrypting your signing key with Data Protection does not merge them, it stacks them: your signing key is now exactly as durable and as shared as the Data Protection key ring that encrypts it. The custody problem moves one layer down, onto a key ring that must be persisted and shared exactly as this article demands of the signing key, and protected on top of that: a smaller, better-understood surface to defend, but not one you can ignore.

  • Persisted. With no explicit configuration the key ring lands in a per-user profile folder on a developer machine, which is exactly why a local restart still decrypts the signing key. Inside a container with no user profile that fallback is not available, and Data Protection degrades to ephemeral in-memory keys lost on the next restart. Persist and back up the ring, or a lost ring leaves every stored signing key unrecoverable. Call PersistKeysToFileSystem(...) a durable location (or a shared store) and SetApplicationName(...) so every instance derives the same keys, rather than relying on the developer-machine default.
  • Shared. Two instances with two different key rings cannot read each other's encrypted signing keys, so an unshared ring has the same multi-instance problem this article opened with, one layer down. Persist the ring to a store every instance reads, using the backend-specific extension: PersistKeysToFileSystem for a shared volume (in-box), PersistKeysToStackExchangeRedis, PersistKeysToAzureBlobStorage, or PersistKeysToDbContext, each in its own package. There is no generic PersistKeysTo.
  • Protected. A persisted ring is itself plaintext at rest unless you protect it with ProtectKeysWithCertificate(...) or a cloud KMS (ProtectKeysWithAzureKeyVault(...)). Without that, the Data Protection keys sit next to the database you were protecting, and the encryption defends only against a leaked database file, not against read access to the host filesystem. The certificate option only defers custody again (the cert's own private key must be loadable to open the ring); a cloud KMS is what actually terminates the chain.

Concrete recommendations, by deployment shape:

  • Single instance, development or low assurance. Data Protection with PersistKeysTo a durable path is enough. Know that without ProtectKeysWith, you are defended against a leaked database backup but not against an attacker who can read the host filesystem.
  • Multiple instances. The key ring must be both shared (a store every instance reads: a mounted volume, a blob container, Redis, a database table) and protected (a certificate or KMS). Configure both: miss the shared store and validation fails across pods, miss the protector and the encryption adds nothing over the database's own protection.
  • High assurance or compliance-bound. Do not keep private key material in your database at all, encrypted or not. Put the key in an HSM or a managed KMS and store only key IDs in the database. Mind the library ceiling from Recipe C: IAuthServiceKeysProvider can fetch and unwrap the private key from the vault per use (it then signs in process), but a key that never leaves the HSM would require replacing the token-signing pipeline, not just the keys provider. This is the real home for signing keys; Data Protection encryption is a pragmatic middle ground, not a substitute.

Key rotation discipline​

Whichever storage you pick, rotation needs the same four moves.

  • Generate a new key ahead of time. Give it a unique kid, a not_before set to about ten minutes from now, and insert it as is_active = TRUE. Your JWKS endpoint will start publishing the new key immediately, but the signing pipeline will keep using the previous one until the not_before window opens.
  • Start signing with the new key. At the configured not_before, your signing code should prefer the newest active key. Relying parties who fetched JWKS in the last ten minutes already have the new kid in their cache. Freshly issued tokens validate.
  • Retain the old key for the token lifetime plus a buffer. Set the old key's not_after to now plus your longest refresh-token lifetime plus a safety margin of at least an hour. During this overlap window, both keys are published in JWKS, old tokens validate against the old key, and new tokens validate against the new key. Nobody is locked out.
  • Retire the old key. Once no extant token could still be signed by it, set is_active = FALSE and drop it from JWKS. Keep the row around for an audit trail, but stop publishing the public half.

A common beginner mistake is to rotate the key and immediately remove the old one, which invalidates every outstanding refresh token. Another is to skip not_before and start signing with a key the JWKS endpoint hasn't published yet, which causes a brief window of 401s for every client that cached JWKS too recently. The overlap window is not a nicety, it is the whole point.

How long should the overlap window be?

At minimum, the longest refresh-token lifetime you issue. If your refresh tokens live for thirty days, the old key must remain verifiable for thirty days after you stop signing with it. For most deployments, forty-five days is a reasonable default. For short-lived tokens in higher-risk environments, shorter overlap windows reduce the blast radius if a key is compromised.

A multi-pod consistency test worth writing​

An integration test you can run in CI, using ephemeral keys but a shared store:

  • Start two auth-service instances pointed at the same Redis or database.
  • Issue a token from instance A.
  • Present that token to instance B's token introspection endpoint.
  • Assert the introspection result is active: true.

Repeat with a forced key rotation between issue and validate. Assert the old-signed token still validates during the overlap window. Assert it fails after the window elapses.

This test catches almost every real configuration mistake in this area before it reaches production. It runs in CI alongside normal integration tests.

Registering your provider​

To override the library's default IAuthServiceKeysProvider, register your implementation after AddOidcServices():

services.AddOidcServices(options => { /* ... */ });
services.AddSingleton<IAuthServiceKeysProvider, MyProvider>();

Order matters. Microsoft's default DI container returns the most recently registered implementation when resolving a single service, so your AddSingleton must come after AddOidcServices() to win. Registering in the opposite order silently leaves the library default in place.

Summary​

Signing keys need three properties to work in production: they must be shared across all instances, they must survive restarts, and they must not live in source control. Abblix OIDC Server provides the IAuthServiceKeysProvider extension point exactly so you can meet all three without fighting the library.

Pick the simplest recipe that meets your rotation needs. Static keys from a secret store handle small deployments with planned rotations. A Redis-backed provider gives you dynamic rotation without a deploy. A database-backed provider fits naturally in teams that already keep durable state in Postgres or SQL Server.

Whatever you pick, back the configuration with the multi-pod consistency test. It is the practical difference between a setup you can trust under rolling-deploy conditions and one whose failures only surface under production load.