Retries Where You Want Them: HTTP Resilience for Abblix OIDC Server
At Abblix, we develop Oidc.Server, our certified OpenID Connect and OAuth 2.0 library for .NET. Most of what it does happens inside your process: minting tokens, validating requests, checking signatures.
But not all of it. Your provider fetches a client's key set to verify a signed request object. It POSTs a logout token to every session a user is signing out of. It notifies a CIBA client that its authentication request has completed. If your keys live in Vault or Azure Key Vault, every signature is a network call too.
Each of those calls crosses a network you do not control, to a server whose availability is somebody else's problem. A key set fetch that fails takes a valid request down with it. A notification that fails leaves a client waiting out its timeout. So sooner or later you want retries, and a circuit breaker to stop hammering a receiver that is already down.
None of that is ours to invent. .NET has it: Microsoft.Extensions.Http.Resilience, built on Polly. What we owe you is a seam it can attach to, on every client we send requests through, without asking you to subclass anything or reach in with reflection.
TL;DR
Every outbound call this library makes goes through IHttpClientFactory. To make all of them resilient, you write one line and name nothing:
services.ConfigureHttpClientDefaults(http => http.AddStandardResilienceHandler());
To treat one of them differently, you name it instead: every client publishes its name as a constant, so nothing is copied out of our source or guessed from a type name. Host-wide and per-client are alternatives, not layers. That is the one thing to get right before you start, and the next sections say why.
Either way you keep everything else you already have. Adding a resilience pipeline is additive: on the three clients that call a URI an OAuth client supplied, the address validation stays exactly where it was, underneath your retries, and every retried attempt is validated afresh. Retries and a circuit breaker cost you nothing in security, which is why this is the path to take.
The one-line version
One package carries everything below:
dotnet add package Microsoft.Extensions.Http.Resilience
The numbers quoted below are the defaults of version 9.10. They are stable across patch releases and have moved across majors, so check them against the version you resolve.
If your answer to "which of these calls should be resilient?" is "all of them", you are done in a line. ConfigureHttpClientDefaults is part of Microsoft.Extensions.Http, and it applies to clients registered before or after the call:
builder.Services.ConfigureHttpClientDefaults(http => http.AddStandardResilienceHandler());
builder.Services.AddOidcServices(options => { /* ... */ });
Order does not matter here. Put the defaults call first or last; every client this library registers picks it up either way. That is worth knowing because our registrations are one AddOidcServices call deep, and you should not have to reason about what happens inside it.
The standard handler is a pipeline of five strategies: a total request timeout, a retry, a circuit breaker and an attempt timeout, with rate limiting in front. Retries cover the outcomes worth repeating: connection failures, 5xx, 408 and 429, and a per-attempt timeout, which is why that timeout sits inside the retry. A 400 is not retried, correctly, since repeating the same request will earn the same answer.
One thing overrides you. A response carrying Retry-After replaces the delay you configured, because the standard retry honours the header by default, so a receiver asking for five minutes spends your whole budget in one wait. Set Retry.ShouldRetryAfterHeader = false if you would rather keep your own schedule.
One consequence of the convenience is worth knowing before you rely on it. The defaults register a single pipeline under one shared name, so every client in your host shares one circuit breaker instance: ours, and your own application's clients too. A relying party that stops answering can therefore open the breaker that stands in front of your key vault.
If that coupling is unacceptable, you have two ways out. Raise MinimumThroughput and FailureRatio so the breaker engages only on a genuinely global outage, or configure clients individually, which gives each its own pipeline and its own breaker.
Turning the knobs
The defaults that ship with that handler are these: thirty seconds for the whole operation, ten for a single attempt, three retries starting at a two-second delay with exponential backoff and jitter, and a breaker that opens when a tenth of the calls fail over a thirty-second window, then stays open for five seconds. Every one of them is a property you can set, and the shape is the same whichever client you are configuring:
services.ConfigureHttpClientDefaults(http => http.AddStandardResilienceHandler(options =>
{
// Timeouts: the ceiling on everything, and on one attempt.
options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(20);
options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(5);
// Retry.
options.Retry.MaxRetryAttempts = 4;
options.Retry.Delay = TimeSpan.FromMilliseconds(500);
options.Retry.BackoffType = DelayBackoffType.Exponential;
options.Retry.UseJitter = true;
// Circuit breaker: open when half the calls fail, and stay open long enough to matter.
options.CircuitBreaker.FailureRatio = 0.5;
options.CircuitBreaker.MinimumThroughput = 10;
options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(30);
options.CircuitBreaker.BreakDuration = TimeSpan.FromSeconds(15);
}));
Two relationships between these are checked for you. The attempt timeout must be shorter than the total, and the breaker's sampling duration must be at least twice the attempt timeout, or resolving the client throws an OptionsValidationException naming both values.
A third is not checked: whether the total leaves room for the retries you asked for. Four retries at half a second exponential spend roughly eight seconds waiting before the last attempt begins, and if that plus the attempts exceeds the total, you get a cancellation where you expected a retry. Do that arithmetic yourself.
The breaker's MinimumThroughput is the one that surprises people. It defaults to a hundred calls within the sampling window, a sensible floor for a busy service and one a provider fetching a key set a few times a minute will never reach, which means the breaker never opens. If you want it to engage on a low-traffic client, lower it.
DelayBackoffType above comes from Polly, so that block needs using Polly; alongside the resilience one.
If you would rather assemble the pipeline yourself than adjust a standard one, you can. AddResilienceHandler takes the strategies you name and nothing else:
using Microsoft.Extensions.Http.Resilience;
using Polly.Retry;
using Polly.Timeout;
services.AddHttpClient(VaultTransport.HttpClientName)
.AddResilienceHandler("vault", pipeline => pipeline
.AddRetry(new HttpRetryStrategyOptions { MaxRetryAttempts = 5 })
.AddTimeout(TimeSpan.FromSeconds(10)));
The strategy methods live in their own Polly namespaces, which is easy to miss: without those two using lines the compiler reports that ResiliencePipelineBuilder<HttpResponseMessage> has no AddRetry.
Naming one client
The one-liner is a floor, not a ceiling. Different calls deserve different treatment, and once you want that, you name the client you are configuring.
Say a client's key set fetch should fail fast, because a user is waiting on the authorization request it blocks, while your Vault calls deserve patience because that server occasionally stalls. Name each and give it what it needs:
// A user is waiting on this one. Fail fast.
services
.AddHttpClient(SecureFetchTransport.HttpClientName)
.AddStandardResilienceHandler(options =>
{
options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(4);
options.AttemptTimeout.Timeout = TimeSpan.FromSeconds(2);
options.Retry.MaxRetryAttempts = 1;
options.CircuitBreaker.SamplingDuration = TimeSpan.FromSeconds(10);
});
// Nothing is waiting on this one. Be patient, and give the total room to hold the backoff.
services
.AddHttpClient(VaultTransport.HttpClientName)
.AddStandardResilienceHandler(options =>
{
options.Retry.MaxRetryAttempts = 5;
options.Retry.Delay = TimeSpan.FromMilliseconds(500); // 0.5 + 1 + 2 + 4 + 8 = 15.5s of waiting
options.TotalRequestTimeout.Timeout = TimeSpan.FromSeconds(90);
});
The second block is the arithmetic from the previous section done in earnest. Leave the delay at its two-second default, and five retries spend 62 seconds waiting before the last attempt begins. A 30-second total cancels that long before it finishes, so you would get four attempts and a timeout where you asked for patience.
Configuration is keyed by client, so what you set on one never reaches another.
Do not do both
Adding a pipeline host-wide and then adding a second one to a single client does not override the first. It stacks. The client ends up wrapped in two pipelines and the attempts multiply: a host-wide handler allowing three attempts plus a per-client handler allowing two gives that client six attempts at the origin.
Reaching back to reconfigure the pipeline the defaults installed does not work either. The defaults register it under one shared options name, with no client name in it, so there is nothing per-client to address.
Pick one style and keep to it. Either one host-wide call that everything shares, or a call per client:
services.AddHttpClient(BackChannelLogoutTransport.HttpClientName).AddStandardResilienceHandler();
services.AddHttpClient(BackChannelNotificationTransport.HttpClientName).AddStandardResilienceHandler();
services
.AddHttpClient(SecureFetchTransport.HttpClientName)
.AddStandardResilienceHandler(options => options.Retry.MaxRetryAttempts = 1);
One exception fits inside the host-wide style, because it adds no second pipeline: a client's own timeout. HttpClient.Timeout caps the whole operation, retries included, so capping it makes one client give up early while the rest keep the shared policy:
services.ConfigureHttpClientDefaults(http => http.AddStandardResilienceHandler());
// A user is waiting on this one, so it gives up while the others are still retrying.
services.AddHttpClient(SecureFetchTransport.HttpClientName, client => client.Timeout = TimeSpan.FromSeconds(3));
What you get is a cancellation, not a graceful failure, and the pipeline's own timeouts still apply underneath. Reach for it when the requirement is genuinely "not a second longer than this", and configure clients individually when the requirement is about retry behaviour.
What to name
Every client carries its name as HttpClientName on a small transport type in its own package. One shape, eight clients, nothing to guess and no string to copy out of our source:
| What it calls | Package | Name it with |
|---|---|---|
A client's JWKS, request_uri, software statement | Abblix.Oidc.Server | SecureFetchTransport.HttpClientName |
| Back-channel logout tokens | Abblix.Oidc.Server | BackChannelLogoutTransport.HttpClientName |
| CIBA ping and push notifications | Abblix.Oidc.Server | BackChannelNotificationTransport.HttpClientName |
| Vault / OpenBao, custodian and key ring | Abblix.Jwt.Vault | VaultTransport.HttpClientName |
| Azure Key Vault | Abblix.Jwt.Azure | AzureKeyVaultTransport.HttpClientName |
| The key ring's blob container | Abblix.Jwt.Azure | AzureKeyRingTransport.HttpClientName |
| An issuer's key set, for event verification | Abblix.SecurityEvents | JwksTransport.HttpClientName |
| Shared Signals push delivery | Abblix.SharedSignals | PushDeliveryTransport.HttpClientName |
Some of these are typed clients underneath and some are named ones, which changes how the factory keys them internally. The constant hides that difference, so you configure all eight the same way.
You will not find a new HttpClient() anywhere in these packages, and that is deliberate. A client constructed inside a library is a client you cannot configure: nothing of yours runs on it, and no policy of yours applies. Every one of these comes from your factory instead.
A retry is a second delivery
Most of these clients POST. When an attempt times out you do not learn whether the receiver ignored the request or processed it and lost the response on the way back, so a retry is a second delivery of the same message, and every receiver in your ecosystem has to tolerate one.
For a logout token that is cheap, because its effect is idempotent. Note though that Back-Channel Logout leaves jti replay detection optional for the relying party, so a receiver that does implement it will reject your retry as a replay: a delivery that timed out after arriving cannot be rescued by retrying it. For CIBA push it is not cheap at all, because the retry re-delivers tokens, and a receiver that mints a session per delivery mints two.
Retry hard where the receiver deduplicates and redelivery is harmless. Keep the attempts low where it is not.
When the breaker is open
An open breaker does not slow calls down, it stops them: the pipeline throws BrokenCircuitException without touching the network. What that means depends on the client.
For a notification or a logout token it means a delivery that never left your process. For a key set fetch it means the authorization request that needed the key fails, and it fails as a protocol error about the client's metadata, because from the fetcher's point of view the fetch did not succeed. For a custodian client it means token issuance stops while the breaker is open.
The tell in an incident is asymmetry: your logs show failures and the receiver's logs show no requests at all. That is the breaker, not the receiver.
What a retry does not reach
Two things sit outside the pipeline you attach, and both are worth knowing before you rely on it.
In the Azure package, the credential authenticates over a transport of its own. What you attach to AzureKeyVaultTransport.HttpClientName and AzureKeyRingTransport.HttpClientName covers the vault and blob calls, not the token requests the credential makes to acquire its access token. If you need those resilient too, that is a property of the Azure SDK's own client options, not of ours.
And a pipeline is bounded by HttpClient.Timeout, which is a ceiling over the whole operation including every retry. The CIBA notification client sets that timeout from OidcOptions, so if you configure retries there, leave the client's timeout enough room to accommodate them. A retry strategy that would take twenty seconds under a five-second client timeout does not retry three times: it is cancelled once.
Where our own retry policy sits, and why it mostly does not exist
You may notice that our senders make one attempt and report what happened. That is deliberate, and it is what makes them safe to wrap.
CIBA notifications are best-effort by design. A failed ping costs the client its own timeout, not a lost protocol state. Building a retry into the delivery service would mean holding attempt counters and timers inside a type you cannot configure, in exchange for a policy that would be wrong for somebody. One honest attempt, and your pipeline decides the rest.
Shared Signals push delivery is the same reasoning at stream level. The sender drains a stream's queue in order, stops at the first transient failure so ordering survives into the next pass, and drops a security event token the receiver judged terminally invalid. Retry lives in the schedule that calls it and in the pipeline underneath it, which is why the sender itself holds no state.
Where the address validation sits
Three of these clients address a URI that came from an OAuth client, not from your configuration: its registered CIBA notification endpoint, its back-channel logout URI, and the request_uri, JWKS and software statement addresses a fetch is pointed at.
A client that registers a link-local metadata address and gets your provider to call it has turned a callback into a server-side request forgery. So those three validate the address on every request, re-resolving the name each time instead of trusting the resolution that passed a moment ago.
That validation is the client's primary handler, the innermost one in the chain, and everything this article asks you to add chains outside it. AddStandardResilienceHandler, AddResilienceHandler, a logging handler, a host-wide default: each of them adds a link above the validation, so the guarantee survives the configuration and every retried attempt is validated afresh, not once per request. That is the whole reason to prefer these calls: you get retries and a circuit breaker without trading anything away.
One call is different in kind. ConfigurePrimaryHttpMessageHandler does not add a link, it replaces the bottom of the chain, and on these three clients the bottom is the validation.
That call is also the standard way to set a client certificate or your own connection pooling, so it is worth saying plainly: here it changes a security control, not a transport setting, and it deserves the review any such change would get. If what you need is a proxy or a certificate on one of these three, raise it with us before reaching for that call.
You will not have to notice it on your own. When one of those three clients ends up with a primary handler that is not the validation, the library says so at Warning, naming the client and what replaced it. A deployment that made the swap deliberately silences that one message by its own log category, and every other log this library writes keeps working:
{ "Logging": { "LogLevel": { "Abblix.Oidc.Server.Features.SecureHttpFetch.SsrfGuardWatch": "None" } } }
Start here
Add the one-liner. It costs a line and covers every call this library makes, which is the right floor for a deployment that has none today:
services.ConfigureHttpClientDefaults(http => http.AddStandardResilienceHandler());
Then spend five minutes on the two places where that floor is wrong for you. The secure-fetch client sits on the authorization path with a user waiting on it, and thirty seconds is generous there. The shared circuit breaker couples clients that have nothing to do with each other, which some deployments will not accept.
Fixing either means switching styles, not layering: drop the defaults call and configure clients one by one, since a per-client pipeline added on top of the host-wide one stacks instead of replacing it. The exception is a client's own HttpClient.Timeout, which caps one client without adding a second pipeline.
The table above is the whole vocabulary you need, and those names are the only API of ours this article asks you to touch.