When the token store is down, the honest answer is 503

The claim

A bearer token proves two different things, and only one of them is in the token. The signature, issuer, audience and lifetime say the token is genuine. Whether the tenant and the person behind it are still active is a fact the resource server has to look up. When that lookup cannot be made, the right answer is not "admitted" and it is not 401. It is 503: the service could not perform the check. The token may be perfectly good.

Context

Hexavera's cloud API, its site runtime and its assistant service shared one authentication setup when this was decided; the cloud API has since moved to the identity client's resource-server registration, which uses the same fault classification and the same 503 answer. After the JWT bearer handler validates a token, an OnTokenValidated handler reads the tenant and the user from the database, confirms both are active, and rewrites the token's identifiers to the host's own. That lookup is the only part of authentication that depends on infrastructure, and it is the part that fails when the database is slow, restarting or unreachable.

The previous rule for a failed lookup was written as a convenience: a fault that looked transient admitted the token with its claims as they stood. The test for "transient" matched the data provider's exception type name and a few words in the message, which is to say it matched every database fault there is, and any fault at all once the caller had disconnected. The rule had to go: it was replaced by a decision, not patched.

The decision

1. Every fault refuses the token

There is no fault kind that admits a token. A disabled person or a switched-off tenant may not pass because a dependency is unwell. The handler catches every exception from the lookup except an out-of-memory condition, drops the principal whole (half of it may already have been rewritten), and marks the authentication as failed with an exception type of its own that says could not check.

Nothing is thrown out of the event. An exception that escapes OnTokenValidated fails every request that carries a token, anonymous routes and health probes included, and one of the hosts had no handler that would turn it into anything but a bare 500.

OnTokenValidated = async context =>
{
    try
    {
        await CheckTenantAndUserAsync(context);
    }
    catch (Exception exception) when (exception is not OutOfMemoryException)
    {
        if (context.HttpContext.RequestAborted.IsCancellationRequested)
        {
            // Nobody is waiting for an answer. Routine while a Blazor circuit starts.
            context.NoResult();
            return;
        }

        context.Fail(new TokenCheckUnavailableException(Classify(exception), exception));
    }
}

2. The fault is classified by what the provider says

The kind of fault no longer decides whether the token is admitted; it never is. It decides the log level and whether the caller is told to try again. The classification reads the whole chain of inner exceptions, because the provider's default execution strategy wraps a transient exception in an InvalidOperationException, and asks the provider rather than guessing from text: DbException.IsTransient, which Npgsql answers for a refused connection, an exhausted pool, a command that timed out and the server's own "try again" codes, plus TimeoutException.

static bool IsTransient(Exception? exception) =>
    exception is not null
    && (exception is TimeoutException or DbException { IsTransient: true }
        || IsTransient(exception.InnerException));

A transient fault is logged as a warning. Anything else (a missing table, a missing privilege, a broken query) is logged as an error, because it will not mend itself and every tokened request is refused until someone acts.

3. The answer is 503, with a Retry-After drawn per response

The bearer handler's default reply to a failed authentication is 401. For a check that was not made, 401 says the wrong thing to every client: the browser app signs its user out, the console signs its operator out, and a service client stops trusting a token that is fine. RFC 9110 defines 503 for a server that "is currently unable to handle the request due to a temporary overload or scheduled maintenance, which will likely be alleviated after some delay", and lets it send a Retry-After header "to suggest an appropriate amount of time for the client to wait before retrying the request"; with a 503, Retry-After "indicates how long the service is expected to be unavailable to the client". For a transient fault that is exactly the message. A broken check is not temporary, so answering 503 there is a design choice rather than the RFC's: it still says "could not check", not "bad token".

OnChallenge = context =>
{
    if (context.AuthenticateFailure is not TokenCheckUnavailableException unavailable)
    {
        return Task.CompletedTask; // Not ours: the handler's 401 stands.
    }

    var response = context.Response;
    response.StatusCode = StatusCodes.Status503ServiceUnavailable;
    if (unavailable.Fault == Fault.Transient)
    {
        // Drawn per answer, between 3 and 8 seconds: a fixed value would make every caller
        // of an outage retry in the same second.
        response.Headers.RetryAfter = Random.Shared.Next(3, 9).ToString(CultureInfo.InvariantCulture);
    }

    context.HandleResponse();
    return Task.CompletedTask;
};

Two details carry weight. The header is sent only for a transient fault; a broken check should not invite retries. And the value differs from one answer to the next. The hosts' HTTP clients use the standard resilience handler from Microsoft.Extensions.Http.Resilience, whose retry strategy honours Retry-After by default (ShouldRetryAfterHeader is true), so the server's draw replaces the clients' own jitter. A fixed value would synchronise every caller of an outage into one burst at the moment the database came back.

4. Anonymous routes ask before they say "not signed in"

A route marked anonymous is served to a caller whose token could not be checked exactly as it would be served to a caller with no token. That is right for a public page and wrong for a route whose answer is the sign-in state: an availability endpoint that tells the client "you are signed out" would, during an outage, tell every signed-in client to sign in again. Such a route asks first whether the token could not be checked, and challenges (which produces the 503) instead of answering.

5. A host that cannot check refuses too

A host whose data model has no tenant table cannot make the check at all. It used to be let off with every token admitted unread. Now it refuses like any other check that was not made, so a misconfigured host fails at its first tokened request rather than admitting everyone quietly.

How it is tested

Stopping a database container to test this is slow and imprecise. The suite uses a command interceptor on the data contexts, switched on per request by a test header, with named fault kinds: transient, starting up, broken, hang. A test can say "fail the tenant read with a transient fault" and assert the 503 and the Retry-After range, and "fail it with a broken fault" and assert the 503 with no Retry-After. At the unit level the event is driven without a database: resolve the bearer options, build a TokenValidatedContext, call Events.TokenValidated and read the result and the log event, a warning for a transient fault and an error for a broken one.

What it cost

• During a database outage every tokened request answers 503. That is the design, and it sets a requirement on the clients: a browser app must treat a 503 from its own API as "try again", never as "signed out", and a service client must wait as long as the header says.
• Two follow-ups, recorded as review findings when the rule was written down: a hub client that does not retry a failed first negotiation, and MCP servers whose bearer registration checked a token's signature, issuer, audience and lifetime but never its tenant or its person. The cloud's three servers have the check now; one server still lacks it. Writing the rule down made both visible.

What a team can take from this

Separate "the token is bad" from "I could not check the token" in your authentication events, and give them different status codes. Classify faults by what the data provider says, not by substrings in messages. Draw Retry-After per response so your own clients do not stampede. And test the outage with fault injection inside the host rather than by pulling the plug on a container; you will test more cases, and you will test them every time.