The quality gate that could not see async code

The claim

A quality gate is a measurement. Before you let it fail a build, you owe it the same scepticism you owe any instrument: what does it measure, what does it not, and how would you know. Our CRAP gate scored all but two of Hexavera's 1,946 async methods at zero coverage, whatever their tests, because complexity came from the source and coverage came from a report that filed the method's body under a compiler-generated class. It surfaced on 2026-09-26, when the gate was measured against methods known to be covered. This note is what it meant, what was not fixed and why, and the two rules that hold instead.

The metric, and the gate

CRAP, Change Risk Analysis and Predictions, is the 2007 metric of Alberto Savoia and Bob Evans: CRAP(m) = comp(m)^2 * (1 - cov(m)/100)^3 + comp(m), where comp is a method's cyclomatic complexity and cov its test coverage. Complexity is punished quadratically, coverage forgives it cubically. The crap4j tool that followed chose, in Savoia's words, to "INITIALLY use a CRAP score of 30 as the threshold for crappiness", and 30 is what our gate uses too.

Two things fall out of the arithmetic that matter for this note. At full coverage a method scores its complexity, so no fully covered method is ever above 30 unless it is a monster. At zero coverage it scores complexity squared plus complexity, so --max-crap 30 is a complexity ceiling of 5: a method of complexity 5 scores 30, one of complexity 6 scores 42, one of complexity 7 scores 56.

The gate itself is a .NET tool (crap4dotnet 0.1.1) driven by two scripts. The runner collects Cobertura coverage and writes one report per project; the verifier fails the build on any method over the ceiling, with two allowlists: hand-written exemptions, each with a reason and an approver (a named person, or a recorded measurement limitation), and a baseline taken on 2026-08-21, when 669 methods were already above 30, that pins each of them not already exempted at its own value, marked "unreviewed-baseline-2026-08-21" so that nobody mistakes the pin for approval. Where a method is in both, the lower value wins, so adding a file can only tighten the gate. Nothing may get worse, and nothing new may join. That is a ratchet, and a ratchet is the only honest gate you can put on a codebase you inherited.

The gate had already taught one lesson about measurements. Its first version enumerated projects from a hard-coded list of 44 paths, 31 of which no longer existed after a reorganisation. It printed "[skip]" for each and reported success over the 13 that remained: four test projects, six projects of an abandoned tree, and three live projects holding about 800 of the 7,760 methods. A gate that measures a tenth of a codebase and reports green is worse than no gate, because it produces a signal people trust. The runner now enumerates projects from disk, declares what it excludes with a reason each, and fails when an exclusion matches nothing or the walk finds fewer projects than a recorded floor.

The blind spot

The C# compiler lowers an async method into a state machine: a nested type named <MethodName>d__N whose MoveNext method holds the original body, the awaits, and every branch. The original method shrinks to a few instructions that create and start the machine. Coverlet instruments the compiled assembly and reports coverage per compiled method, so the body's lines appear in the Cobertura report under the state machine class and its MoveNext. The compiler-generated classes show up beside the real ones in the report; that has been known since a Coverlet issue in 2018, where a contributor explained that the classes "are generated to build the state engine that implements async/await" and that ReportGenerator folds them back when it renders a report.

The gate tool does not render; it matches. It computes complexity from the source with a Roslyn walker and matches source methods to the report's entries, and, as far as its output shows, it finds no entry under the source method's name for the body. CreateAdministratorAsync has a complexity; <CreateAdministratorAsync>d__N.MoveNext has the coverage. They never meet, and the method is scored with no coverage.

The reports the runner had written on 2026-09-26 show it: of 1,946 methods declared async, 2 were scored with any coverage at all, both in generic classes. The console's CreateAdministratorAsync, which its unit tests run through every line (Coverlet line rate 1.0), was scored at no coverage and CRAP 56. The kernel's user management service, exercised by two suites, was scored at no coverage on every async method. Methods that return a Task without the async keyword were measured normally; they have no state machine. Iterator methods have one too and would presumably behave the same, though that was not measured.

For the gate this means: for an async method, 30 is a complexity ceiling of 5, and tests do not move it. The baseline was measured the same way, so the ratchet is consistent. It is simply blind to tests of async code.

What was not fixed, and why

The obvious fix is to fold each state machine's coverage back into its method before analysis. It was not done, for two reasons, and both were written next to the threshold.

First, the runner deliberately never rewrites a report, because a rewrite that inflated coverage would lower CRAP silently, and a gate that can be made to pass by a bug in its own preprocessing is not a gate. Second, folding would move hundreds of baselined scores at once, which means re-measuring the baseline in the same change, with the whole suite green on the new measurement. That is its own decision, with its own review, and it should not ride in on a bug fix.

The two rules instead

Keep an async method to the awaiting and the plumbing, and put its branching in synchronous methods, where the tests are counted. The console's setup service reads a status code in OutcomeOf and a problem-details body in ReasonsIn; the site runtime's setup-code store decides whether to adopt a code from disk in Adopt. Its previous shape, one async method of complexity 7, scored 56.

public async Task<SetupOutcome> CreateAdministratorAsync(SetupRequest request, CancellationToken cancellationToken)
{
    using var response = await client.PostAsJsonAsync("/setup/administrator", request, cancellationToken);
    var body = await response.Content.ReadAsStringAsync(cancellationToken);

    return OutcomeOf(response.StatusCode, body);     // every branch lives here, where coverage is counted
}

internal static SetupOutcome OutcomeOf(HttpStatusCode status, string body) => status switch
{
    HttpStatusCode.Created => SetupOutcome.Created,
    HttpStatusCode.Conflict => SetupOutcome.AlreadySetUp,
    HttpStatusCode.BadRequest => SetupOutcome.Rejected(ReasonsIn(body)),
    _ => SetupOutcome.Failed(status),
};

Or split until each async method is at complexity 5 or below. That is how the authorization handler passed: two methods of complexity 5 and 3, scored 30 and 12 with their tests unseen.

Both rules produce code worth having anyway. An async method that only awaits and delegates is easy to read, and a synchronous classifier with a switch is easy to test without a host. The gate's blind spot pushed in the direction good design already pointed, which is the only reason living with it was accepted.

What it cost

• A known blind spot, written down beside the threshold, that every contributor has to learn.
• Some methods split for the gate's sake that a human reviewer would have left whole.
• One streaming method, async IAsyncEnumerable with yield return interleaved with its logic, scored 31,862 when it was first measured, at complexity 178, and could not be split by hand. It was rebuilt as channel-based stages with plain Task methods and came down to 6; its hand-approved exemption outlived the rebuild until a check for dead allowlist entries found it on the day the baseline was taken, and it was removed.

What a team can take from this

Point your gate at a method you know is covered and see what it says before you trust it with a build. Read the report format your gate consumes, because the compiler's output is not your source and the two disagree in exactly the places you care about. When you find a blind spot, decide explicitly whether to fix the instrument or to live with it, write the decision next to the threshold, and never lower a floor or widen an exclusion to make a failure go away. A gate that reports green over nothing is the most expensive thing a team can own.