Detect breaking changes before merge

Why semantic versioning does not prevent runtime breaking changes

Semantic versioning is a communication protocol, not an enforcement mechanism. A library author who increments the major version correctly has communicated that something breaking changed. The downstream consumer who has not yet updated their code still gets a runtime failure the moment they update the package reference without reading the changelog.

More importantly, semver requires the library author to know which changes are breaking in the first place. Research by Dig and Johnson found that 80 percent of the 147 breaking API changes they studied across open source Java projects were caused by refactoring: changes the author considered routine cleanup rather than intentional API evolution [1]. The study examined open-source Java projects; no large-scale .NET-specific dataset substantially contradicts the proportion, though the precise rate likely varies by ecosystem and project type. A developer who renames a method during a refactor sprint is unlikely to reach for the changelog before committing. Semver provides no signal until after the commit is merged and the package is published.

The intent gap matters here. The developer who renamed that method was not negligent: they were doing their job. The build passed, the tests passed, the linter was clean. The breaking change was an unintended side effect of a correct, intentional action. A pre-commit structural check is designed for exactly this case: flagging the unintended consequence before the developer has moved on.

The cost of discovering a breaking change at the consumeris higher than the cost of preventing it at the author. Hora et al. measured how downstream projects reacted when their library dependencies introduced breaking changes and found that many projects simply stopped updating the dependency, accumulating security and correctness debt rather than absorbing the migration cost [2]. The practical implication is that library authors who introduce unintentional breaking changes do not just cause immediate runtime failures; they cause long-term ecosystem fragmentation. Pre-commit detection converts this class of problem from an ecosystem-level event into a single-developer edit.

The multi-assembly problem in .NET: why internal compilation success does not mean external compatibility

In a .NET solution with multiple projects, the build system ensures every project within the solution compiles against the current source. When project A references project B, and a developer changes a public method in B, the build fails at project A immediately. This is the happy path. It is also the narrowest case.

The multi-assembly problem appears in three common real-world shapes:

• 1.Published NuGet packages. Once a library is published to NuGet, any consumer pinned to the previous version will not recompile when the author pushes a new patch. Consumers that accept floating version ranges and update the package will get a binary that no longer matches the signatures their code was compiled against. The result is a MissingMethodException or BadImageFormatException at the first call into the changed surface.
• 2.Plugin and extension architectures. Any host application that loads assemblies at runtime (MEF, Roslyn analyzers, ASP.NET Core middleware loaded via reflection) cannot recompile the plugins when the host API changes. The failure surface is entirely in the runtime layer, and it typically surfaces in production long after the host was updated.
• 3.Microservice contracts. When two services communicate over HTTP or a message bus, each service is compiled independently. Changing the shape of a shared data transfer object in one service does not cause a compile error in the other, even if both services live in the same repository. The failure appears at runtime when a payload of the old shape arrives at a deserializer expecting the new shape, often silently producing null-valued fields rather than a thrown exception.

Microsoft tracks hundreds of compatibility breaks introduced across .NET framework versions, many of which involve scenarios where the compiler produced no error [3]. The documentation distinguishes binary incompatible changes from source incompatible changes: two distinct failure modes that require different mitigation strategies and that a build server running inside a single solution boundary cannot detect.

Breaking changes in .NET OSS history: real examples from well-known libraries

The .NET ecosystem provides several well-documented examples of breaking changes that caused significant migration effort across the community. Each one follows the same pattern: correct compilation, runtime failure, delayed discovery.

Newtonsoft.Json to System.Text.Json. When Microsoft introduced System.Text.Json as the default serializer in ASP.NET Core 3.0, teams that migrated from Newtonsoft.Json encountered serialization contract breaks that were invisible at compile time. Newtonsoft serializes public fields by default; System.Text.Json does not. Newtonsoft performs case-insensitive property matching by default; System.Text.Json requires explicit configuration for the same behavior. Properties decorated with [JsonProperty] from Newtonsoft were silently ignored by the new serializer, causing wire format changes that surfaced only when stored payloads were read back or when downstream services that had not yet migrated received responses in the new format. None of these differences produced compile-time errors.

Entity Framework Core 6 to 7. EF Core 7 introduced breaking changes to how owned entity types are mapped to tables. Applications that relied on the previous behavior compiled without modification but produced different SQL at runtime. In some cases the queries returned wrong data rather than throwing an exception, which is the hardest failure mode to detect in automated testing. The EF Core team documented these changes in their migration guide, but developers who upgraded the package without reading the guide had no automated signal that behavior had changed.

AutoMapper major versions. AutoMapper removed its static API in version 9.0. Applications using the static API compiled against version 8.x would fail to compile against 9.x, but those that had already compiled and were running against 8.x would break only when the package version was updated and the host restarted. Teams running integration tests against the actual binary rather than recompiling from source would not catch the failure until deployment. The pattern repeats across the ecosystem because the toolchain provides no mechanism for flagging the risk at the point where the change is authored.

A breaking change the compiler cannot see

The following example represents the class of failures GauntletCI is designed to catch. The library compiles without warnings. The application compiles without warnings. The failure appears at runtime.

Suppose a shared library ships this public API in version 1.0:

public class OrderProcessor
{
    public void Process(Order order, bool sendNotification)
    {
        // process the order, optionally send notification
    }
}

A developer decides that notifications should always be sent and removes the parameter during a cleanup commit. The library is published as v1.1, a minor version bump, because the developer considers it a simplification rather than a breaking change:

public class OrderProcessor
{
    // sendNotification removed -- notifications always sent now
    public void Process(Order order)
    {
        // process the order
    }
}

Any application compiled against v1.0 that calls processor.Process(order, true) will receive the following exception the first time that code path executes after the library is updated in production:

System.MissingMethodException: Method not found:
  'Void MyLibrary.OrderProcessor.Process(MyLibrary.Order, Boolean)'

GauntletCI rule GCI0004 detects this at diff time. When it sees that a public method signature was removed with no matching overload added, it produces a high-severity finding before the commit is created. The developer learns about the break while still in their editor, not after the library is deployed and a consumer integration test suite begins to fail.

The correct fix is straightforward once the risk is visible: keep the old overload, mark it [Obsolete], and delegate to the new signature. Consumers continue to compile and run. The old overload can be removed in the next major version after consumers have had time to migrate. This is the standard .NET deprecation path, and it is the action GCI0004 suggests in its finding output.

This is the core insight behind diff-based analysis: by analyzing exactly the lines that are about to change, GauntletCI identifies the structural risk at the only moment when it is still cheap to address it.

Breaking change patterns in .NET

The patterns below represent the structural changes that most commonly cause runtime failures across .NET applications. Each pattern can be introduced by a well-intentioned commit (a refactor, a cleanup, a schema update) that passes all automated tests because the test suite was written against the post-change codebase and does not exercise the consumer boundary at the binary level.

What GauntletCI checks: the rule set for breaking change detection

GauntletCI ships a library of deterministic rules that map directly to the breaking change categories above. Each rule is applied to the staged diff (the lines that are actually changing), so the analysis is scoped to the risk introduced by this specific commit, not the entire codebase. The following rules are most relevant to breaking change detection:

Code review alone cannot reliably catch these patterns at the pace modern teams merge. Boehm and Basili documented that the cost to fix a defect rises by roughly an order of magnitude for each phase it survives past the point of introduction [4]. The specific multipliers vary by project context, but the directional finding has been replicated across multiple software engineering cost models. A breaking change caught in the diff before the commit costs a single edit. The same breaking change caught in production costs an incident, a rollback, and an investigation. Why code review misses bugs explores this cost curve in more detail.

The right time to detect is before the commit

Finding a breaking change in post-deploy monitoring means a rollback, an incident, and a post-mortem. Finding it in code review means a comment and a revision. Finding it before the commit is created means a fix before anyone else is involved.

Static analysis scoped to the staged diff rather than the entire codebase delivers feedback at the only moment when the cost of a fix is minimal: before the commit exists. The analysis surface is bounded by the size of the commit, so the check completes in milliseconds regardless of codebase size. GauntletCI implements this model with deterministic structural rules: every public API removal, every serialization contract break, and every unsafe concurrency pattern in the diff produces a finding on every run.

Determinism is what separates structural rules from probabilistic analysis. A tool that produces inconsistent results trains developers to dismiss findings as noise. A structural rule that fires every time a public method is removed without a replacement overload carries no false-negative rate for that pattern. Developers quickly learn which finding types require mandatory action and which allow deliberate acknowledgment, and that distinction preserves the signal value of every finding the tool produces.

Post-publish compatibility tools such as Microsoft.DotNet.ApiCompat and binary compatibility analyzers verify API surface after a library is built or released. They are valuable for library maintainers managing a public NuGet surface across versions. GauntletCI is designed for an earlier moment: before the commit is created, before CI runs, and before any consumer is affected: the only point where the cost of the fix is zero and the developer still has full context to address it.

References

1. [1]Dig, D. and Johnson, R. "How Do APIs Evolve? A Story of Refactoring." Journal of Software Maintenance and Evolution, 2006. Studied 147 breaking API changes across open source Java projects and found 80 percent were caused by refactoring. https://dl.acm.org/doi/10.1002/smr.328
2. [2]Hora, A., et al. "How Do Developers React When Their Libraries Break?" ICSME 2015. https://ieeexplore.ieee.org/document/7332473
3. [3]Microsoft .NET Breaking Changes documentation. https://learn.microsoft.com/en-us/dotnet/core/compatibility/breaking-changes
4. [4]Boehm, B. and Basili, V.R. "Software Defect Reduction Top 10 List." IEEE Computer, 34(1), 2001. Documents the order-of-magnitude cost increase for defects that survive each successive development phase.

Real-world examples from .NET OSS

These case studies show GauntletCI catching the exact patterns described above in real pull requests to widely-used .NET libraries.