Skip to content

Consul discovery: fix heartbeat and shutdown leaks, send health checks consistently, simplify registration - #1758

Merged
bart-vmware merged 6 commits into
mainfrom
consul-fixes
Oct 20, 2026
Merged

bart-vmware merged 6 commits into
mainfrom
consul-fixes

Conversation

@bart-vmware

@bart-vmware bart-vmware commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Summary

Fixes leaks and a stream of failing heartbeat requests in the Consul discovery client, and simplifies the registration code behind it.

For Steeltoe users

Everything here is a fix. No configuration changes are needed to adopt it.

  • Configuration reloads no longer start phantom heartbeats for an instance that was never registered (a failing request and an error log every ~20 seconds, never stopped).
  • Apps without a known port, or with RegisterHealthCheck=false, no longer send heartbeats that fail with a 404.
  • Apps without a port now use a TTL health check, instead of unintentionally staying healthy forever.
  • A missing port for an HTTP health check is no longer silent: a warning is logged.
  • A changed heartbeat interval now applies to the running heartbeat (before, it was ignored when the instance ID was generated).
  • ShutdownAsync always releases the heartbeat and its subscription, also for disabled clients, and a failed start leaves nothing running.
  • Deregistration is reliable: a registered app is deregistered even when a reload later sets Register=false, and a failed registration no longer causes a deregistration error at shutdown.
  • IsSecure and the secure metadata now agree when Scheme is not lowercase (for example HTTPS).
  • ConsulDiscoveryClient.GetLocalServiceInstance() is annotated as non-null (nullability only; it never returned null).

For reviewers

Structure

Type Role
ConsulServiceInstance One immutable IServiceInstance for the local app (FromOptions) and remote instances (FromServiceEntry). Replaces ThisServiceInstance, ConsulRegistration and the old constructor-based version.
ConsulServiceRegistration What to register with Consul, decided once from an options snapshot: Payload, CheckMode (None/Ttl/Http), HeartbeatOptions (only for a TTL check), and a flag for the unknown-port warning.
ConsulServiceRegistrar Lifecycle only: start, retry/FailFast, ownership of the single heartbeat, shutdown, deregistration. Replaces the old registrar, ConsulServiceRegistry and TtlScheduler.
PeriodicHeartbeat Sends the TTL heartbeats, owns its options subscription, applies interval changes.

ConsulServiceRegistration is unrelated to the deleted ConsulRegistration. The old type was a third copy of the instance data; the new one only describes what is sent to Consul. The diff shows one type removed and one added, not a rename.

Also deleted: SetStatusAsync/GetStatusAsync (no callers) and four container registrations that only tests resolved. All removed types were internal.

Design decisions

  • Decide once. The check type used to be decided in two places (payload: only if Port > 0; heartbeat: whenever enabled), which caused the 404s. The ConsulServiceRegistration constructor is now the only decision point.
  • One owner per resource. The heartbeat is created (and subscribed to options changes) before ServiceRegister, so a reload during a slow registration is not lost. It is started after success and disposed on failure, cancellation or shutdown.
  • No lock-free code. ConsulServiceRegistrar is deliberately not thread-safe: StartAsync runs once in the client constructor (the container serializes singleton construction) and ShutdownAsync runs afterwards, from the hosted service. The only concurrent code is the options-change callback, confined to PeriodicHeartbeat behind one private lock that is never held across an await. This follows from experiments on .NET 8, 9 and 10: a callback can still run after unsubscribing (so the heartbeat ignores changes once disposed, or setting the period of a disposed PeriodicTimer throws on the reload thread), and setting PeriodicTimer.Period restarts the countdown even for the same value (so unchanged intervals are not applied, or unrelated reloads would postpone heartbeats).
  • StartAsync reads the options once, so payload, heartbeat and retry cannot disagree. Shutdown deregisters based on whether registration succeeded.
  • ConsulServiceInstance keeps Host separate while Port and IsSecure come from the URI, because Uri.Host is normalized and Host is what is sent to Consul.

Tests

  • Test folders and namespaces now mirror the project. Registered content, registrar behavior and reload handling are covered by ConsulServiceRegistrationTest, ConsulServiceRegistrarTest and PeriodicHeartbeatTest.
  • Tests configure scenarios with app settings, dispose everything they create, and do not depend on elapsed time (CI thread pool starvation).
  • TestOptionsMonitor (Common.TestResources) now returns a real subscription and exposes ListenerCount, which makes leak assertions possible.

Quality checklist

  • Your code complies with our Coding Style.
  • You've updated unit and/or integration tests for your change, where applicable.
  • You've updated documentation for your change, where applicable.
    If your change affects other repositories, such as Documentation and/or Samples, add linked PRs here.
  • There's an open issue for the PR that you are making. If you'd like to propose a new feature or change, please open an issue to discuss the change or find an existing issue.
  • You've added required license files and/or file headers (explaining where the code came from with proper attribution), where code is copied from StackOverflow, a blog, or OSS.

@bart-vmware bart-vmware changed the title Consul fixes Consul discovery: fix heartbeat and shutdown leaks, send health checks consistently, simplify registration Oct 16, 2026
@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary - All Code Coverage (ubuntu-latest)

Line coverage Branch coverage

Assembly Line coverage Branch coverage
Steeltoe.Bootstrap.AutoConfiguration 97.4% 83.3%
Steeltoe.Common 92.9% 84.4%
Steeltoe.Common.Certificates 95.6% 83.3%
Steeltoe.Common.Hosting 83.5% 65%
Steeltoe.Common.Http 97.2% 80.9%
Steeltoe.Common.Logging 82.9% 56.2%
Steeltoe.Common.Net 64.5% 66.6%
Steeltoe.Configuration.Abstractions 96.3% 89.4%
Steeltoe.Configuration.CloudFoundry 98.4% 92.7%
Steeltoe.Configuration.ConfigServer 92.1% 87.2%
Steeltoe.Configuration.Encryption 97.6% 87.5%
Steeltoe.Configuration.Kubernetes.ServiceBindings 95.6% 82.1%
Steeltoe.Configuration.Placeholder 94% 82.6%
Steeltoe.Configuration.RandomValue 93.2% 83.3%
Steeltoe.Configuration.SpringBoot 98.3% 85%
Steeltoe.Connectors 94.5% 85.5%
Steeltoe.Connectors.EntityFrameworkCore 81% 70.8%
Steeltoe.Discovery.Configuration 96.3% 75%
Steeltoe.Discovery.Consul 97.2% 90.6%
Steeltoe.Discovery.Eureka 90% 81.3%
Steeltoe.Discovery.HttpClients 89.9% 91.1%
Steeltoe.Logging.Abstractions 99.4% 95.6%
Steeltoe.Logging.DynamicConsole 100% 92.3%
Steeltoe.Logging.DynamicSerilog 99.1% 95.4%
Steeltoe.Management.Abstractions 100% 100%
Steeltoe.Management.Endpoint 95.9% 86.9%
Steeltoe.Management.GitProperties.Build 95.2% 83.9%
Steeltoe.Management.Prometheus 96.8% 81.2%
Steeltoe.Management.Tasks 95.2% 80%
Steeltoe.Management.Tracing 100% 75%
Steeltoe.Security.Authentication.JwtBearer 100% 95.6%
Steeltoe.Security.Authentication.OpenIdConnect 87.8% 77.2%
Steeltoe.Security.Authorization.Certificate 96.8% 66.6%
Steeltoe.Security.DataProtection.Redis 100% ****

@bart-vmware
bart-vmware marked this pull request as ready for review October 16, 2026 15:16
@bart-vmware bart-vmware added Component/Discovery Issues related to Steeltoe Service Discovery ReleaseLine/4.x Identified as a feature/fix for the 4.x release line labels Oct 16, 2026
@bart-vmware bart-vmware added this to the 4.3.1 milestone Oct 16, 2026
@bart-vmware
bart-vmware requested a review from TimHess October 16, 2026 15:16
@bart-vmware
bart-vmware requested a review from TimHess October 20, 2026 09:03
@sonarqubecloud

Copy link
Copy Markdown

@TimHess TimHess left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bart-vmware
bart-vmware merged commit 8497ec6 into main Oct 20, 2026
18 checks passed
@bart-vmware
bart-vmware deleted the consul-fixes branch October 20, 2026 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Component/Discovery Issues related to Steeltoe Service Discovery ReleaseLine/4.x Identified as a feature/fix for the 4.x release line

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants