Skip to content

perf: reduce allocations when pushing scopes via ILogger.BeginScope - #5828

Merged
jamescrosswell merged 4 commits into
getsentry:mainfrom
martincostello:performance-improvements
Oct 29, 2026
Merged

jamescrosswell merged 4 commits into
getsentry:mainfrom
martincostello:performance-improvements

Conversation

@martincostello

@martincostello martincostello commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Description

Every call to ILogger.BeginScope made while Sentry's logging integration is registered calls Hub.PushScope, which clones the current Scope. Applications that create logging scopes for each request or message therefore clone a scope many times a second. In a Pyroscope profile of an ASP.NET Core application under a load test, Scope.Clone (and the Scope constructor) was the largest single source of allocations, at about 21% of all the bytes allocated by the process, so this PR reduces what a scope costs to create and clone. Each change is in its own commit.

  1. Create the collections of a Scope lazily

    Every Scope eagerly allocated three ConcurrentDictionary instances (tags, extras and the one inside SentryContexts), which made up around 80% of the 4.4 KB each scope cost on a 20 core machine, as well as the attachments and the SDK version. The tags, extras, contexts, attachments and SDK version are now created on first use, and Clone and Apply skip copying them when the source has none. The public properties still return the same live instance once it has been created, including when first read concurrently. Reading one of the properties creates the (empty) collection, so code that reads them on every scope pays for it on the first read.

  2. Copy breadcrumbs without intermediate allocations in Scope.Clone

    Clone enumerated the breadcrumbs of the source scope (which takes a snapshot array) and enqueued them into the clone one by one, repeatedly growing the clone's backing list. With the default of 100 breadcrumbs that was around 3 KB for each clone. A new ConcurrentQueueLite.EnqueueRange copies them with a single, exactly sized allocation instead. Ordering and the maximum number of breadcrumbs are unchanged.

    I considered making Clone O(1) with copy-on-write breadcrumbs, but that means replacing the list-backed queue with a persistent or chunked structure, which changes breadcrumb ordering, trimming and thread-safety semantics, so I left it out.

  3. Only format a log message once it is known to be needed in SentryLogger

    SentryLogger.Log invoked the formatter before checking whether the entry came from the Sentry SDK itself, was an EF exception message, or was excluded by a log entry filter, so the message was formatted (and allocated) for every entry that was then discarded. The filter check also allocated a closure for every entry through LINQ's Any(). The formatting now happens after those checks and the Any() is a loop.

There are no public API changes.

Issues

No related issue.

Benchmarks

The benchmarks measure, with the real Hub and a no-op transport:

  • ScopeClone: Scope.Clone() of a scope that has one tag and the given number of breadcrumbs.
  • PushScope_NoState: IHub.PushScope() and disposing the returned scope.
  • PushScope_Dictionary8: IHub.PushScope(state) where the state is a Dictionary<string, object> with 8 entries, which is what ILogger.BeginScope is typically given by applications that use structured scopes, and disposing the returned scope.
  • Log_*: SentryLogger.Log for an entry whose category is from the Sentry SDK, an Information entry that adds a breadcrumb, and a Debug entry below the configured thresholds.

Breadcrumbs is the number of breadcrumbs on the scope that is cloned (the default maximum is 100). ConcurrentDictionary allocates for each processor, so the figures below are for a machine with 20 logical processors; with DOTNET_PROCESSOR_COUNT=2 (a small container) the savings are smaller but still substantial, for example PushScope_Dictionary8 with no breadcrumbs allocates 3.33 KB on main and 2,056 B on this branch (a ratio of 0.60).

The baseline is main at f39654d, the commit this branch is based on. Both were run back-to-back in alternating order for two rounds, with --affinity 4095 (0xFFF) so that measurements only use the performance cores of the hybrid CPU. The means below are the average of the two rounds. Ratios are branch / main, so lower is better, and allocation reductions are in bold.

Allocations are deterministic, while timings vary from run to run. The Scope benchmarks were consistently faster in both rounds (per-round time ratios between 0.11 and 0.66). Log_Information_Breadcrumb is unchanged in time (the two rounds were 1.13 and 0.86), and Log_BelowThreshold_Debug measured 4.7 ns against 6.6 ns, which is a difference of about 2 ns on a path that returns before reaching any of the changed code, so I believe it to be noise or code layout.

BenchmarkDotNet v0.15.2, Windows 11 (10.0.26300.9711)
13th Gen Intel Core i7-13700H 2.90GHz, 1 CPU, 20 logical and 14 physical cores
.NET SDK 10.0.402
  [Host] : .NET 10.0.14 (10.0.1426.52326), X64 RyuJIT AVX2
Affinity=00000000111111111111  Toolchain=InProcessNoEmitToolchain  IterationCount=20  WarmupCount=5
Method Breadcrumbs Mean (main) Mean (branch) Ratio Allocated (main) Allocated (branch) Ratio
ScopeClone 0 692 ns 361 ns 0.52 4.43 KB 1,888 B 0.42
PushScope_NoState 0 735 ns 271 ns 0.37 4.66 KB 968 B 0.20
PushScope_Dictionary8 0 1.27 μs 804 ns 0.63 5.23 KB 2,704 B 0.50
ScopeClone 10 944 ns 375 ns 0.40 4.85 KB 1,992 B 0.40
PushScope_NoState 10 945 ns 264 ns 0.28 5.09 KB 1,072 B 0.21
PushScope_Dictionary8 10 1.62 μs 881 ns 0.54 5.65 KB 2,808 B 0.49
ScopeClone 100 3.07 μs 413 ns 0.13 7.38 KB 2,712 B 0.36
PushScope_NoState 100 2.51 μs 291 ns 0.12 7.61 KB 1,792 B 0.23
PushScope_Dictionary8 100 3.17 μs 844 ns 0.27 8.17 KB 3,528 B 0.42
Log_FromSentryCategory - 30.6 ns 8.9 ns 0.29 128 B - 0.00
Log_Information_Breadcrumb - 210 ns 206 ns 0.98 760 B 704 B 0.93
Log_BelowThreshold_Debug - 4.7 ns 6.6 ns 1.39 - - -

Tests

Sentry.Tests (2,694 passed, 5 skipped) and Sentry.Extensions.Logging.Tests (151 passed) pass on net10.0 (Sentry.AspNetCore.Tests passed before the review feedback was addressed, and was not re-run), and Sentry builds without warnings for net8.0, net9.0, net10.0, netstandard2.0, netstandard2.1 and net462. The Android, iOS and Mac Catalyst targets were not built locally. New tests cover the lazy collections (including concurrent first access), clones being independent of their source, breadcrumb ordering on clones, ConcurrentQueueLite.EnqueueRange, and SentryLogger not invoking the formatter for Sentry-originated or below-threshold entries (both of those logger tests fail on main). ScopeTests.Clone_EveryFieldHandled is updated for the renamed _sdk backing field.

Benchmark code

The SentryRepo property is the root of the clone to benchmark. To compare main to the branch, build the project twice with -p:SentryRepo=<path to each clone> and -o set to separate folders, with NO_ANDROID=1, NO_IOS=1 and NO_MACCATALYST=1 set to avoid building the mobile targets, then run each build with dotnet <folder>/sentrybench.dll --filter * --affinity 4095. Set DOTNET_PROCESSOR_COUNT=2 to run with 2 logical processors.

sentrybench.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <Optimize>true</Optimize>
    <NoWarn>$(NoWarn);CS8002;NU1701</NoWarn>
    <SentryRepo Condition="'$(SentryRepo)' == ''">..\sentry-dotnet</SentryRepo>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.15.2" />
    <ProjectReference Include="$(SentryRepo)\src\Sentry\Sentry.csproj" />
    <ProjectReference Include="$(SentryRepo)\src\Sentry.Extensions.Logging\Sentry.Extensions.Logging.csproj" />
  </ItemGroup>
</Project>

Program.cs

using System.Reflection;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
using BenchmarkDotNet.Toolchains.InProcess.NoEmit;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Sentry;
using Sentry.Extensibility;
using Sentry.Extensions.Logging;
using Sentry.Protocol.Envelopes;

var config = DefaultConfig.Instance
    .AddJob(Job.Default
        .WithToolchain(InProcessNoEmitToolchain.Instance)
        .WithLaunchCount(1)
        .WithWarmupCount(5)
        .WithIterationCount(20))
    .WithOptions(ConfigOptions.DisableOptimizationsValidator);
BenchmarkRunner.Run<ScopeBenchmarks>(config, args);
BenchmarkRunner.Run<LoggerBenchmarks>(config, args);

sealed class NoOpTransport : ITransport
{
    public Task SendEnvelopeAsync(Envelope envelope, CancellationToken cancellationToken = default) => Task.CompletedTask;
}

static class HubFactory
{
    public static IHub CreateHub()
    {
        var options = new SentryLoggingOptions
        {
            Dsn = "https://publickey@localhost/1",
            Transport = new NoOpTransport(),
            AutoSessionTracking = false,
            IsGlobalModeEnabled = false,
            DisableFileWrite = true,
            Debug = false,
        };
        SentrySdk.Init(options);
        return HubAdapter.Instance;
    }
}

[MemoryDiagnoser]
public class ScopeBenchmarks
{
    private IHub _hub = null!;
    private Scope _scope = null!;
    private Dictionary<string, object> _state = null!;

    [Params(0, 10, 100)]
    public int Breadcrumbs { get; set; }

    [GlobalSetup]
    public void Setup()
    {
        _hub = HubFactory.CreateHub();
        for (var i = 0; i < Breadcrumbs; i++)
        {
            _hub.AddBreadcrumb($"crumb {i}", "cat");
        }

        // The state of a scope as created by structured logging, e.g. ILogger.BeginScope(Dictionary<string, object>)
        _state = new Dictionary<string, object>(8)
        {
            ["GitHub.Delivery"] = Guid.NewGuid().ToString(),
            ["GitHub.Event"] = "pull_request",
            ["GitHub.Action"] = "opened",
            ["GitHub.Repository.HtmlUrl"] = "https://github.com/martincostello/costellobot",
            ["GitHub.Repository.Name"] = "martincostello/costellobot",
            ["GitHub.Sender.HtmlUrl"] = "https://github.com/dependabot[bot]",
            ["GitHub.Sender.Login"] = "dependabot[bot]",
            ["GitHub.PullRequest.Number"] = "1234",
        };

        var scope = new Scope(new SentryOptions());
        for (var i = 0; i < Breadcrumbs; i++)
        {
            scope.AddBreadcrumb(new Breadcrumb($"crumb {i}", "cat"));
        }

        scope.SetTag("a", "b");
        _scope = scope;
    }

    [Benchmark]
    public Scope ScopeClone() => _scope.Clone();

    [Benchmark]
    public void PushScope_NoState() { using (_hub.PushScope()) { } }

    [Benchmark]
    public void PushScope_Dictionary8() { using (_hub.PushScope(_state)) { } }
}


[MemoryDiagnoser]
public class LoggerBenchmarks
{
    private ILogger _logger = null!;
    private ILogger _sentryCategoryLogger = null!;
    private static readonly Func<int, Exception?, string> Formatter = static (s, _) => $"Processed webhook with ID {s} for event pull_request.";

    [GlobalSetup]
    public void Setup()
    {
        var hub = HubFactory.CreateHub();
        var options = new SentryLoggingOptions { Dsn = "https://publickey@localhost/1" };
        var type = typeof(SentryLoggingOptions).Assembly.GetType("Sentry.Extensions.Logging.SentryLoggerProvider", throwOnError: true)!;
        var provider = (ILoggerProvider)Activator.CreateInstance(type, Options.Create(options), hub)!;
        _logger = provider.CreateLogger("MartinCostello.Costellobot.GitHubWebhookDispatcher");
        _sentryCategoryLogger = provider.CreateLogger("Sentry.ISentryClient");
    }

    [Benchmark]
    public void Log_FromSentryCategory() => _sentryCategoryLogger.Log(LogLevel.Information, new EventId(1), 42, null, Formatter);

    [Benchmark]
    public void Log_Information_Breadcrumb() => _logger.Log(LogLevel.Information, new EventId(1), 42, null, Formatter);

    [Benchmark]
    public void Log_BelowThreshold_Debug() => _logger.Log(LogLevel.Debug, new EventId(1), 42, null, Formatter);
}

🤖 Generated with Claude Code

martincostello and others added 3 commits October 29, 2026 12:01
…xts and attachments

Scope.Clone runs on every ILogger.BeginScope call made while Sentry's logging
integration is registered. Every Scope eagerly allocated three
ConcurrentDictionary instances (tags, extras and the one inside SentryContexts),
which made up around 80% of the 4.4 KB each scope cost on a 20 core machine
(about 60% of 2.5 KB with 2 cores).

Create the tags, extras, contexts, attachments and SDK version lazily, and skip
copying them in Clone and Apply when the source has none. The public properties
still return the same live instance once created.

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…ocations

Scope.Clone enumerated the breadcrumbs of the source scope (which takes a
snapshot array) and enqueued them one by one into the clone, repeatedly growing
the clone's backing list. With the default of 100 breadcrumbs that was around
3 KB per clone.

Copy the items of the source queue directly into the clone with a single sized
allocation instead.

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
…e used

SentryLogger.Log invoked the formatter before checking whether the entry came
from the Sentry SDK itself, was an EF exception message or was excluded by a
log entry filter, so the message was formatted (and allocated) for every entry
that was then discarded. The filter check also allocated a closure for every
entry through LINQ's Any().

Move the formatting after those checks and replace the LINQ call with a loop.

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
@martincostello
martincostello marked this pull request as ready for review October 29, 2026 12:29
Copilot AI balanced review requested due to automatic review settings October 29, 2026 12:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@github-actions github-actions Bot added the risk: medium PR risk score: medium label Oct 29, 2026
@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.32%. Comparing base (f39654d) to head (f99b572).
⚠️ Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5828      +/-   ##
==========================================
+ Coverage   76.27%   76.32%   +0.04%     
==========================================
  Files         529      528       -1     
  Lines       19523    19528       +5     
  Branches     3810     3826      +16     
==========================================
+ Hits        14892    14905      +13     
+ Misses       3752     3744       -8     
  Partials      879      879              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ric-oliv ric-oliv left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @martincostello, thank you for the contribution!
This is a nice win for apps that open a lot of logging scopes, and the benchmarks in the description made it easy to follow.

I left a few comments. The one I'd judge more important is the extra asserts in the clone test. The LazyInitializer suggestion and the test cleanup are optional, so take them or leave them as you / @jamescrosswell prefer.

Comment thread src/Sentry/Scope.cs Resolved
Comment thread src/Sentry/Scope.cs Resolved
Comment thread src/Sentry/Scope.cs Resolved
Comment thread src/Sentry/Scope.cs Resolved
Comment thread src/Sentry/Scope.cs Resolved
Comment thread test/Sentry.Tests/ScopeTests.cs Resolved
Comment thread test/Sentry.Tests/ScopeTests.cs Resolved
Use LazyInitializer.EnsureInitialized for the lazily created collections of
Scope instead of hand-written compare-and-swap loops.

Assert in the Clone test that the contexts and attachments of a clone are
independent of the scope they were cloned from, and remove new tests that only
duplicate the coverage of existing tests.

Assisted-by: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
@martincostello

Copy link
Copy Markdown
Contributor Author

@ric-oliv Thanks - the latest commit should resolve your feedback.

@jamescrosswell

jamescrosswell commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Benchmarks

The benchmarks measure, with the real Hub and a no-op transport:

Nice - thanks @martincostello 🙏🏻

The repo already has a benchmarks/Sentry.Benchmarks/ScopeManagementBenchmarks.cs so we could look at adding these to the repo's benchmarks. We don't yet use these in any automations but if we ever do, this might prevent a regression.

Not a blocker though - we can do that as a follow up so as not to hold this PR up.

@jamescrosswell jamescrosswell left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jamescrosswell
jamescrosswell merged commit 5d33189 into getsentry:main Oct 29, 2026
55 checks passed
@martincostello
martincostello deleted the performance-improvements branch October 29, 2026 22:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

risk: medium PR risk score: medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants