Skip to content

[Perf] Windows/x64: 27 Regressions on 1/11/2024 10:37:40 PM #97093

Closed
Closed
[Perf] Windows/x64: 27 Regressions on 1/11/2024 10:37:40 PM#97093
Assignees
jakobbotsch
Labels
arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-windowsruntime-coreclrspecific to the CoreCLR runtime
Milestone
9.0.0

Description

@performanceautofiler
performanceautofiler
bot
opened on Jan 16, 2024

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEach<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.22 μs 2.41 μs 1.09 0.28 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach&lt;Int32&gt;*'
Details

Payloads

Baseline
Compare

System.Collections.IterateForEach<Int32>.ConcurrentQueue(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.ReadOnlySpan

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
162.96 ns 177.93 ns 1.09 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.ReadOnlySpan*'
Details

Payloads

Baseline
Compare

System.Memory.ReadOnlySpan.IndexOfString(input: "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAXAAAAAAAAAAAAAAAAAAAAAAAAAAAAA", value: "x", comparisonType: InvariantCultureIgnoreCase)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
17.50 μs 19.09 μs 1.09 0.12 False
1.70 μs 1.95 μs 1.15 0.08 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear&lt;String&gt;*'
Details

Payloads

Baseline
Compare

System.Collections.CreateAddAndClear<String>.ConcurrentBag(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.CreateAddAndClear<String>.Array(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.ContainsFalse<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
17.50 μs 21.54 μs 1.23 0.04 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.ContainsFalse&lt;Int32&gt;*'
Details

Payloads

Baseline
Compare

System.Collections.ContainsFalse<Int32>.Array(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Xml.Linq.Perf_XName

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
38.21 ns 45.50 ns 1.19 0.48 False
13.96 ns 14.97 ns 1.07 0.07 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Xml.Linq.Perf_XName*'
Details

Payloads

Baseline
Compare

System.Xml.Linq.Perf_XName.CreateElementWithNamespaceImplicitOperator

ETL Files

Histogram

JIT Disasms

System.Xml.Linq.Perf_XName.CreateElement

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_String

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
123.62 ns 146.69 ns 1.19 0.01 True
9.30 ns 10.71 ns 1.15 0.05 True
50.58 ns 59.10 ns 1.17 0.03 True
30.39 ns 44.11 ns 1.45 0.08 True

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_String*'
Details

Payloads

Baseline
Compare

System.Tests.Perf_String.IndexerCheckPathLength

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.ToUpper(s: "TEST")

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.Join_List

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_String.IndexerCheckBoundCheckHoist

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IterateForEachNonGeneric<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.16 μs 1.59 μs 1.36 0.03 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEachNonGeneric&lt;String&gt;*'
Details

Payloads

Baseline
Compare

System.Collections.IterateForEachNonGeneric<String>.Stack(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Globalization.Tests.StringSearch

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
497.06 ns 559.61 ns 1.13 0.05 False
497.60 ns 558.89 ns 1.12 0.07 False
17.63 ns 19.51 ns 1.11 0.06 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Globalization.Tests.StringSearch*'
Details

Payloads

Baseline
Compare

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, IgnoreCase, False))

ETL Files

Histogram

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (, IgnoreCase, False))

ETL Files

Histogram

JIT Disasms

System.Globalization.Tests.StringSearch.IndexOf_Word_NotFound(Options: (en-US, Ordinal, False))

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
10.63 ns 11.93 ns 1.12 0.21 False
17.46 ns 20.68 ns 1.18 0.17 False
13.05 ns 14.56 ns 1.12 0.24 False
10.50 ns 12.16 ns 1.16 0.06 False

graph
graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'
Details

Payloads

Baseline
Compare

System.Memory.Span<Char>.StartsWith(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.SequenceEqual(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.IndexOfAnyTwoValues(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.BinarySearch(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_Frozen<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
14.38 ns 16.10 ns 1.12 0.25 False
11.71 ns 14.24 ns 1.22 0.17 True

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_Frozen&lt;Int16&gt;*'
Details

Payloads

Baseline
Compare

System.Collections.Perf_Frozen<Int16>.TryGetValue_True(Count: 4)

ETL Files

Histogram

JIT Disasms

System.Collections.Perf_Frozen<Int16>.Contains_True(Count: 4)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Run Information

Name Value
Architecture x64
OS Windows 10.0.22621
Queue OwlWindows
Baseline 8f79b66e76081559f71969988037790d3e53367e
Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in Benchstone.BenchI.TreeInsert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
43.34 μs 47.30 μs 1.09 0.03 True

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.TreeInsert*'
Details

Payloads

Baseline
Compare

Benchstone.BenchI.TreeInsert.Test

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

Activity

  1. performanceautofiler
    added
    arch-x64
    os-windows
    runtime-coreclrspecific to the CoreCLR runtime
    untriagedNew issue has not been triaged by the area owner
    on Jan 16, 2024

  2. performanceautofiler commented on Jan 16, 2024

    @performanceautofiler
    performanceautofilerbot
    on Jan 16, 2024 – with PerformanceAutoFiler
    Author

    Run Information

    Name Value
    Architecture x64
    OS Windows 10.0.22621
    Queue OwlWindows
    Baseline 8f79b66e76081559f71969988037790d3e53367e
    Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
    Diff Diff
    Configs CompilationMode:tiered, RunKind:micro

    Regressions in System.Collections.IterateForEach<String>

    Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
    2.44 μs 2.58 μs 1.06 0.08 False
    1.31 μs 1.45 μs 1.10 0.11 False

    graph
    graph
    Test Report

    Repro

    General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

    git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IterateForEach&lt;String&gt;*'
    Details

    Payloads

    Baseline
    Compare

    System.Collections.IterateForEach<String>.FrozenDictionary(Size: 512)

    ETL Files

    Histogram

    JIT Disasms

    System.Collections.IterateForEach<String>.IEnumerable(Size: 512)

    ETL Files

    Histogram

    JIT Disasms

    Docs

    Profiling workflow for dotnet/runtime repository
    Benchmarking workflow for dotnet/runtime repository

    Run Information

    Name Value
    Architecture x64
    OS Windows 10.0.22621
    Queue OwlWindows
    Baseline 8f79b66e76081559f71969988037790d3e53367e
    Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
    Diff Diff
    Configs CompilationMode:tiered, RunKind:micro

    Regressions in Benchstone.BenchI.NDhrystone

    Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
    343.06 ms 406.60 ms 1.19 0.17 False

    graph
    Test Report

    Repro

    General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

    git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'Benchstone.BenchI.NDhrystone*'
    Details

    Payloads

    Baseline
    Compare

    Benchstone.BenchI.NDhrystone.Test

    ETL Files

    Histogram

    JIT Disasms

    Docs

    Profiling workflow for dotnet/runtime repository
    Benchmarking workflow for dotnet/runtime repository

    Run Information

    Name Value
    Architecture x64
    OS Windows 10.0.22621
    Queue OwlWindows
    Baseline 8f79b66e76081559f71969988037790d3e53367e
    Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
    Diff Diff
    Configs CompilationMode:tiered, RunKind:micro

    Regressions in System.Xml.Tests.Perf_XmlConvert

    Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
    69.25 ns 82.43 ns 1.19 0.09 True

    graph
    Test Report

    Repro

    General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

    git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Xml.Tests.Perf_XmlConvert*'
    Details

    Payloads

    Baseline
    Compare

    System.Xml.Tests.Perf_XmlConvert.TimeSpan_ToString

    ETL Files

    Histogram

    JIT Disasms

    Docs

    Profiling workflow for dotnet/runtime repository
    Benchmarking workflow for dotnet/runtime repository

    Run Information

    Name Value
    Architecture x64
    OS Windows 10.0.22621
    Queue OwlWindows
    Baseline 8f79b66e76081559f71969988037790d3e53367e
    Compare 22ba7d607bb1d9caa0db9afcdc47eb5cef641fcb
    Diff Diff
    Configs CompilationMode:tiered, RunKind:micro

    Regressions in System.Threading.Tasks.ValueTaskPerfTest

    Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
    16.69 ns 17.78 ns 1.07 0.04 False

    graph
    Test Report

    Repro

    General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

    git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Threading.Tasks.ValueTaskPerfTest*'
    Details

    Payloads

    Baseline
    Compare

    System.Threading.Tasks.ValueTaskPerfTest.CreateAndAwait_FromCompletedValueTaskSource_ConfigureAwait

    ETL Files

    Histogram

    JIT Disasms

    Docs

    Profiling workflow for dotnet/runtime repository
    Benchmarking workflow for dotnet/runtime repository

  3. EgorBo
    closed this as completedon Jan 16, 2024

  4. jakobbotsch commented on Jan 16, 2024

    @jakobbotsch
    jakobbotsch
    on Jan 16, 2024
    Member

    @EgorBo Some of these regressions are likely from #96753 (annoying that both our PRs ended up in this span of commits...). I'm going to take a look at some of the diffs.

  5. jakobbotsch commented on Jan 16, 2024

    @jakobbotsch
    jakobbotsch
    on Jan 16, 2024
    Member

    In fact, only #96731 of the cast work is in this span of commits -- do you expect regressions from that one too?

  6. jakobbotsch commented on Jan 16, 2024

    @jakobbotsch
    jakobbotsch
    on Jan 16, 2024
    Member

    Report: https://gist.github.com/jakobbotsch/a09b4a367d4764e27c0a19798dbadb79

  7. EgorBo
    reopened this on Jan 17, 2024
  8. EgorBo
    transferred this issue fromdotnet/perf-autofiling-issueson Jan 17, 2024
  9. ghost added
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Jan 17, 2024
  10. EgorBo
    added
    area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMI
    and removed
    needs-area-labelAn area label is needed to ensure this gets routed to the appropriate area owners
    on Jan 17, 2024
  11. EgorBo
    assigned
    jakobbotsch
    on Jan 17, 2024
  13. JulieLeeMSFT
    added this to the 9.0.0 milestone on Jan 27, 2024
  14. ghost removed
    untriagedNew issue has not been triaged by the area owner
    on Jan 27, 2024

  15. EgorBo commented on Mar 1, 2024

    @EgorBo
    EgorBo
    on Mar 1, 2024
    Member

    All are no longer relevant

  16. EgorBo
    closed this as completedon Mar 1, 2024
  17. github-actions
    locked and limited conversation to collaborators on Apr 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

arch-x64area-CodeGen-coreclrCLR JIT compiler in src/coreclr/src/jit and related components such as SuperPMIos-windowsruntime-coreclrspecific to the CoreCLR runtime

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions