[PERF] Regression on AMD

Between 3.1 and master (5.0) ... 33% degradation

These numbers are for Plaintext, but the same regression appears on all scenarios.

-n Plaintext --webHost KestrelSockets -j /var/jenkins_home/workspace/baseline_citrine_amd/build/../src/Benchmarks/benchmarks.plaintext.json --description Baseline31 --aspnetCoreVersion 3.1 --runtimeVersion 3.1 --sdk 3.1.102

RequestsPerSecond:           3,980,990
Max CPU (%):                 89
WorkingSet (MB):             96
Avg. Latency (ms):           3.25
Startup (ms):                275
First Request (ms):          30.53
Latency (ms):                0.1
Total Requests:              80,017,797
Duration: (ms)               20,100
Socket Errors:               0
Bad Responses:               0
Build Time (ms):             8,502
Published Size (KB):         115,446
SDK:                         3.1.102
Runtime:                     3.1.2
ASP.NET Core:                3.1.2

-n Plaintext --webHost KestrelSockets -j /var/jenkins_home/workspace/baseline_citrine_amd/build/../src/Benchmarks/benchmarks.plaintext.json --description Baseline 

RequestsPerSecond:           2,988,782
Max CPU (%):                 77
WorkingSet (MB):             100
Avg. Latency (ms):           3.42
Startup (ms):                253
First Request (ms):          28.98
Latency (ms):                0.16
Total Requests:              60,073,627
Duration: (ms)               20,100
Socket Errors:               0
Bad Responses:               0
Build Time (ms):             3,501
Published Size (KB):         119,145
SDK:                         5.0.100-preview.2.20120.3
Runtime:                     5.0.0-preview.2.20125.16
ASP.NET Core:                5.0.0-preview.2.20126.7

Running runtime with the AMD fix? #31904

Do these benchmarks hit dotnet/aspnetcore#19009?
The two PRs should mitigate the regression on AMD.

This is using the latest builds, so not older than a day. You can see the exact versions that contain the day of the month in it.

We can get the whole published folder, and also traces if needed.

I have isolated the regression between:

5.0.0-alpha.1.20103.10 https://github.com/dotnet/runtime/tree/a7f8cdc0de945561ae3fcaa72440c0ec56d69060 (Feb 3 12:26 PM PST)
5.0.0-alpha.1.20111.2 https://github.com/dotnet/runtime/tree/f496497de42b80b6d031d5de445ec74b2b01bb17 (Feb 11 1:59 AM PST)

By running the Plaintext NonPipelined scenario (60% regression) like so:

dotnet run -- --server "http://asp-citrine-amd:5001" --client "http://asp-citrine-load:5002" -n plaintextnonpipelined -j ..\Benchmarks\benchmarks.plaintext.json --warmup 5 --duration 10 --self-contained  --aspnetcoreversion 5.0.0-preview.1.20107.8 --sdk 5.0.100-preview.1.20110.1 --runtimeversion 5.0.0-alpha.1.20111.2

The aspnet core version was pinned for both runs showing that the reason seems to be a runtime change. Unfortunately there was no successful runtime build+publication during this window so I can't get more precise that this.

At that point I suggest that someone tries to identify which commits might be related, and try to rebuild the impacted assets of these commits, so we can test them on the benchmark to identity the root cause.

@adamsitnik

Does the regression appear only on Linux or also on Windows?

If it weren't for the fact that I know you and @adamsitnik said you did lots of testing on #2346, that would be my first guess.

Why I thought it might need the AMD fix #31904 is recently Kestrel changed from using its own ASCII encoding to use Encoding.ASCII.GetBytes dotnet/aspnetcore#18404 so would be dependent on and effected by any runtime regressions. However that was merged on 11 Feb so should be in the before and not the after.

on #2346, that would be my first guess.

Is in the commit range

It's interesting that the CPU consumption was never 100% and the RPS for this 48 core machine is smaller compared to the machine which has only 12 cores.

I suspect that there is some underlying scalability issue and my change in #2346 has for some reason made it even worse.

As soon as we get PerfCollect to work on this machine I would be more than happy to analyze the trace files and identify the problem.

The AMD machine has been available for a week only, and is on Linux.

I have seen other unusual behaviors on this machine that I need to investigate, related to the number of cores, so this might be related to Adam's suspicion. I will gather some data.

While I was working on the TechEmpower recently, I found out that the max number of IO Threads is 16 (for some reason, imho it should be Math.Max(Environment.ProcessorCount / 2, 1)).

Could you try to run some benchmarks on this machine with a bigger number? If I remember correctly, it's configurable via --kestrelthreadcount argument

https://github.com/dotnet/aspnetcore/blob/34ee3324fb4a8c8de4ebdee6a6c8ecdbe5891236/src/Servers/Kestrel/Transport.Sockets/src/SocketTransportOptions.cs#L17

I have gathered some data by varying the # of core and IOQueue threads. I'll share the results when I have find some time.

@halter73

Here are some intermediate results: https://gist.github.com/sebastienros/ac204507c98bb2f1d2d38f15d444e572

I ran Plaintext Non Pipelined on the AMD machine (Ubuntu 18.03) with the two runtime versions that show the regression. I did so by making from 1 to 48 cores available. Then for each version pinned to 48 cores and varied the IOQueue threads from 1 to 48, instead of its default value of 16 (https://github.com/dotnet/aspnetcore/blob/34ee3324fb4a8c8de4ebdee6a6c8ecdbe5891236/src/Servers/Kestrel/Transport.Sockets/src/SocketTransportOptions.cs#L17)

What it tells us is:

even by varying the threads count, the results are still bad after the regression
taking the runtime before the change, we can get better results with a different thread count than the default
the performance is not a function of the number of cpus (we would hope for linear)

Right now I am running another script that is varying the available cpus from 1..48, and also the numbers of threads for each number of cpus. That should let us find what default value is best based on the number of cpus. This should also be done on an Intel, citrine having 24 cores.

/cc @halter73

@GrabYourPitchforks

Triage: This may be related to some SIMD code not being that efficient on AMD (@GrabYourPitchforks @tannergooding).
We should re-benchmark - @adamsitnik do you have latest info?

cc @alnikola can you check if it still shows up in latest results when you get a chance?

I am almost sure it's more related to thread scheduling than instructions. If you check the link you will see that we can get 712K RPS at 20 cores, but it goes down to 187K RPS when using the full 48 cores. This machine being an AMD might not be the cause, but the fact that is has more cores than the other machines we usually test on is adding some new information.

@adamsitnik

In that case, I wonder if it is even Networking related ... @adamsitnik can you please help shed some light here?

I have a 12-core/24-thread AMD box I can test on.

Do we know if it also reproduces on Windows (I can also test in Ubuntu 18.04)? Could someone list where to get the benchmark from and what command line to test with (I didn't see it in dotnet/performance)?

You might also have some luck trying to collect a profile using https://developer.amd.com/amd-uprof/ to see what the hotspot differences are.

@sebastienros

@sebastienros I am getting a new error when downloading the trace file.

dotnet run --- --jobs ..\BenchmarksApps\Kestrel\PlatformBenchmarks\benchmarks.plaintext.json --scenario "PlaintextPlatform" --collect-trace

[07:10:15.559] Using worker Wrk
[07:10:15.648] Running session 'e7fcddccb34b4b7284dc4c3a235ba3d9' with description ''
[07:10:15.655] Starting scenario PlaintextPlatform on benchmark server...
[07:10:15.877] Fetching job: http://$secret-amd:5001/jobs/3
[07:10:17.357] Job has been selected by the server ...
[07:10:17.586] Job is now building ...
[07:10:22.122] Job is running
[07:10:22.329] Warmup
[07:10:22.341] Starting scenario PlaintextPlatform on benchmark client...
[07:10:23.016] Client Job ready: http://$secret/Jobs/301
[07:10:49.928] Scenario PlaintextPlatform completed on benchmark client
[07:10:49.935] Deleting scenario PlaintextPlatform on benchmark client...
[07:10:50.351] Starting scenario PlaintextPlatform on benchmark client...
[07:10:51.009] Client Job ready: http://$secret/Jobs/303
[07:11:11.406] Scenario PlaintextPlatform completed on benchmark client
[07:11:11.411] Deleting scenario PlaintextPlatform on benchmark client...
[07:11:12.077] Post-processing profiler trace, this can take 10s of seconds...
[07:11:12.087] Trace arguments: BufferSizeMB=1024;CircularMB=1024;clrEvents=JITSymbols;kernelEvents=process+thread+ImageLoad+Profile
[07:11:44.149] Downloading trace: trace.03-12-19-11-44.RPS-2971K.trace.zip
[07:11:44.378] FAILED: The trace was not successful
RequestsPerSecond:           2,971,231
Max CPU (%):                 49
WorkingSet (MB):             78
Avg. Latency (ms):           1.01
Startup (ms):                8
First Request (ms):          31.62
Latency (ms):                0.18
Total Requests:              44,865,302
Duration: (ms)               15,100
Socket Errors:               0
Bad Responses:               0
Build Time (ms):             2,501
Published Size (KB):         259
SDK:                         5.0.100-preview.2.20120.3
Runtime:                     5.0.0-preview.2.20155.3
ASP.NET Core:                5.0.0-preview.3.20161.8
[07:11:44.405] Stopping scenario 'PlaintextPlatform' on benchmark server...
[07:11:48.738] Interrupting due to an unexpected exception
[07:11:48.786] System.Net.Http.HttpRequestException: No connection could be made because the target machine actively refused it.
 ---> System.Net.Sockets.SocketException (10061): No connection could be made because the target machine actively refused it.
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at BenchmarksDriver.Program.Run(Uri serverUri, Uri[] clientUris, String sqlConnectionString, ServerJob serverJob, String session, String description, Int32 iterations, Int32 exclude, String shutdownEndpoint, TimeSpan span, List`1 downloadFiles, Boolean fetch, String fetchDestination, Boolean collectR2RLog, String traceDestination, CommandOption outputFileOption, CommandOption sourceOption, CommandOption scriptFileOption, CommandOption markdownOption, CommandOption writeToFileOption, Nullable`1 requiredOperatingSystem, CommandOption saveOption, CommandOption diffOption) in C:\Projects\aspnet_benchmarks\src\BenchmarksDriver\Program.cs:line 2260

Could someone list where to get the benchmark from and what command line to test with (I didn't see it in dotnet/performance)?

It's one of the ASP.NET TechEmpower benchmarks. As soon as I get the trace I am going to share it with you

Closing this as it was fixed with Adam reverting the epoll change.

Navigation Menu

Repository navigation

[PERF] Regression on AMD #33084

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[PERF] Regression on AMD #33084

Description

Activity

benaadams commented on Feb 28, 2020

gfoidl commented on Feb 28, 2020

sebastienros commented on Feb 28, 2020

sebastienros commented on Mar 2, 2020

stephentoub commented on Mar 3, 2020

benaadams commented on Mar 3, 2020

benaadams commented on Mar 3, 2020

adamsitnik commented on Mar 3, 2020

sebastienros commented on Mar 3, 2020

adamsitnik commented on Mar 3, 2020

sebastienros commented on Mar 4, 2020

sebastienros commented on Mar 5, 2020

karelz commented on Mar 12, 2020

sebastienros commented on Mar 12, 2020

karelz commented on Mar 12, 2020

tannergooding commented on Mar 12, 2020

tannergooding commented on Mar 12, 2020

adamsitnik commented on Mar 12, 2020

adamsitnik commented on Mar 12, 2020

sebastienros commented on Mar 23, 2020

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions