You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Hyper unexpectedly way slower when using HTTP/2 #3071
[eijebong@polonium] ~/code/hyper-slow (main) >>> time ./target/release/hyper-slow
./target/release/hyper-slow 0.93s user 2.55s system 99% cpu 3.477 total
[eijebong@polonium] ~/code/hyper-slow (main) >>> time ./target/release/hyper-slow http2
./target/release/hyper-slow http2 6.34s user 5.61s system 18% cpu 1:04.53 total
As you can see, if the client is set to do HTTP/2 only, the whole thing is around 20 times slower.
Now, if I remove the tokio::spawn around the client async block, I get ./target/release/hyper-slow http2 3.90s user 4.77s system 111% cpu 7.804 total which is still twice as slow as the HTTP/1 implementation.
Reacted by Eric Zhang, Daniel Sharifi, Igor Novgorodov, Rene Leonhardt and MarkusSintonen
Interesting. We do know HTTP/2 can be slower, depending on the workflow. There's room for optimization. But when I run the exact code you pasted, I only see 2x slower (2s to 4s).
We have a benchmark suite comparing HTTP/1 and HTTP/2 end-to-end in hyper in benches/end_to_end.rs, with a variety of concurrency and body sizes. It does show (and has for a while) that HTTP/2 can be slower. (They're disabled in master, since they need to be updated to the new split-per-version connection types, but the 0.14.x branch still works well.)
The hope is that work done in h2 could be easily compared by checking those benchmarks.
If someone wanted to setup some profiling, and/or produce a flamegraph, of hyper with HTTP/2, that might be a good start to identifying low-hanging fruit to improve performance.
Sorry to add noise here, but I'm getting the same slowness on HTTP/2, and it's quite confusing. These are repeated requests to a default hyper server on localhost:7609 that just returns a plaintext Body with 10 bytes, using the same hyper::Client. And with TCP_NODELAY set.
Most of the time it just takes <300 us, but the HTTP/2 server/client combo (with http2_only(true)) unexpectedly takes 40 milliseconds sometimes, and I'm not sure why.
I can reproduce this on multiple AWS EC2 Ubuntu 20.04 instances. But I can't reproduce it on my ARM-based macOS laptop, nor can I reproduce it on Docker Desktop for macOS.
Not sure how to create a flamegraph but will do my best to look into it more.
similar performance problem observed where even under low load, the h2 connection has tail latencies of > 40ms, but switching to h1 results in <1ms tail latencies. Since this seems to be related to the internals of the h2 crate, is there any guidance you could provide on how to debug this? I wouldn't mind trying to root cause this in the h2 crate.
This was on hyper 1.0.1 - i've measured timings in different parts of the hyper stack and the latency is purely from reading the request body via the collect() method on the body.
We are also observing 2x increase in latencies when we enabled the ALPN that allows using of h2. The load is quite low but the latencies are always higher. In artificial test I can confirm this also - seeing roughly 2x increase.
I wonder if that can be mitigated somehow, we'd like to use h2 anyway at some point to reduce number of connections. Is any work is happening on this? Can hyper 1.x help?
I can reproduce this on multiple AWS EC2 Ubuntu 20.04 instances. But I can't reproduce it on my ARM-based macOS laptop, nor can I reproduce it on Docker Desktop for macOS
That's weird. I was able to reproduce the issue on my ARM-based Macbook Pro with M1 Pro. I see an increase from ~2.5s to ~3.5s when benchmarking with the code in the issue description.
@ekzhang 40ms could be delayed ack. You could try setting tcp_quickack flag to the tcp connection(you need to set it before/after every recv/send, since it gets reset).
Got it, maybe that was the issue. I'll have to test it out. I was setting TCP_NODELAY to turn off Nagle's algorithm which I believed was enough, perhaps not.
There's been a focus on improving HTTP/2 performance this year, mostly in the h2 crate. Several significant improvements have already landed, and more are being worked on. (Some, but not all: hyperium/h2#918hyperium/h2#933hyperium/h2#927)
I'm going to close this as too general, if there are specific things that are problematic we can address them as separate issues.
Activity
seanmonstar commentedon Dec 2, 2022
Interesting. We do know HTTP/2 can be slower, depending on the workflow. There's room for optimization. But when I run the exact code you pasted, I only see 2x slower (2s to 4s).
kaimast commentedon Dec 4, 2022
I also see a latencies that are 2-3x higher when using HTTP/2 over HTTP/1.
smart--petea commentedon Dec 8, 2022
hyper: 0.14.23
tokio: 1.23.0
os: Linux Mint 20.2
only http2: real 0.075s, user 0.023s, sys 0.022s
both: real 0.010s, user 0.004s, sys 0.007s
seanmonstar commentedon Dec 8, 2022
We have a benchmark suite comparing HTTP/1 and HTTP/2 end-to-end in hyper in
benches/end_to_end.rs, with a variety of concurrency and body sizes. It does show (and has for a while) that HTTP/2 can be slower. (They're disabled in master, since they need to be updated to the new split-per-version connection types, but the 0.14.x branch still works well.)The hope is that work done in
h2could be easily compared by checking those benchmarks.seanmonstar commentedon Dec 8, 2022
If someone wanted to setup some profiling, and/or produce a flamegraph, of hyper with HTTP/2, that might be a good start to identifying low-hanging fruit to improve performance.
ekzhang commentedon Jul 21, 2023
Sorry to add noise here, but I'm getting the same slowness on HTTP/2, and it's quite confusing. These are repeated requests to a default hyper server on
localhost:7609that just returns a plaintextBodywith 10 bytes, using the samehyper::Client. And withTCP_NODELAYset.Most of the time it just takes <300 us, but the HTTP/2 server/client combo (with
http2_only(true)) unexpectedly takes 40 milliseconds sometimes, and I'm not sure why.I can reproduce this on multiple AWS EC2 Ubuntu 20.04 instances. But I can't reproduce it on my ARM-based macOS laptop, nor can I reproduce it on Docker Desktop for macOS.
Not sure how to create a flamegraph but will do my best to look into it more.
tjandra-amzn commentedon Jan 3, 2024
similar performance problem observed where even under low load, the h2 connection has tail latencies of > 40ms, but switching to h1 results in <1ms tail latencies. Since this seems to be related to the internals of the h2 crate, is there any guidance you could provide on how to debug this? I wouldn't mind trying to root cause this in the h2 crate.
This was on hyper 1.0.1 - i've measured timings in different parts of the hyper stack and the latency is purely from reading the request body via the
collect()method on the body.blind-oracle commentedon Apr 7, 2024
We are also observing 2x increase in latencies when we enabled the ALPN that allows using of h2. The load is quite low but the latencies are always higher. In artificial test I can confirm this also - seeing roughly 2x increase.
I wonder if that can be mitigated somehow, we'd like to use h2 anyway at some point to reduce number of connections. Is any work is happening on this? Can hyper 1.x help?
DSharifi commentedon Apr 8, 2024
That's weird. I was able to reproduce the issue on my ARM-based Macbook Pro with M1 Pro. I see an increase from ~2.5s to ~3.5s when benchmarking with the code in the issue description.
bngreen commentedon Sep 11, 2024
@ekzhang 40ms could be delayed ack. You could try setting tcp_quickack flag to the tcp connection(you need to set it before/after every recv/send, since it gets reset).
ekzhang commentedon Sep 11, 2024
Got it, maybe that was the issue. I'll have to test it out. I was setting
TCP_NODELAYto turn off Nagle's algorithm which I believed was enough, perhaps not.NiiightmareXD commentedon Jun 29, 2025
It's been about three years. Is this issue fixed now? Can I help in any way? 😊
seanmonstar commentedon Aug 28, 2026
There's been a focus on improving HTTP/2 performance this year, mostly in the
h2crate. Several significant improvements have already landed, and more are being worked on. (Some, but not all: hyperium/h2#918 hyperium/h2#933 hyperium/h2#927)I'm going to close this as too general, if there are specific things that are problematic we can address them as separate issues.