Repository navigation
Description
While troubleshooting #241 and #239, I discovered that sometime sending requests with big body could end up in the stream locking up.
I originally thought it was a server issue as the client-side (using http-lua) was timing-out waiting on connection flow credit, while the server (not using lua-http) was not sending WINDOW_UPDATE.
Adding some logging, I noticed a big jump in flow credits here:
Lines 344 to 354 in ee3cf4b
| -- The entire DATA frame payload is included in flow control, | |
| -- including Pad Length and Padding fields if present | |
| local new_stream_peer_flow_credits = self.peer_flow_credits - #payload | |
| local new_connection_peer_flow_credits = self.connection.peer_flow_credits - #payload | |
| if new_stream_peer_flow_credits < 0 or new_connection_peer_flow_credits < 0 then | |
| h2_errors.FLOW_CONTROL_ERROR("not enough flow credits") | |
| end | |
| local ok, err, errno = self:write_http2_frame(frame_types.DATA, flags, payload, timeout, flush) | |
| if not ok then return nil, err, errno end | |
| self.peer_flow_credits = new_stream_peer_flow_credits | |
| self.connection.peer_flow_credits = new_connection_peer_flow_credits |
It seems to be possible for a window_update to arrive during the call to self:write_http2_frame, which then breaks the flow-control once the function returns. (confirmed by comparing self.peer_flow_credit and self.connection.peer_flow_credit with backup of themselves taken before the write).
I have 2 fixes in mind:
Decrement the peer_flow_credit before calling self:write_http2_frame (need some way to "reimburse them" if the call fail and to signal it changed in case another stream is waiting to write)
Decrement the peer_flow_credit after the call instead of assigning the precomputed value.
The first one, while more complex would also avoid a secondary issue: If it's possible for the connection to receive a window_update, it might also be possible for another stream to begin sending a chunk while the connection doesn't have enough credit.
Activity
daurnimator commentedon Apr 29, 2026
I note that
:write_chunk(which is the main caller ofwrite_data_frame) already does some splitting up based on flow credit (lua-http/http/h2_stream.lua
Lines 1359 to 1403 in ee3cf4b
If new credits arrive before the packet is sent, should we try and return control flow to up there?
Would this be sufficient for a fix?
Ph4ntomas commentedon Apr 29, 2026
I'm not sure what you mean by that.
That's the second fix I proposed. It should be enough to prevent setting the credits to a bogus value, but my concern is that it wouldn't handle this case :
(T1 and T2 are green-threads client-side each handling a different stream. S is the server,
|on 2 lines or more means they happens at the same time)In that scenario, if
Sreceive the frame from T2 before it sent aWINDOW_UPDATEframe, it will send an error instead (FLOW_CONTROL_ERROR).I saw at least one
FLOW_CONTROL_ERRORwhile trying to reproduce the data-race, which I believe was caused by this scenario, but I haven't been able to confirm it was the case.Ph4ntomas commentedon Apr 29, 2026
Regarding the fix, I believe this could work:
I'm starting to doubt we actually need to deal with failure and rollback the peer_flow_credits.
stream:write_http2_framewill throw if the stream id is unset, I don't think there's much to be done in that case:connection:write_http2_framemay fail if the payload is bigger than the frame-size, but that's checked bystream:write_chunk, so it should never happen.I'm not sure what can be done if
self.socket:xwritefails. The comment mentions that timeout aren't tied to the write. Assuming that's true, we can ignore timeout.The question becomes what other errors are possible in this context.
AFAIK cqueues should deal with EAGAIN internally, so that error should not be possible.
This should leave us with critical errors (EPIPE, EBADF), which can't be recovered from anyway.
On a side-note, is there a reason to split the
xwritefor the header? If we want to rollback the credits (or rather re-add whatever we've substracted), we'd need to know which of the two writes failed.Having a single write should ensure the operation is atomic (wrt other green threads) and that it either succeed or fails, without having to deal with partial writes
Ph4ntomas commentedon Apr 29, 2026
Okay, so I ran a tests with this.
I haven´t exactly reduced the setup, but I have a somewhat reliable way to reproduce it.
For completeness sake:
write_chunkwith a 1sec timeout, tries to read trailers (this fails, but that's another issue), then shutdown the streamThe server batches
WINDOW_UPDATE, and usually wait for streams to end before sending aWINDOW_UPDATE.Without the changes, starting 5 streams is usually enough to break the connection after ~5-10 iteration.
With the changes quoted above, the connection lasts longer and not longer lockup. However, with enough concurrent runner (~10-15), I eventually receive a GO_AWAY from the server, with the following server-side logs:
That matches the secondary issue I mentioned earlier. I'll rollback the changes, apply the version with pre-decrement and report back.
EDIT: Decrementing before calling
connection:write_http2_framesolves both issues.I'm still unsure about rolling-back if the write fails (or rather, I'm unsure whether the write can fail in such a way that rolling back make sense). I'll open a PR as-is for now, feel free to let me know what you think about that either here or on the PR.
As for the double write in
connection:write_http2_frame, it doesn't seem to be causing issues, but I don't know if it's due to the buffering preventing a yield of if it's due to luck during my tests.Changing to a single write is not hard either, but I'll let you decide:
function connection_methods:write_http2_frame(typ, flags, streamid, payload, timeout, flush) if #payload > self.peer_settings[known_settings.MAX_FRAME_SIZE] then return nil, h2_error.errors.FRAME_SIZE_ERROR:new_traceback("frame too large"), ce.E2BIG end - local header = spack(">I3 B B I4", #payload, typ, flags, streamid) + local header = spack(">I3 B B I4 c"..tostring(#payload), #payload, typ, flags, streamid, payload) return self.socket:xwrite(header, flush, timeout) - local ok, err, errno = self.socket:xwrite(header, "f", 0) - if not ok then - return nil, err, errno - end - return self.socket:xwrite(payload, flush, timeout) enddaurnimator commentedon May 1, 2026
The two
xwritecalls is what makes it atomic: the first one uses"f"as the buffer mode argument; which is "full buffering". i.e. atomically put the whole string into the socket buffer. The secondxwritecall is what (optionally) flushes and actually calls thewritesyscall.We do it this way to
Ph4ntomas commentedon May 1, 2026
Got it, thanks!
daurnimator commentedon May 3, 2026
Idea: should the
:write_http2_framecall itself use full buffering"f"and then only after doing the flow credit decrement, we flush?Ph4ntomas commentedon May 3, 2026
I'll give it a try. It sounds like it would allow us to have a proper error-handling while ensuring we avoid a potential context-switch before the decrement.
If it works, I'll update the PR :)
Ph4ntomas commentedon May 3, 2026
It works on my setup, but giving a bit more thought, I don't know if it will work all the time.
cqueues.socket documentation mention that 'f' may still flush the buffer:
It should be fine tho, since
new_connection()set the buffer size tomath.huge. If we consider changing the buffer size after creating the connection a user-error, that shouldn't be an issue.daurnimator commentedon May 3, 2026
I think that was my intention.