Skip to content

Data race in HTTP/2 flow_control with multiplexed streams #242

Open

Description

@Ph4ntomas

While troubleshooting #241 and #239, I discovered that sometime sending requests with big body could end up in the stream locking up.

I originally thought it was a server issue as the client-side (using http-lua) was timing-out waiting on connection flow credit, while the server (not using lua-http) was not sending WINDOW_UPDATE.

Adding some logging, I noticed a big jump in flow credits here:

lua-http/http/h2_stream.lua

Lines 344 to 354 in ee3cf4b

-- The entire DATA frame payload is included in flow control,
-- including Pad Length and Padding fields if present
local new_stream_peer_flow_credits = self.peer_flow_credits - #payload
local new_connection_peer_flow_credits = self.connection.peer_flow_credits - #payload
if new_stream_peer_flow_credits < 0 or new_connection_peer_flow_credits < 0 then
h2_errors.FLOW_CONTROL_ERROR("not enough flow credits")
end
local ok, err, errno = self:write_http2_frame(frame_types.DATA, flags, payload, timeout, flush)
if not ok then return nil, err, errno end
self.peer_flow_credits = new_stream_peer_flow_credits
self.connection.peer_flow_credits = new_connection_peer_flow_credits

It seems to be possible for a window_update to arrive during the call to self:write_http2_frame, which then breaks the flow-control once the function returns. (confirmed by comparing self.peer_flow_credit and self.connection.peer_flow_credit with backup of themselves taken before the write).

I have 2 fixes in mind:
Decrement the peer_flow_credit before calling self:write_http2_frame (need some way to "reimburse them" if the call fail and to signal it changed in case another stream is waiting to write)
Decrement the peer_flow_credit after the call instead of assigning the precomputed value.

The first one, while more complex would also avoid a secondary issue: If it's possible for the connection to receive a window_update, it might also be possible for another stream to begin sending a chunk while the connection doesn't have enough credit.

Activity

  1. daurnimator commented on Apr 29, 2026

    @daurnimator
    Owner

    I note that :write_chunk (which is the main caller of write_data_frame) already does some splitting up based on flow credit (

    lua-http/http/h2_stream.lua

    Lines 1359 to 1403 in ee3cf4b

    while true do
    while self.peer_flow_credits <= 0 do
    local which = cqueues.poll(self.peer_flow_credits_change, self.connection, timeout)
    if which == self.connection then
    local ok, err, errno = self.connection:step(0)
    if not ok then
    return nil, err, errno
    end
    elseif which == timeout then
    return nil, ce.strerror(ce.ETIMEDOUT), ce.ETIMEDOUT
    end
    timeout = deadline and (deadline-monotime())
    end
    while self.connection.peer_flow_credits <= 0 do
    local which = cqueues.poll(self.connection.peer_flow_credits_change, self.connection, timeout)
    if which == self.connection then
    local ok, err, errno = self.connection:step(0)
    if not ok then
    return nil, err, errno
    end
    elseif which == timeout then
    return nil, ce.strerror(ce.ETIMEDOUT), ce.ETIMEDOUT
    end
    timeout = deadline and (deadline-monotime())
    end
    local SETTINGS_MAX_FRAME_SIZE = self.connection.peer_settings[known_settings.MAX_FRAME_SIZE]
    local max_available = math.min(self.peer_flow_credits, self.connection.peer_flow_credits, SETTINGS_MAX_FRAME_SIZE)
    if max_available < (#payload - sent) then
    if max_available > 0 then
    -- send partial payload
    local ok, err, errno = self:write_data_frame(payload:sub(sent+1, sent+max_available), false, false, timeout)
    if not ok then
    return nil, err, errno
    end
    sent = sent + max_available
    end
    else
    break
    end
    timeout = deadline and (deadline-monotime())
    end
    local ok, err, errno = self:write_data_frame(payload:sub(sent+1), end_stream, false, timeout)
    if not ok then
    return nil, err, errno
    end
    )
    If new credits arrive before the packet is sent, should we try and return control flow to up there?

    Would this be sufficient for a fix?

    -	local new_stream_peer_flow_credits = self.peer_flow_credits - #payload
    -	local new_connection_peer_flow_credits = self.connection.peer_flow_credits - #payload
    -	if new_stream_peer_flow_credits < 0 or new_connection_peer_flow_credits < 0 then
    +	if self.peer_flow_credits < #payload or self.connection.peer_flow_credits < #payload then
     		h2_errors.FLOW_CONTROL_ERROR("not enough flow credits")
     	end
    +	-- Note, this may yield and increment flow credits, do not hoist the calculation above here
     	local ok, err, errno = self:write_http2_frame(frame_types.DATA, flags, payload, timeout, flush)
    	if not ok then return nil, err, errno end
    -	self.peer_flow_credits = new_stream_peer_flow_credits
    -	self.connection.peer_flow_credits = new_connection_peer_flow_credits
    +	self.peer_flow_credits = self.peer_flow_credits - #payload
    +	self.connection.peer_flow_credits = self.connection.peer_flow_credits - #payload
  2. Ph4ntomas commented on Apr 29, 2026

    @Ph4ntomas
    Author

    If new credits arrive before the packet is sent, should we try and return control flow to up there?

    I'm not sure what you mean by that.

    Would this be sufficient for a fix?

    That's the second fix I proposed. It should be enough to prevent setting the credits to a bogus value, but my concern is that it wouldn't handle this case :

    (T1 and T2 are green-threads client-side each handling a different stream. S is the server, | on 2 lines or more means they happens at the same time)

    | Server side: connection control flow credit: 16400
    | T1 -> Pass the flow control check and enter stream:write_data_frame with a payload of 16384 bytes.
    T1 -> enter stream:write_http2_frame and yield
    T2 -> Pass the flow control check and enter stream:write_data_frame with a payload of 16384 bytes
    T2 -> enter stream:write_http2_frame and yield
    | T1 -> write_data_frame resume, connection credit is decremented by the payload size -> connection peer_flow credits =16
    | S -> receive data from T1, decrementing the flow credits to = 16
    T2 -> Write finish. Credit flow is decremented again -> -16368.
    

    In that scenario, if S receive the frame from T2 before it sent a WINDOW_UPDATE frame, it will send an error instead (FLOW_CONTROL_ERROR).

    I saw at least one FLOW_CONTROL_ERROR while trying to reproduce the data-race, which I believe was caused by this scenario, but I haven't been able to confirm it was the case.

  3. Ph4ntomas commented on Apr 29, 2026

    @Ph4ntomas
    Author

    Regarding the fix, I believe this could work:

    -local new_stream_peer_flow_credits = self.peer_flow_credits - #payload
    -local new_connection_peer_flow_credits = self.connection.peer_flow_credits - #payload
    -if new_stream_peer_flow_credits < 0 or new_connection_peer_flow_credits < 0 then
    +if self.peer_flow_credits < #payload or self.connection.peer_flow_credits < #payload then
     	h2_errors.FLOW_CONTROL_ERROR("not enough flow credits")
     end
    +self.peer_flow_credits = self.peer_flow_credits - #payload
    +self.connection.peer_flow_credits = self.connection.peer_flow_credits - #payload
     local ok, err, errno = self:write_http2_frame(frame_types.DATA, flags, payload, timeout, flush)
     if not ok then return nil, err, errno end
    -self.peer_flow_credits = new_stream_peer_flow_credits
    -self.connection.peer_flow_credits = new_connection_peer_flow_credits

    I'm starting to doubt we actually need to deal with failure and rollback the peer_flow_credits.

    stream:write_http2_frame will throw if the stream id is unset, I don't think there's much to be done in that case:

    function stream_methods:write_http2_frame(typ, flags, payload, timeout, flush)
    	local stream_id = assert(self.id, "stream has unset id")
    	return self.connection:write_http2_frame(typ, flags, stream_id, payload, timeout, flush)
    end

    connection:write_http2_frame may fail if the payload is bigger than the frame-size, but that's checked by stream:write_chunk, so it should never happen.

    I'm not sure what can be done if self.socket:xwrite fails. The comment mentions that timeout aren't tied to the write. Assuming that's true, we can ignore timeout.

    The question becomes what other errors are possible in this context.
    AFAIK cqueues should deal with EAGAIN internally, so that error should not be possible.
    This should leave us with critical errors (EPIPE, EBADF), which can't be recovered from anyway.

    -- If this times out, it was the flushing; not the write itself
    -- hence it's not always total failure.
    -- It's up to the caller to take some action (e.g. closing) rather than doing it here
    function connection_methods:write_http2_frame(typ, flags, streamid, payload, timeout, flush)
    	if #payload > self.peer_settings[known_settings.MAX_FRAME_SIZE] then
    		return nil, h2_error.errors.FRAME_SIZE_ERROR:new_traceback("frame too large"), ce.E2BIG
    	end
    	local header = spack(">I3 B B I4", #payload, typ, flags, streamid)
    	local ok, err, errno = self.socket:xwrite(header, "f", 0)
    	if not ok then
    		return nil, err, errno
    	end
    	return self.socket:xwrite(payload, flush, timeout)
    end

    On a side-note, is there a reason to split the xwrite for the header? If we want to rollback the credits (or rather re-add whatever we've substracted), we'd need to know which of the two writes failed.
    Having a single write should ensure the operation is atomic (wrt other green threads) and that it either succeed or fails, without having to deal with partial writes

  4. Ph4ntomas commented on Apr 29, 2026

    @Ph4ntomas
    Author

    Would this be sufficient for a fix?

    -	local new_stream_peer_flow_credits = self.peer_flow_credits - #payload
    -	local new_connection_peer_flow_credits = self.connection.peer_flow_credits - #payload
    -	if new_stream_peer_flow_credits < 0 or new_connection_peer_flow_credits < 0 then
    +	if self.peer_flow_credits < #payload or self.connection.peer_flow_credits < #payload then
     		h2_errors.FLOW_CONTROL_ERROR("not enough flow credits")
     	end
    +	-- Note, this may yield and increment flow credits, do not hoist the calculation above here
     	local ok, err, errno = self:write_http2_frame(frame_types.DATA, flags, payload, timeout, flush)
    	if not ok then return nil, err, errno end
    -	self.peer_flow_credits = new_stream_peer_flow_credits
    -	self.connection.peer_flow_credits = new_connection_peer_flow_credits
    +	self.peer_flow_credits = self.peer_flow_credits - #payload
    +	self.connection.peer_flow_credits = self.connection.peer_flow_credits - #payload

    Okay, so I ran a tests with this.

    I haven´t exactly reduced the setup, but I have a somewhat reliable way to reproduce it.
    For completeness sake:

    • server-side is a tonic grpc server (the stack is tonic/hyper/h2/tokio).
    • client side is grpc client using lua-http for the http2 connection. The payload consists on a few metadata + a 256x256 argb image, and I can start multiple runner concurrently. Each runner is a green-thread that opens a stream and send its payload, then wait for a second. For each iteration, the runner create a new stream, send a header, calls write_chunk with a 1sec timeout, tries to read trailers (this fails, but that's another issue), then shutdown the stream

    The server batches WINDOW_UPDATE, and usually wait for streams to end before sending a WINDOW_UPDATE.

    Without the changes, starting 5 streams is usually enough to break the connection after ~5-10 iteration.
    With the changes quoted above, the connection lasts longer and not longer lockup. However, with enough concurrent runner (~10-15), I eventually receive a GO_AWAY from the server, with the following server-side logs:

    RUST_LOG=warn,h2=debug,h2::proto::streams::recv=trace target/debug/pinnacle -c ~/.config/pinnacle_lua_testbug | grep -C 5 -e "FLOW_CONTROL_ERROR"                                              130 ↵ ──(Wed,Apr29)─┘
    [...]
    2026-04-29T21:04:16.976157Z TRACE Connection: h2::proto::streams::recv: recv_data; size=16384; connection=32325; stream=851968 peer=Server
    2026-04-29T21:04:16.976187Z DEBUG Connection: h2::codec::framed_read: received frame=Data { stream_id: StreamId(223) } peer=Server
    2026-04-29T21:04:16.976204Z TRACE Connection: h2::proto::streams::recv: recv_data; size=16384; connection=15941; stream=835584 peer=Server
    2026-04-29T21:04:16.976236Z DEBUG Connection: h2::proto::streams::recv: connection error FLOW_CONTROL_ERROR -- window_size (15941) < sz (16384); peer=Server
    2026-04-29T21:04:16.976261Z DEBUG Connection: h2::proto::connection: Connection::poll; connection error error=GoAway(b"", FLOW_CONTROL_ERROR, Library) peer=Server
    2026-04-29T21:04:16.976321Z DEBUG Connection: h2::codec::framed_read: received frame=WindowUpdate { stream_id: StreamId(131), size_increment: 29 } peer=Server
    2026-04-29T21:04:16.976397Z DEBUG Connection: h2::codec::framed_write: send frame=GoAway { error_code: FLOW_CONTROL_ERROR, last_stream_id: StreamId(225) } peer=Server
    2026-04-29T21:04:16.976424Z DEBUG Connection: h2::proto::connection: Connection::poll; connection error error=GoAway(b"", FLOW_CONTROL_ERROR, Library) peer=Server
    [...]
    

    That matches the secondary issue I mentioned earlier. I'll rollback the changes, apply the version with pre-decrement and report back.

    EDIT: Decrementing before calling connection:write_http2_frame solves both issues.

    I'm still unsure about rolling-back if the write fails (or rather, I'm unsure whether the write can fail in such a way that rolling back make sense). I'll open a PR as-is for now, feel free to let me know what you think about that either here or on the PR.

    As for the double write in connection:write_http2_frame, it doesn't seem to be causing issues, but I don't know if it's due to the buffering preventing a yield of if it's due to luck during my tests.

    Changing to a single write is not hard either, but I'll let you decide:

    function connection_methods:write_http2_frame(typ, flags, streamid, payload, timeout, flush)
    	if #payload > self.peer_settings[known_settings.MAX_FRAME_SIZE] then
    		return nil, h2_error.errors.FRAME_SIZE_ERROR:new_traceback("frame too large"), ce.E2BIG
    	end
    -	local header = spack(">I3 B B I4", #payload, typ, flags, streamid)
    +	local header = spack(">I3 B B I4 c"..tostring(#payload), #payload, typ, flags, streamid, payload)
     	return self.socket:xwrite(header, flush, timeout)
    -	local ok, err, errno = self.socket:xwrite(header, "f", 0)
    -	if not ok then
    -		return nil, err, errno
    -	end
    -	return self.socket:xwrite(payload, flush, timeout)
     end
    
  5. daurnimator commented on May 1, 2026

    @daurnimator
    Owner

    is there a reason to split the xwrite for the header? If we want to rollback the credits (or rather re-add whatever we've substracted), we'd need to know which of the two writes failed.
    Having a single write should ensure the operation is atomic (wrt other green threads) and that it either succeed or fails, without having to deal with partial writes

    The two xwrite calls is what makes it atomic: the first one uses "f" as the buffer mode argument; which is "full buffering". i.e. atomically put the whole string into the socket buffer. The second xwrite call is what (optionally) flushes and actually calls the write syscall.

    We do it this way to

    1. make sure we don't have partial writes
    2. avoid a write syscall (and potentially TCP packet) that contains just a frame header
  6. Ph4ntomas commented on May 1, 2026

    @Ph4ntomas
    Author

    Got it, thanks!

  7. daurnimator commented on May 3, 2026

    @daurnimator
    Owner

    Idea: should the :write_http2_frame call itself use full buffering "f" and then only after doing the flow credit decrement, we flush?

  8. Ph4ntomas commented on May 3, 2026

    @Ph4ntomas
    Author

    I'll give it a try. It sounds like it would allow us to have a proper error-handling while ensuring we avoid a potential context-switch before the decrement.

    If it works, I'll update the PR :)

  9. Ph4ntomas commented on May 3, 2026

    @Ph4ntomas
    Author

    Idea: should the :write_http2_frame call itself use full buffering "f" and then only after doing the flow credit decrement, we flush?

    It works on my setup, but giving a bit more thought, I don't know if it will work all the time.

    cqueues.socket documentation mention that 'f' may still flush the buffer:

    f | fully buffered output -- all buffer-size blocks are flushed after every write operation.

    It should be fine tho, since new_connection() set the buffer size to math.huge. If we consider changing the buffer size after creating the connection a user-error, that shouldn't be an issue.

  10. daurnimator commented on May 3, 2026

    @daurnimator
    Owner

    It should be fine tho, since new_connection() set the buffer size to math.huge. If we consider changing the buffer size after creating the connection a user-error, that shouldn't be an issue.

    I think that was my intention.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions