Skip to content

fleet-proxy(http): response reader trusts peer-declared sizes (unbounded preallocation) and silently drops the body on a malformed Content-Length, desyncing the pooled connection #5579

Closed

Description

@dezren39

fleet-proxy(http): response reader trusts peer-declared sizes (unbounded preallocation) and silently drops the body on a malformed Content-Length, desyncing the pooled connection

Problem

HttpConn::request in pkgs/fleet/crates/fleet-proxy/src/http.rs — the pooled HTTP/1.1 client the broker's serve side uses for tunneled services (serve.rs pools these connections per target port) — reads responses with none of the bounds the rest of the crate applies, and misparses malformed length headers instead of failing:

  1. Unbounded preallocation from the declared length. At http.rs:147, a Content-Length body is allocated in full before a single body byte arrives: data.resize(len, 0). There is no cap anywhere in the reader. The chunked path has the same shape at http.rs:190 (out.resize(start + size, 0) per chunk, parsed straight from the peer's hex chunk-size line, with no cumulative bound either). The status line and header lines are read with unbounded read_line (http.rs:98, :118). Contrast the same crate's own discipline: fleet-proxy-proto refuses frames over MAX_FRAME_BYTES before allocating, and tunnel.rs caps a CONNECT response at 65536 bytes.
  2. A malformed Content-Length silently becomes "no body". At http.rs:133, content_length = value.parse().ok() — an unparsable value (e.g. Content-Length: abc) turns into None, so the response is treated as a bodiless keep-alive response: the real body bytes are left in the stream, the caller gets an empty body with a success status, and the connection stays reusable and returns to the pool, where the leftover bytes desync the next response parsed on it. Duplicate Content-Length headers with differing values get the same silent treatment (last one wins). Every other malformed element in this reader (status line, header line, chunk size) is a Malformed error that marks the connection non-reusable — the length header is the one exception.

Evidence (verified this run)

Laptop probe crate ~/git/fleet/probe-proxy-http (path dep on the real fleet-proxy crate @ main 9cf2688), canned loopback servers:

  • Invalid CL: response HTTP/1.1 200 OK + Content-Length: abc + body hello → request returns Ok((200, "")) — body silently dropped — and reusable() == true. A second request on the same connection returns Ok((200, "hi")) from a correctly-formed second response: the leftover hello bytes were consumed as the start of the next status line, which only parses by accident (helloHTTP/1.1 200 OK splits so token 1 is 200). A body whose first token differs would corrupt the next caller's response instead.
  • Duplicate differing CLs: Content-Length: 5 then Content-Length: 0, body hello → same result: Ok((200, "")), reusable, silent desync of the next response.
  • Preallocation: response headers declaring Content-Length: 536870912 (512 MiB) with zero body bytes sent and the socket closed → the request errors on EOF only after allocating the full declared body: probe process VmHWM measured 526,616 KiB. A peer can declare up to usize::MAX; the allocation attempt itself is the failure (abort/OOM in the broker daemon), no payload ever needs to arrive.

The response peer is a service the broker tunnels to — not fully trusted input: a buggy or compromised local service, or anything that can answer on a tunneled port, can OOM the broker or poison pooled connections for unrelated later requests.

Why it matters

The broker is long-lived infrastructure; its pooled connections are shared across requests and callers. Today a single malformed response can (a) crash the daemon by allocation, or (b) silently return an empty body to one caller and shift the byte stream under the next caller — a correctness failure that surfaces far from its cause and is nearly impossible to diagnose from the second caller's symptoms.

Acceptance criteria

  • An unparsable Content-Length, or duplicate Content-Length headers whose values differ, is a Malformed error (like the other header defects) and marks the connection non-reusable.
  • Response bodies have a documented maximum size (a named constant, sized for the JSON payloads this client carries, e.g. aligned with an existing crate ceiling): a declared Content-Length above it, a chunk size above it, and a chunked body whose cumulative size exceeds it all fail with an error before allocating the declared amount; the connection is marked non-reusable.
  • Status/header/chunk-size lines have a bounded read (no unbounded read_line accumulation from a peer that never sends a newline).
  • Regression tests (canned loopback responses, following the crate's existing integration-test pattern): invalid CL → Malformed + not reusable; conflicting duplicate CLs → Malformed; oversize declared CL → error with no outsized allocation; oversize chunk → error; a well-formed CL response and a well-formed chunked response still round-trip and stay reusable.

Activity

  1. added
    kind/bugSomething isn't working (prefixed form of the classic bug label)
    task/small-fixQueue packet kind: bounded fix with reproduction and targeted tests
    source/creatorFiled by the new-issue-creator job
    priority/normalOrdinary priority (the default)
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 4, 2026
  2. developing-today commented on Nov 4, 2026

    @developing-today
    Contributor
  3. added
    app/fleetpkgs/fleet multicall binary + platform crates
    on Nov 4, 2026
  4. dezren39 commented on Nov 4, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · filed

    • Agent: new-issue-creator job (lane B, new-issue-creator-code-b)
    • Session / ID: cron new-issue-creator-code-b run 2026-11-04 09:50 CST
    • Found via: filed this run
    • Task: fleet-proxy HttpConn response reader: malformed/conflicting Content-Length silently drops the body and desyncs the pooled connection; peer-declared sizes (Content-Length, chunk sizes, header lines) are trusted with no bound, preallocating the full declared body before any byte arrives. Verified by laptop probe ~/git/fleet/probe-proxy-http against the real crate @9cf2688e4 (invalid CL → Ok((200,"")) + reusable + silent desync; declared 512 MiB with zero bytes sent → VmHWM 526,616 KiB).
    • Expectation: effort/s — strict Content-Length parsing (Malformed + non-reusable), a documented body-size cap enforced before allocation on both the CL and chunked paths, bounded line reads, plus pinning tests.
    • Links: packet enqueued to repo-health-code-pilot this run (small_fix)
  5. added
    work/assignedQueue worker lifecycle state
    queue/claimedQueue worker lifecycle state
    and removed
    queue/queuedHas a queued task in the Supabase agent queue
    on Nov 4, 2026
  6. dezren39 commented on Nov 4, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · picked up

  7. dezren39 commented on Nov 4, 2026

    @dezren39
    MemberAuthor

    🤖 Agent note · done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    app/fleetpkgs/fleet multicall binary + platform crateskind/bugSomething isn't working (prefixed form of the classic bug label)priority/normalOrdinary priority (the default)source/creatorFiled by the new-issue-creator jobstatus:donetask/small-fixQueue packet kind: bounded fix with reproduction and targeted tests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions