Repository navigation
fix: negotiate Opus NACK for received audio in single peer connection mode - #2270
Conversation
… mode Client offers, including the initial offer sent with the join, now add Opus NACK to recvonly audio sections, so subscribed audio can be retransmitted when RED is off (e.g. with E2EE). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: b1ce8ac The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
size-limit report 📦
|
Store the browser-generated initial offer and apply the NACK munging when it is set locally, so a rejected munge still falls back to the browser's own offer. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assert the NACK sections once per offer path, drop the createAndSetAnswer test that this change does not touch, and cut the changeset to one line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same recvonly mid lookup was inlined at all three NACK munge sites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
We have recently updated the CONTRIBUTING guide - please only open PRs for issues that are accepted already (via label). This gives us the chance to comment on how a specific issue should be fixed prior to the fix's implementation. Thanks for your understanding - closing this PR for now. |
|
@1egoman sorry wasn't aware of the contribution workflow, filed an issue |
| function recvonlyMids(media: MediaDescription) { | ||
| return media.direction === 'recvonly' ? [getMidString(media.mid!)] : []; | ||
| } |
There was a problem hiding this comment.
You mention the React Native case in the description. I think it may be the common case there: an RN voice agent app publishes the mic on connect and the agent joins later, so the first audio section the server sees is the sendonly mic. pion takes audio feedback from that first section only (livekit-server doesn't enable multi-codec negotiation), so NACK stays off for the whole connection.
Would it work to add NACK to every audio section in single-PC offers, sendonly included? livekit/livekit#3889 already drops it for RED tracks on the server. Or is there a reason to keep it off the mic sections? #555 took it out of publisher offers, but that was for dual-PC.
There was a problem hiding this comment.
Good catch. React Native skips the initial receive section preallocation, so if the mic is published first, the first audio section is sendonly and NACK is disabled. Pion then uses that feedback configuration for subsequent audio sections, leaving NACK disabled for the connection.
Could we enable Opus NACK on all audio sections in single PC offers, including sendonly, while explicitly preserving the existing dual PC behavior?
Also, it will be great if we can add a regression test for the RN mic first scenario, where no receiving sections are preallocated.
There was a problem hiding this comment.
single pc offers now put opus NACK on every audio section, sendonly included, and dual pc publishers keep none. Added a test for both modes.
Also added the RN mic regression test, where the first offer has no receive sections, just the data channel and a sendonly mic
const MIC_FIRST_OFFER = `v=0
...
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
a=mid:0
m=audio 9 UDP/TLS/RTP/SAVPF 111
a=mid:1
a=sendonly
a=msid:s mic
a=rtpmap:111 opus/48000/2`;
it('offers Opus NACK on a mic published before any subscribed audio section, as on React Native', async () => {
stubPeerConnectionReturning(MIC_FIRST_OFFER);
expect(opusNackMids(await publisherOfferSdp('publisher-only'))).toEqual(['1']);
});| const sdpParsed = parse(initialOffer.sdp ?? ''); | ||
| sdpParsed.media.forEach((media) => { | ||
| ensureIPAddrMatchVersion(media); | ||
| if (media.type === 'audio') { | ||
| ensureAudioNackAndStereo(media, [], recvonlyMids(media)); | ||
| } | ||
| }); | ||
| this.log.debug('setting pending initial offer before processing answer'); | ||
| await this.setMungedSDP(initialOffer, write(sdpParsed)); |
There was a problem hiding this comment.
Nit: this repeats the munge from createInitialOffer, and the two copies must stay in sync. Your "same offer set locally" test catches drift, but what do you think about keeping both in pendingInitialOffer (for example { original, munged }) and passing them straight to setMungedSDP here? The fallback stays the same, and the duplicated ensureIPAddrMatchVersion pass goes away too.
There was a problem hiding this comment.
Refactored this so the offer is only munged once. Could you check it over? I was trying not to touch too much in my first PR, but I think this is nicer.
| expect(transport.getLocalDescription()?.sdp).toBe(initial.offer.sdp); | ||
| }); | ||
|
|
||
| it("falls back to the browser's own offer when the munged one is rejected", async () => { |
There was a problem hiding this comment.
Could we verify initial connection and subsequent audio subscription in Safari and Firefox? Safari with CompressionStream exercises the deferred initial-offer application; Firefox exercises the separate-offer path. Please confirm that NACK is present in the locally applied offer and negotiated answer.
The fallback unit test confirms selection of the original offer, but its permissive mock cannot establish whether a real browser accepts the answer after fallback. That case needs browser coverage if we want to claim the fallback preserves a working connection.
There was a problem hiding this comment.
Ran the following tests on Safari and Firefox against LiveKit Cloud, with two participants each publishing and subscribing:
| Safari 27 (offer with join) | Firefox 160 (separate offer) | |
|---|---|---|
| Connects and subscribes | yes | yes |
| NACK in applied offer and answer | yes | yes |
| Concealed at 10% loss, before → after | ~10% → 3.4–5.2% | ~10% → 5.9–7.2% |
| Munged offer rejected | still connects | still connects |
Firefox doesn't send NACKs for received audio, so only its uplink is repaired.
|
We should port such fix to the rest of SDKs, like Rust, Swift |
|
Hi @jamesdawsonWD, any cycle to address the comments so that we can land this fix soon ? Thanks |
… mode
The server takes audio feedback from the first audio section of the first
offer only. On React Native the mic is often published before any
subscribed audio, so the first audio section is sendonly and NACK stayed
off for the whole connection. Single peer connection offers now carry Opus
NACK on every audio section, whatever its direction. Dual peer connection
publisher offers stay without NACK.
The initial offer is munged once and kept as { original, munged }, so the
offer set locally is the one sent with the join, and a rejected munge still
falls back to the browser's own offer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xianshijing-lk
left a comment
There was a problem hiding this comment.
lgtm, thanks @jamesdawsonWD
|
FYI, we are porting this fix to other SDKs as well, like Rust : livekit/rust-sdks#1695 |
The server takes audio feedback from the first audio section of the first offer only. That section can be the client's own sendonly mic, so single PC offers now add Opus NACK to every audio section, whatever its direction. This matches the final behavior of livekit/client-sdk-js#2270. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Single-PC offers omit plain NACK feedback for Opus, including when the first audio section is send-only. Dual-PC subscriber answers also need to restore NACK when the server offers it. This adds NACK to every Opus payload in single-PC audio offers and accepts server-offered NACK in the corresponding dual-PC answer sections. With RED disabled, this enables retransmission of lost audio. This PR targets `hiroshi/single-pc`, which contains Flutter's single-PC implementation and has not yet merged into `main`. It ports livekit/client-sdk-js#2270 and aligns dual-PC answers with the existing JS and Swift behavior. The offer setting follows the effective connection mode, preserving dual-PC publisher offers after fallback. Subscriber answers match sections by MID and resolve their own Opus payload types. Existing plain or wildcard feedback is retained, NACK PLI does not suppress plain NACK, and repeated munging does not duplicate feedback. The existing SDP fallback handles rejection of modified offers and answers. Only the applied description is signaled; if both modified and original descriptions are rejected, negotiation fails without signaling an unapplied description. Includes a changeset. Validation: - 161 core tests passed with one existing skip, using Flutter 3.44.9. - 20 Opus NACK tests cover offer-mode gating, audio directions, multiple payloads, feedback preservation, send-only-first offers, renegotiation, selective subscriber answers, reordered MIDs, different payload types, fallback, and final failure. - Static analysis on changed Dart files, formatting, import sorting, and git diff --check passed. - Device/server negotiation and controlled packet-loss tests were not run. No dependency or lockfile changes are included.
Subscribed audio gets no NACK today, so lost packets are not retransmitted. libwebrtc advertises only `transport-cc` feedback for Opus. In single PC mode the client sends the offer, and the server answer keeps only feedback that both sides offer. The server also takes audio feedback only from the first audio section of the first offer. The server wants NACK for tracks without RED (livekit/livekit#3889), but no audio NACK is negotiated. Rust turns RED off for E2EE (#706), so Rust E2EE users get neither RED nor NACK. This ports livekit/client-sdk-js#2270, which fixes livekit/client-sdk-js#2277. ## Change - Single PC offers add `a=rtcp-fb:<pt> nack` for every Opus payload type in every audio section, whatever its direction. This includes the initial offer sent with the join, which is the one that counts. The first audio section can be the client's own `sendonly` mic, so `recvonly` sections alone are not enough. This matches the final commit of livekit/client-sdk-js#2270. Dual PC publisher offers do not change. - Existing plain NACK is not added twice. The `*` wildcard counts as existing NACK. `nack pli` is a different feedback type and does not count as NACK. - The munge runs in `munge_offer`, so the initial offer gets it too. The SDP sent with the join is the SDP set as the local description. - Dual PC answers add Opus NACK to each audio section whose offer section, matched by mid, has it. libwebrtc drops it from the answer otherwise. client-sdk-js and client-sdk-swift already do this. - If the munged SDP does not parse, the original SDP is used, as for the existing stereo munge. ## Verified - `cargo test -p livekit --lib`: 91 passed. The new tests cover these cases: - String tests: every Opus payload type in every audio direction gets NACK, with LF and CRLF. Other codecs and video sections do not. Existing, wildcard or tab-separated NACK is not added twice. The answer mirrors the offer by mid. - Tests with real libwebrtc peer connections: the initial offer has NACK in all three directions. The stored offer is the one sent with the join, and NACK stays after the answer is applied. Later offers have NACK in single PC mode and none in dual PC mode. If the remote offer has NACK, the dual PC answer has it, and otherwise not. - With the munges disabled, 7 of the 8 NACK tests fail. The eighth checks that an answer stays unchanged. With only the answer munge disabled, the dual PC test fails, so libwebrtc does drop NACK from the answer. - New e2e test `v1_opus_nack_negotiates_initial_offer` against a local `livekit-server --dev` (1.13.10): the initial offer and the server answer both carry Opus NACK. It fails with the munge disabled. `audio_test` also passes. - `cargo fmt --all --check` passes. `cargo clippy -p livekit --no-deps` has no new findings in the changed code. The tests with real peer connections and the e2e test are adapted from #1695. ## Not done yet - A test under packet loss that compares `nack_count` in the inbound audio stats. Loss injection on macOS needs root, so it did not run. The JS SDK test under 10% loss went from about 300 lost packets per 30 s to 0. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
In single peer connection mode, the default, subscribed audio arrives on
recvonlysections of the client's own offer. Those sections never offered Opus NACK (PCTransport.ts#L704), so lost audio was never retransmitted. This matters most with E2EE, where RED is off.The initial offer needs it too: the server negotiates audio feedback once per connection, from the first offer (pion
mediaengine.go).Evidence
Unit tests: NACK goes on
recvonlyaudio and stays offsendonlyandsendrecvsections in both offer paths, and the initial offer set locally matches the one sent. Two fail onmainand pass here.Chrome, E2EE, single peer connection, local livekit-server, 10% audio loss each way, 30 s runs. Concealed is
concealedSamples / totalSamplesReceivedfor the audio each side receives:Merge Danger
Door: two-way
Blast Radius: single-PC audio
recvonlysections, so if it publishes the mic before any subscribed audio section exists, the connection still negotiates no NACK.🤖 Generated with Claude Code