Skip to content

fix: negotiate Opus NACK for received audio in single peer connection mode - #2270

Merged
jamesdawsonWD merged 5 commits into
mainfrom
fix/single-pc-audio-nack
Nov 18, 2026
Merged

jamesdawsonWD merged 5 commits into
mainfrom
fix/single-pc-audio-nack

Conversation

@jamesdawsonWD

@jamesdawsonWD jamesdawsonWD commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

In single peer connection mode, the default, subscribed audio arrives on recvonly sections of the client's own offer. Those sections never offered Opus NACK (PCTransport.ts#L704), so lost audio was never retransmitted. This matters most with E2EE, where RED is off.

 createInitialOffer, createAndSendOffer
   for each audio section
+    if recvonly, add Opus NACK

 on the answer to the initial offer
   munge the stored browser offer
+    adding the same Opus NACK
   set it locally, or the browser's offer if the munge is rejected

The initial offer needs it too: the server negotiates audio feedback once per connection, from the first offer (pion mediaengine.go).

Evidence

Unit tests: NACK goes on recvonly audio and stays off sendonly and sendrecv sections in both offer paths, and the initial offer set locally matches the one sent. Two fail on main and pass here.

Chrome, E2EE, single peer connection, local livekit-server, 10% audio loss each way, 30 s runs. Concealed is concealedSamples / totalSamplesReceived for the audio each side receives:

Opus NACK negotiated Concealed at listener Concealed at far end
Before no 14.1–15.9% 13.5–13.7%
After yes 2.8–3.5% 2.2–3.1%

Merge Danger

Door: two-way

Blast Radius: single-PC audio

  • The server's answer also puts NACK on the client's own mic section, so non-RED publishers now retransmit lost uplink audio. This helped in testing. RED publishers are unaffected.
  • React Native skips the initial recvonly sections, so if it publishes the mic before any subscribed audio section exists, the connection still negotiates no NACK.
  • Dual peer connection mode is unchanged.
  • Firefox and Safari not tested.

🤖 Generated with Claude Code

… mode

Client offers, including the initial offer sent with the join, now add
Opus NACK to recvonly audio sections, so subscribed audio can be
retransmitted when RED is off (e.g. with E2EE).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: b1ce8ac

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
livekit-client Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

size-limit report 📦

Path Size
dist/livekit-client.esm.mjs 117.21 KB (-0.03% 🔽)
dist/livekit-client.umd.js 126.68 KB (-0.02% 🔽)
dist/core/core.mjs 79.74 KB (+0.1% 🔺)

@jamesdawsonWD
jamesdawsonWD marked this pull request as ready for review November 13, 2026 13:00
devin-ai-integration[bot]

This comment was marked as resolved.

jamesdawsonWD and others added 3 commits November 13, 2026 14:57
Store the browser-generated initial offer and apply the NACK munging
when it is set locally, so a rejected munge still falls back to the
browser's own offer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Assert the NACK sections once per offer path, drop the createAndSetAnswer
test that this change does not touch, and cut the changeset to one line.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The same recvonly mid lookup was inlined at all three NACK munge sites.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@1egoman

1egoman commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

We have recently updated the CONTRIBUTING guide - please only open PRs for issues that are accepted already (via label).

This gives us the chance to comment on how a specific issue should be fixed prior to the fix's implementation. Thanks for your understanding - closing this PR for now.

@jamesdawsonWD

Copy link
Copy Markdown
Contributor Author

@1egoman sorry wasn't aware of the contribution workflow, filed an issue

@lukasIO lukasIO reopened this Nov 16, 2026
Comment thread src/room/PCTransport.ts Outdated
Comment on lines +1316 to +1318
function recvonlyMids(media: MediaDescription) {
return media.direction === 'recvonly' ? [getMidString(media.mid!)] : [];
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mention the React Native case in the description. I think it may be the common case there: an RN voice agent app publishes the mic on connect and the agent joins later, so the first audio section the server sees is the sendonly mic. pion takes audio feedback from that first section only (livekit-server doesn't enable multi-codec negotiation), so NACK stays off for the whole connection.

Would it work to add NACK to every audio section in single-PC offers, sendonly included? livekit/livekit#3889 already drops it for RED tracks on the server. Or is there a reason to keep it off the mic sections? #555 took it out of publisher offers, but that was for dual-PC.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. React Native skips the initial receive section preallocation, so if the mic is published first, the first audio section is sendonly and NACK is disabled. Pion then uses that feedback configuration for subsequent audio sections, leaving NACK disabled for the connection.

Could we enable Opus NACK on all audio sections in single PC offers, including sendonly, while explicitly preserving the existing dual PC behavior?

Also, it will be great if we can add a regression test for the RN mic first scenario, where no receiving sections are preallocated.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

single pc offers now put opus NACK on every audio section, sendonly included, and dual pc publishers keep none. Added a test for both modes.

Also added the RN mic regression test, where the first offer has no receive sections, just the data channel and a sendonly mic

const MIC_FIRST_OFFER = `v=0
...
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
a=mid:0
m=audio 9 UDP/TLS/RTP/SAVPF 111
a=mid:1
a=sendonly
a=msid:s mic
a=rtpmap:111 opus/48000/2`;

it('offers Opus NACK on a mic published before any subscribed audio section, as on React Native', async () => {
  stubPeerConnectionReturning(MIC_FIRST_OFFER);

  expect(opusNackMids(await publisherOfferSdp('publisher-only'))).toEqual(['1']);
});

Comment thread src/room/PCTransport.ts Outdated
Comment on lines 493 to 501
const sdpParsed = parse(initialOffer.sdp ?? '');
sdpParsed.media.forEach((media) => {
ensureIPAddrMatchVersion(media);
if (media.type === 'audio') {
ensureAudioNackAndStereo(media, [], recvonlyMids(media));
}
});
this.log.debug('setting pending initial offer before processing answer');
await this.setMungedSDP(initialOffer, write(sdpParsed));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: this repeats the munge from createInitialOffer, and the two copies must stay in sync. Your "same offer set locally" test catches drift, but what do you think about keeping both in pendingInitialOffer (for example { original, munged }) and passing them straight to setMungedSDP here? The fallback stays the same, and the duplicated ensureIPAddrMatchVersion pass goes away too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored this so the offer is only munged once. Could you check it over? I was trying not to touch too much in my first PR, but I think this is nicer.

Comment thread src/room/PCTransport.test.ts Outdated
expect(transport.getLocalDescription()?.sdp).toBe(initial.offer.sdp);
});

it("falls back to the browser's own offer when the munged one is rejected", async () => {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we verify initial connection and subsequent audio subscription in Safari and Firefox? Safari with CompressionStream exercises the deferred initial-offer application; Firefox exercises the separate-offer path. Please confirm that NACK is present in the locally applied offer and negotiated answer.

The fallback unit test confirms selection of the original offer, but its permissive mock cannot establish whether a real browser accepts the answer after fallback. That case needs browser coverage if we want to claim the fallback preserves a working connection.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ran the following tests on Safari and Firefox against LiveKit Cloud, with two participants each publishing and subscribing:

Safari 27 (offer with join) Firefox 160 (separate offer)
Connects and subscribes yes yes
NACK in applied offer and answer yes yes
Concealed at 10% loss, before → after ~10% → 3.4–5.2% ~10% → 5.9–7.2%
Munged offer rejected still connects still connects

Firefox doesn't send NACKs for received audio, so only its uplink is repaired.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@xianshijing-lk

Copy link
Copy Markdown
Contributor

We should port such fix to the rest of SDKs, like Rust, Swift

@xianshijing-lk

Copy link
Copy Markdown
Contributor

Hi @jamesdawsonWD, any cycle to address the comments so that we can land this fix soon ?

Thanks

… mode

The server takes audio feedback from the first audio section of the first
offer only. On React Native the mic is often published before any
subscribed audio, so the first audio section is sendonly and NACK stayed
off for the whole connection. Single peer connection offers now carry Opus
NACK on every audio section, whatever its direction. Dual peer connection
publisher offers stay without NACK.

The initial offer is munged once and kept as { original, munged }, so the
offer set locally is the one sent with the join, and a rejected munge still
falls back to the browser's own offer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@xianshijing-lk xianshijing-lk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks @jamesdawsonWD

@xianshijing-lk

Copy link
Copy Markdown
Contributor

FYI, we are porting this fix to other SDKs as well, like Rust : livekit/rust-sdks#1695

@github-actions github-actions Bot mentioned this pull request Nov 18, 2026
lukasIO added a commit to livekit/rust-sdks that referenced this pull request Nov 18, 2026
The server takes audio feedback from the first audio section of the
first offer only. That section can be the client's own sendonly mic, so
single PC offers now add Opus NACK to every audio section, whatever its
direction. This matches the final behavior of
livekit/client-sdk-js#2270.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
xianshijing-lk added a commit to livekit/client-sdk-flutter that referenced this pull request Nov 18, 2026
Single-PC offers omit plain NACK feedback for Opus, including when the
first audio section is send-only. Dual-PC subscriber answers also need
to restore NACK when the server offers it. This adds NACK to every Opus
payload in single-PC audio offers and accepts server-offered NACK in the
corresponding dual-PC answer sections. With RED disabled, this enables
retransmission of lost audio.

This PR targets `hiroshi/single-pc`, which contains Flutter's single-PC
implementation and has not yet merged into `main`. It ports
livekit/client-sdk-js#2270 and aligns dual-PC
answers with the existing JS and Swift behavior.

The offer setting follows the effective connection mode, preserving
dual-PC publisher offers after fallback. Subscriber answers match
sections by MID and resolve their own Opus payload types. Existing plain
or wildcard feedback is retained, NACK PLI does not suppress plain NACK,
and repeated munging does not duplicate feedback.

The existing SDP fallback handles rejection of modified offers and
answers. Only the applied description is signaled; if both modified and
original descriptions are rejected, negotiation fails without signaling
an unapplied description. Includes a changeset.

Validation:
- 161 core tests passed with one existing skip, using Flutter 3.44.9.
- 20 Opus NACK tests cover offer-mode gating, audio directions, multiple
payloads, feedback preservation, send-only-first offers, renegotiation,
selective subscriber answers, reordered MIDs, different payload types,
fallback, and final failure.
- Static analysis on changed Dart files, formatting, import sorting, and
git diff --check passed.
- Device/server negotiation and controlled packet-loss tests were not
run. No dependency or lockfile changes are included.
lukasIO added a commit to livekit/rust-sdks that referenced this pull request Nov 19, 2026
Subscribed audio gets no NACK today, so lost packets are not
retransmitted. libwebrtc advertises only `transport-cc` feedback for
Opus. In single PC mode the client sends the offer, and the server
answer keeps only feedback that both sides offer. The server also takes
audio feedback only from the first audio section of the first offer. The
server wants NACK for tracks without RED (livekit/livekit#3889), but no
audio NACK is negotiated. Rust turns RED off for E2EE (#706), so Rust
E2EE users get neither RED nor NACK. This ports
livekit/client-sdk-js#2270, which fixes livekit/client-sdk-js#2277.

## Change

- Single PC offers add `a=rtcp-fb:<pt> nack` for every Opus payload type
in every audio section, whatever its direction. This includes the
initial offer sent with the join, which is the one that counts. The
first audio section can be the client's own `sendonly` mic, so
`recvonly` sections alone are not enough. This matches the final commit
of livekit/client-sdk-js#2270. Dual PC publisher offers do not change.
- Existing plain NACK is not added twice. The `*` wildcard counts as
existing NACK. `nack pli` is a different feedback type and does not
count as NACK.
- The munge runs in `munge_offer`, so the initial offer gets it too. The
SDP sent with the join is the SDP set as the local description.
- Dual PC answers add Opus NACK to each audio section whose offer
section, matched by mid, has it. libwebrtc drops it from the answer
otherwise. client-sdk-js and client-sdk-swift already do this.
- If the munged SDP does not parse, the original SDP is used, as for the
existing stereo munge.

## Verified

- `cargo test -p livekit --lib`: 91 passed. The new tests cover these
cases:
- String tests: every Opus payload type in every audio direction gets
NACK, with LF and CRLF. Other codecs and video sections do not.
Existing, wildcard or tab-separated NACK is not added twice. The answer
mirrors the offer by mid.
- Tests with real libwebrtc peer connections: the initial offer has NACK
in all three directions. The stored offer is the one sent with the join,
and NACK stays after the answer is applied. Later offers have NACK in
single PC mode and none in dual PC mode. If the remote offer has NACK,
the dual PC answer has it, and otherwise not.
- With the munges disabled, 7 of the 8 NACK tests fail. The eighth
checks that an answer stays unchanged. With only the answer munge
disabled, the dual PC test fails, so libwebrtc does drop NACK from the
answer.
- New e2e test `v1_opus_nack_negotiates_initial_offer` against a local
`livekit-server --dev` (1.13.10): the initial offer and the server
answer both carry Opus NACK. It fails with the munge disabled.
`audio_test` also passes.
- `cargo fmt --all --check` passes. `cargo clippy -p livekit --no-deps`
has no new findings in the changed code.

The tests with real peer connections and the e2e test are adapted from
#1695.

## Not done yet

- A test under packet loss that compares `nack_count` in the inbound
audio stats. Loss injection on macOS needs root, so it did not run. The
JS SDK test under 10% loss went from about 300 lost packets per 30 s to
0.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants