Skip to content

Clarify VideoFrame.copyTo() parallel steps and buffer handling - #958

Open
Djuffin wants to merge 2 commits into
w3c:mainfrom
Djuffin:copyTo
Open

Djuffin wants to merge 2 commits into
w3c:mainfrom
Djuffin:copyTo

Conversation

@Djuffin

@Djuffin Djuffin commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Avoid accessing or mutating JS objects from background parallel steps in VideoFrame.copyTo():

  • Check if destination is detached before starting the copy.
  • Capture [[resource reference]], options, and PlaneLayout results synchronously on the event loop thread (fixing PlaneLayout.stride to use destinationStride).
  • Perform RGB pixel format and color space conversion on the media resource in parallel instead of synchronously constructing a temporary VideoFrame and recursing.
  • Write directly to shared memory in parallel when destination is backed by a SharedArrayBuffer; otherwise, copy into destination on a queued event loop task after re-verifying detachment and byteLength (while allowing UAs to copy synchronously when the resource is already in CPU memory).
  • Set the default value of VideoFrameCopyToOptions.colorSpace to "srgb" in IDL.

Fixes #955


Preview | Diff

Comment thread index.src.html Outdated
|combinedLayout|, and |sharedMemory|.
2. [=Queue a task=] to [=resolve=] |p| with |planeLayouts|.
3. Otherwise:
1. Wait until |resource| is available in CPU memory.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems it should also be happening if |sharedMemory| is not null.

Comment thread index.src.html Outdated
[=BufferSource/underlying buffer=] is a {{SharedArrayBuffer}}, set
|sharedMemory| to a reference to the `destination.byteLength` bytes of
|destination|'s [=BufferSource/underlying buffer=]'s
`[[ArrayBufferData]]` starting at `destination.byteOffset`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

|sharedMemory| definition is somehow ambiguous, it is not clear whether keeping a ref to it is protecting destination's memory from being GCed.
Maybe destination.[[ArrayBufferData]] is what we need.

Comment thread index.src.html Outdated

NOTE: When |sharedMemory| is `null` and |resource| is already accessible
in CPU memory, User Agents can run [=VideoFrame/Copy Planes=]
synchronously before returning |p| to avoid an extra copy.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This note will likely trigger web compatibility troubles, for instance:

let buffer = new Uint8Array(...);
videoFrame.copyTo(buffer); // do not await here.
channel.postMessage(buffer, [buffer]);

As per the above algorithm, no data would be copied to buffer in copyTo.
If implementing the note, copyTo will copy data to buffer before buffer is transferred.

It seems we should either remove that note or make it mandatory.
I am also not sure why doing things synchronously avoids an extra copy.

Avoid accessing or mutating JS objects from background parallel steps in
VideoFrame.copyTo():

- Check if destination is detached before starting the copy.
- Capture [[resource reference]], options, and PlaneLayout results
  synchronously on the event loop thread (fixing PlaneLayout.stride to
  use destinationStride).
- Perform RGB pixel format and color space conversion on the media
  resource in parallel instead of synchronously constructing a temporary
  VideoFrame and recursing.
- Write directly to shared memory in parallel when destination is backed
  by a SharedArrayBuffer; otherwise, copy into destination on a queued
  event loop task after re-verifying detachment and byteLength (while
  allowing UAs to copy synchronously when the resource is already in CPU
  memory).
- Set the default value of VideoFrameCopyToOptions.colorSpace to "srgb"
  in IDL.

Fixes w3c#955

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

VideoFrame.copyTo is manipulating JS objects in a background queue

2 participants