Skip to content

Encoder: skip the YUV -> BGR -> YUV conversion - #14

Merged
berndpfrommer merged 3 commits into
ros-misc-utilities:masterfrom
lukas-pospichal:low-cpu-encoding
Nov 18, 2026
Merged

berndpfrommer merged 3 commits into
ros-misc-utilities:masterfrom
lukas-pospichal:low-cpu-encoding

Conversation

@lukas-pospichal

Copy link
Copy Markdown
Contributor

We use H.264 compression for a RealSense D435 on a Jetson AGX Orin with
the hardware encoder from the guide. At 1920x1080 and 30 Hz, the
RealSense driver process used ~87% of one CPU core, even for a static
scene. We found the image went through YUYV -> RGB8 (RealSense driver) -> BGR8
(cv_bridge in the ffmpeg_encoder_decoder) -> YUV420P (swscale), i.e. YUV to RGB and
back.

This PR:

  • passes the source image to swscale directly instead of converting it
    to bgr8 first (and without copying it, when possible),
  • fixes the mapping of yuv422 / yuv422_yuy2, which was needed for YUYV
    input and also fixes the Decoder output for such streams,
  • drops SWS_ACCURATE_RND in the Encoder (slightly different rounding).

Measured CPU of the RealSense driver process in % of one core:

camera RGB8 camera YUYV
without this PR 87%, 30 Hz 71%, drops to 18 Hz
with this PR 70%, 30 Hz 22%, 30 Hz

Questions welcome.

@berndpfrommer berndpfrommer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall things look good.

The only thing missing is a test. Should not be too hard for Claude to write a test for the following scenario:

  • setCVBridgeTargetFormat() is not called
  • the format does not need cvBridge conversion
    If possible this should be done by leveraging the existing EncoderDecoderTester class.

Comment thread src/utils.cpp Resolved
Comment thread src/encoder.cpp Resolved
Comment thread src/encoder.cpp Resolved
Comment thread src/encoder.cpp Resolved
@berndpfrommer

Copy link
Copy Markdown
Contributor

The tests fail for CI, but they pass on my local machine under jazzy/noble. Weird.

@lukas-pospichal

lukas-pospichal commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

The tests fail for CI, but they pass on my local machine under jazzy/noble. Weird.

There is a bug, I have a fix coming. It manifests only in some format combinations.

I expect humble, lyrical and rolling to pass now. I do not know what caused the 404 on jazzy and kilted.

lukas-pospichal and others added 3 commits November 18, 2026 15:55
Before this commit, ros_to_av_pix_format() in src/utils.cpp mapped the
ROS encoding yuv422 to AV_PIX_FMT_YUV422P and yuv422_yuy2 to
AV_PIX_FMT_YUV422P16LE. Both AV_* pixel formats are planar and have 2
and 4 bytes per pixel, respectively.

sensor_msgs/image_encodings.hpp lists yuv422 as a deprecated alias of
uyvy and yuv422_yuy2 as a deprecated alias of yuyv. All four encodings
are packed (not planar) formats with 2 bytes per pixel. With this wrong
pixel format mapping, swscale reads or writes a different memory layout
than the ROS image has:
- By default, the Decoder converts each decoded frame back to the
  original encoding of the stream. For a stream that an Encoder
  created from yuv422_yuy2 images, e.g. from a RealSense camera that
  publishes YUYV, the Decoder allocates the output image for 2 bytes
  per pixel, but swscale writes planar data with 4 bytes per pixel
  into it.
- If an application calls Encoder::setCVBridgeTargetFormat() with
  yuv422_yuy2, swscale reads the source image as 16-bit planar data
  and reads beyond the end of the image buffer.

This commit maps yuv422 to AV_PIX_FMT_UYVY422 and yuv422_yuy2 to
AV_PIX_FMT_YUYV422, the pixel formats that ros_to_av_pix_format()
already uses for uyvy and yuyv.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Michal Sojka <michal.sojka@cvut.cz>
Before this commit, Encoder::encodeImage() converted every image to bgr8
with cv_bridge::toCvCopy(), because the cv_bridge target format of the
Encoder defaulted to bgr8 and foxglove_compressed_video_transport does
not change it. Encoder::doEncodeImage() then calls sws_scale() to
convert the bgr8 image to the pixel format of the codec, e.g. YUV420P.
For an rgb8 image, cv_bridge::toCvCopy() copied the full frame (6 MB at
1920x1080) and swapped two channels. For a yuv422_yuy2 (YUYV) image,
cv_bridge converted YUV to BGR, and sws_scale() immediately converted
BGR back to YUV. Both conversions run in the image callback of the
publisher, so they add to the processing time of every frame. On a
Jetson AGX Orin with 1920x1080 images, sws_scale() alone took about
17 ms per frame, half of the frame period at 30 Hz.

This commit changes the Encoder to hand source images to sws_scale()
unchanged:
- The cv_bridge target format now defaults to empty. If the user does
  not set one with Encoder::setCVBridgeTargetFormat(),
  Encoder::doOpenCodec() uses the source encoding when sws_scale() reads
  it directly (rgb8, bgr8, rgba8, bgra8, yuv422, yuv422_yuy2; see
  utils::requires_cv_bridge_conversion()), and bgr8 otherwise, as
  before. A target format set by the user stays as it is.
- If the message encoding equals the cv_bridge target format and the
  image rows have no padding, Encoder::encodeImage() wraps the message
  data in a cv::Mat of the same size and OpenCV type, instead of
  copying it with cv_bridge::toCvCopy(). Encoder::encodeImage() still
  copies images with row padding, because Encoder::doEncodeImage()
  expects rows without padding.
- New tests encode rgb8, yuv422_yuy2 and yuv422 images without a
  cv_bridge target format and check the decoded images and the packet
  sizes that the Encoder reports during encoding.

For a YUYV source and a YUV420P codec format, sws_scale() now only
unpacks the image into planes and subsamples its chroma vertically; no
RGB conversion takes place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Michal Sojka <michal.sojka@cvut.cz>
Before this commit, Encoder::doOpenCodec() created the Encoder's
swscale context with the flag SWS_ACCURATE_RND. SWS_ACCURATE_RND makes
swscale use more precise rounding and disables some of its faster
conversion routines. sws_scale() pays for this precision on every
frame in the image callback of the publisher. The precision brings no
visible benefit, because the lossy codec that follows discards far
more detail than the rounding preserves.

This commit adds Encoder::setAccurateRounding(bool), which defaults to
false, and creates the Encoder's swscale context with SWS_ACCURATE_RND
only when it is enabled. By default, the color conversion in the
Encoder now rounds slightly differently, so encoded pixel values may
change. This commit does not change the Decoder's swscale context,
which keeps SWS_ACCURATE_RND.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Michal Sojka <michal.sojka@cvut.cz>
@berndpfrommer

Copy link
Copy Markdown
Contributor

The tests fail for CI, but they pass on my local machine under jazzy/noble. Weird.

There is a bug, I have a fix coming. It manifests only in some format combinations.

I expect humble, lyrical and rolling to pass now. I do not know what caused the 404 on jazzy and kilted.

They are all passing now. What was the bug you mentioned? I didn't see any fix (and also don't have your original files anymore since I pulled your new commits).

@lukas-pospichal

lukas-pospichal commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author

The tests fail for CI, but they pass on my local machine under jazzy/noble. Weird.

There is a bug, I have a fix coming. It manifests only in some format combinations.
I expect humble, lyrical and rolling to pass now. I do not know what caused the 404 on jazzy and kilted.

They are all passing now. What was the bug you mentioned? I didn't see any fix (and also don't have your original files anymore since I pulled your new commits).

Sorry, the diff before/after my rebase is here. I opted for clean commit messages. The bug fix I mentioned is this in src/encoder.cpp:
image

I had originally not relized packets returned during encoding reported the row length in bytes as their width (1920 instead of 640 for a bgr8 test image)

Thus: the failed test

@berndpfrommer

Copy link
Copy Markdown
Contributor

OK, looks good now. Thanks for contributing to this module!

@berndpfrommer
berndpfrommer merged commit 303ace0 into ros-misc-utilities:master Nov 18, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants