Repository navigation
Encoder: skip the YUV -> BGR -> YUV conversion - #14
Conversation
berndpfrommer
left a comment
There was a problem hiding this comment.
Overall things look good.
The only thing missing is a test. Should not be too hard for Claude to write a test for the following scenario:
- setCVBridgeTargetFormat() is not called
- the format does not need cvBridge conversion
If possible this should be done by leveraging the existing EncoderDecoderTester class.
|
The tests fail for CI, but they pass on my local machine under jazzy/noble. Weird. |
There is a bug, I have a fix coming. It manifests only in some format combinations. I expect humble, lyrical and rolling to pass now. I do not know what caused the 404 on jazzy and kilted. |
Before this commit, ros_to_av_pix_format() in src/utils.cpp mapped the ROS encoding yuv422 to AV_PIX_FMT_YUV422P and yuv422_yuy2 to AV_PIX_FMT_YUV422P16LE. Both AV_* pixel formats are planar and have 2 and 4 bytes per pixel, respectively. sensor_msgs/image_encodings.hpp lists yuv422 as a deprecated alias of uyvy and yuv422_yuy2 as a deprecated alias of yuyv. All four encodings are packed (not planar) formats with 2 bytes per pixel. With this wrong pixel format mapping, swscale reads or writes a different memory layout than the ROS image has: - By default, the Decoder converts each decoded frame back to the original encoding of the stream. For a stream that an Encoder created from yuv422_yuy2 images, e.g. from a RealSense camera that publishes YUYV, the Decoder allocates the output image for 2 bytes per pixel, but swscale writes planar data with 4 bytes per pixel into it. - If an application calls Encoder::setCVBridgeTargetFormat() with yuv422_yuy2, swscale reads the source image as 16-bit planar data and reads beyond the end of the image buffer. This commit maps yuv422 to AV_PIX_FMT_UYVY422 and yuv422_yuy2 to AV_PIX_FMT_YUYV422, the pixel formats that ros_to_av_pix_format() already uses for uyvy and yuyv. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Michal Sojka <michal.sojka@cvut.cz>
Before this commit, Encoder::encodeImage() converted every image to bgr8 with cv_bridge::toCvCopy(), because the cv_bridge target format of the Encoder defaulted to bgr8 and foxglove_compressed_video_transport does not change it. Encoder::doEncodeImage() then calls sws_scale() to convert the bgr8 image to the pixel format of the codec, e.g. YUV420P. For an rgb8 image, cv_bridge::toCvCopy() copied the full frame (6 MB at 1920x1080) and swapped two channels. For a yuv422_yuy2 (YUYV) image, cv_bridge converted YUV to BGR, and sws_scale() immediately converted BGR back to YUV. Both conversions run in the image callback of the publisher, so they add to the processing time of every frame. On a Jetson AGX Orin with 1920x1080 images, sws_scale() alone took about 17 ms per frame, half of the frame period at 30 Hz. This commit changes the Encoder to hand source images to sws_scale() unchanged: - The cv_bridge target format now defaults to empty. If the user does not set one with Encoder::setCVBridgeTargetFormat(), Encoder::doOpenCodec() uses the source encoding when sws_scale() reads it directly (rgb8, bgr8, rgba8, bgra8, yuv422, yuv422_yuy2; see utils::requires_cv_bridge_conversion()), and bgr8 otherwise, as before. A target format set by the user stays as it is. - If the message encoding equals the cv_bridge target format and the image rows have no padding, Encoder::encodeImage() wraps the message data in a cv::Mat of the same size and OpenCV type, instead of copying it with cv_bridge::toCvCopy(). Encoder::encodeImage() still copies images with row padding, because Encoder::doEncodeImage() expects rows without padding. - New tests encode rgb8, yuv422_yuy2 and yuv422 images without a cv_bridge target format and check the decoded images and the packet sizes that the Encoder reports during encoding. For a YUYV source and a YUV420P codec format, sws_scale() now only unpacks the image into planes and subsamples its chroma vertically; no RGB conversion takes place. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Michal Sojka <michal.sojka@cvut.cz>
Before this commit, Encoder::doOpenCodec() created the Encoder's swscale context with the flag SWS_ACCURATE_RND. SWS_ACCURATE_RND makes swscale use more precise rounding and disables some of its faster conversion routines. sws_scale() pays for this precision on every frame in the image callback of the publisher. The precision brings no visible benefit, because the lossy codec that follows discards far more detail than the rounding preserves. This commit adds Encoder::setAccurateRounding(bool), which defaults to false, and creates the Encoder's swscale context with SWS_ACCURATE_RND only when it is enabled. By default, the color conversion in the Encoder now rounds slightly differently, so encoded pixel values may change. This commit does not change the Decoder's swscale context, which keeps SWS_ACCURATE_RND. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: Michal Sojka <michal.sojka@cvut.cz>
9973acd to
a2d1d55
Compare
They are all passing now. What was the bug you mentioned? I didn't see any fix (and also don't have your original files anymore since I pulled your new commits). |
Sorry, the diff before/after my rebase is here. I opted for clean commit messages. The bug fix I mentioned is this in I had originally not relized packets returned during encoding reported the row length in bytes as their width (1920 instead of 640 for a bgr8 test image) Thus: the failed test |
|
OK, looks good now. Thanks for contributing to this module! |
303ace0
into
ros-misc-utilities:master

We use H.264 compression for a RealSense D435 on a Jetson AGX Orin with
the hardware encoder from the guide. At 1920x1080 and 30 Hz, the
RealSense driver process used ~87% of one CPU core, even for a static
scene. We found the image went through YUYV -> RGB8 (RealSense driver) -> BGR8
(cv_bridge in the ffmpeg_encoder_decoder) -> YUV420P (swscale), i.e. YUV to RGB and
back.
This PR:
to bgr8 first (and without copying it, when possible),
input and also fixes the Decoder output for such streams,
Measured CPU of the RealSense driver process in % of one core:
Questions welcome.