PR #20864 opened by thomasdullien
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20864
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/20864.patch
For transparency: I am experimenting with an AI-assisted patch process, where
the AI agent attempts to help
root-cause analyze a crash by means of reproducing the crash with ASAN, making
a recording with 'rr', and
documenting the analysis at a granular level with verbatim quotes from the 'rr'
trace. This root-cause analysis
is then iteratively reviewed (e.g. as a human reviewer I check it for accuracy
and plausibility) before a patch is
generated. The process generates a detailed analysis report, an 'rr' trace that
can be shared with other to help
with the verification, and a patch. Given that it is unlear how to best share
the 'rr' trace, I have only attached the
detailed root-cause analysis document.
This patch attempts to address https://trac.ffmpeg.org/ticket/11691
It should be noted that I am very much not an expert in the ffmpeg codebase, so
please bear with me if the proposed
patch isn't what is needed. I've re-run the test suite, and it looks very
plausible to me.
- Description -
Many YUV-to-RGB conversion functions in yuv2rgb.c process 2 scanlines
per iteration. When sws_scale() is called with insufficient source rows
available (e.g., srcSliceY=1 with only 2 total rows, leaving 1 row),
these functions would read beyond the allocated buffer bounds.
This occurred because the convert_unscaled path did not verify that at
least 2 rows were available before calling the conversion function. The
issue was particularly evident with formats like yuv422p when converting
to rgb4 or similar pixel formats.
Add bounds checking in scale_internal() before invoking convert_unscaled
to ensure at least 2 source rows remain after accounting for the slice
offset. If fewer than 2 rows are available, return 0 (no rows processed)
instead of attempting conversion, preventing the out-of-bounds access.
Fixes: https://trac.ffmpeg.org/ticket/11691
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude
---
From 6138573dd0364eee7051eaa1ba27cebee738bb5a Mon Sep 17 00:00:00 2001
From: "Thomas Dullien (AI assisted)"
Date: Sat, 8 Nov 2025 14:30:28 +0100
Subject: [PATCH] swscale: Fix heap-buffer-overflow in unscaled YUV-to-RGB
conversion
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Many YUV-to-RGB conversion functions in yuv2rgb.c process 2 scanlines
per iteration. When sws_scale() is called with insufficient source rows
available (e.g., srcSliceY=1 with only 2 total rows, leaving 1 row),
these functions would read beyond the allocated buffer bounds.
This occurred because the convert_unscaled path did not verify that at
least 2 rows were available before calling the conversion function. The
issue was particularly evident with formats like yuv422p when converting
to rgb4 or similar pixel formats.
Add bounds checking in scale_internal() before invoking convert_unscaled
to ensure at least 2 source rows remain after accounting for the slice
offset. If fewer than 2 rows are available, return 0 (no rows processed)
instead of attempting conversion, preventing the out-of-bounds access.
Fixes: https://trac.ffmpeg.org/ticket/11691
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude
---
libswscale/swscale.c | 18 ++
1 file changed, 18 insertions(+)
diff --git a/libswscale/swscale.c b/libswscale/swscale.c
index f4c7eccac4..7086b0f0e2 100644
--- a/libswscale/swscale.c
+++ b/libswscale/swscale.c
@@ -1140,6 +1140,23 @@ static int scale_internal(SwsContext *sws,
int offset = srcSliceY_internal;
int slice_h = srcSliceH;
+/* Many YUV-to-RGB conversion functions process 2 scanlines per
iteration
+ * and require at least 2 rows to be available in the source buffer.
+ * Check if we have sufficient rows after considering the offset. */
+int src_rows_remaining = sws->src_h - offset;
+if (slice_h > src_rows_remaining) {
+/* Clamp to available rows */
+slice_h = src_rows_remaining;
+}
+if (slice_h < 2) {
+/* Cannot safely process: conversion functions that process 2 rows
+ * per iteration would read beyond buffer bounds. Return 0 to
indicate
+ * no rows were processed. This is a safety check to prevent
+ * heap-buffer-overflow in yuv2rgb.c and similar conversion
functions. */
+ret = 0;
+goto unscaled_end;
+}
+
// for dst slice scaling, offset the pointers to match the unscaled API
if (scale_dst) {
av_assert0(offset == 0);
@@ -1160,6 +1177,7 @@ static int scale_internal(SwsContext *sws,
ret = c->convert_unscaled(c, src2, srcStride2, offset, slice_h,
