Bug#364024: gdImageAALine() can cause segfault

[Paul]

Package: libgd2-xpm
Version: 2.0.33-3
Severity: important

I found an off-by-one bug that repeatedly causes segfaults:

$ cat test_segfault.c
#include <gd.h>

int main()
{
   gdImagePtr im = gdImageCreateTrueColor(100,100);
   int colour = gdTrueColorAlpha(100,100,100,100);
   gdImageSetAntiAliased(im,colour);
   gdImageLine(im, 95, 100, 100, 98, gdAntiAliased);
   return 0;
}
$ gcc -o test_segfault test_segfault.c -lgd
$ ./test_segfault
Segmentation fault
$


I made a patch that solves the problem.  Its attached.


thanks,
Paul




-- System Information:
Debian Release: testing/unstable
  APT prefers testing
  APT policy: (650, 'testing'), (600, 'unstable')
Architecture: i386 (i686)
Shell:  /bin/sh linked to /bin/bash
Kernel: Linux 2.6.15.20060419
Locale: LANG=en_AU, LC_CTYPE=en_AU (charmap=ISO-8859-1)

Versions of packages libgd2-xpm depends on:
ii  libc6                     2.3.6-7        GNU C Library: Shared libraries
ii  libfontconfig1            2.3.2-5.1      generic font configuration library
ii  libfreetype6              2.1.10-1       FreeType 2 font engine, shared lib
ii  libjpeg62                 6b-12          The Independent JPEG Group's JPEG
ii  libpng12-0                1.2.8rel-5     PNG library - runtime
ii  libx11-6                  6.9.0.dfsg.1-6 X Window System protocol client li
ii  libxpm4                   6.9.0.dfsg.1-6 X pixmap library
ii  zlib1g                    1:1.2.3-11     compression library - runtime

libgd2-xpm recommends no packages.

-- no debconf information

--- gd.c        2006-04-21 10:58:02.000000000 +0800
+++ gd.c        2006-04-21 10:58:18.000000000 +0800
@@ -3095,7 +3095,9 @@
                /* TBB: set the last pixel for consistency (<=) */
                while ((x >> 16) <= x2) {
                        gdImageSetAAPixelColor(im, x >> 16, y >> 16, col, (y >> 
8) & 0xFF);
-                       gdImageSetAAPixelColor(im, x >> 16, (y >> 16) + 1,col, 
(~y >> 8) & 0xFF);
+         // the +1 can push it past the image bounds
+         if ((y >> 16) + 1 <= im->cy2)
+            gdImageSetAAPixelColor(im, x >> 16, (y >> 16) + 1,col, (~y >> 8) & 
0xFF);
                        x += (1 << 16);
                        y += inc;
                }
@@ -3116,6 +3118,9 @@
                /* TBB: set the last pixel for consistency (<=) */
                while ((y>>16) <= y2) {
                        gdImageSetAAPixelColor(im, x >> 16, y >> 16, col, (x >> 
8) & 0xFF);
+         // the +1 can push it past the image bounds
+         if ((x >> 16) + 1 <= im->cx2)
+            gdImageSetAAPixelColor(im, x >> 16, (y >> 16) + 1,col, (~y >> 8) & 
0xFF);
                        gdImageSetAAPixelColor(im, (x >> 16) + 1, (y >> 
16),col, (~x >> 8) & 0xFF);
                        x += inc;
                        y += (1<<16);