RFA: libiberty: cope with integer overflow in _objalloc_alloc

[Nick Clifton]

Hi DJ, Hi Ian,

  The _objalloc_alloc() function is currently vulnerable to an integer
  overflow if it is passed a negative length.  For example if called
  with len = -3 and assuming that OBJALLOC_ALIGN is 4 then:

    line 122:  len = (len + OBJALLOC_ALIGN - 1) &~ (OBJALLOC_ALIGN - 1);

  So len = (-3 + 3) & ~ 3 = 0, and then the function returns a pointer
  that will be reused the next time _objalloc_alloc is called.

  Or suppose that len = -4, then:

    line 122:  len = (len + OBJALLOC_ALIGN - 1) &~ (OBJALLOC_ALIGN - 1);

  Which gives len = (-4 + 3) & ~3 = -4 and then:

    line 136:  ret = (char *) malloc (CHUNK_HEADER_SIZE + len);

  So now the function returns a pointer to a memory block that is not
  even big enough to contain the chunk header.

  The proposed patch should take care of both of these scenarios. OK
  to apply ?

Cheers
  Nick

libiberty/ChangeLog
2012-08-31  Jan Lieskovsky <jlies...@redhat.com>
            Nick Clifton  <ni...@redhat.com>

        * objalloc.c (_objalloc_alloc): Align length before testing for
        zero length.  Refuse to allocate when length is so big that
        integer overflow would occur during internal calculations.

Index: libiberty/objalloc.c
===================================================================
RCS file: /cvs/src/src/libiberty/objalloc.c,v
retrieving revision 1.8
diff -u -3 -p -r1.8 objalloc.c
--- libiberty/objalloc.c        22 Jul 2005 03:26:05 -0000      1.8
+++ libiberty/objalloc.c        31 Aug 2012 10:28:33 -0000
@@ -114,12 +114,12 @@ objalloc_create (void)
 PTR
 _objalloc_alloc (struct objalloc *o, unsigned long len)
 {
+  len = (len + OBJALLOC_ALIGN - 1) &~ (OBJALLOC_ALIGN - 1);
+
   /* We avoid confusion from zero sized objects by always allocating
-     at least 1 byte.  */
+     at least OBJALLOC_ALIGN bytes.  */
   if (len == 0)
-    len = 1;
-
-  len = (len + OBJALLOC_ALIGN - 1) &~ (OBJALLOC_ALIGN - 1);
+    len = OBJALLOC_ALIGN;
 
   if (len <= o->current_space)
     {
@@ -133,6 +133,10 @@ _objalloc_alloc (struct objalloc *o, uns
       char *ret;
       struct objalloc_chunk *chunk;
 
+      /* Protect against integer overflow.  */
+      if (len + CHUNK_HEADER_SIZE < len)
+        return NULL;
+
       ret = (char *) malloc (CHUNK_HEADER_SIZE + len);
       if (ret == NULL)
        return NULL;