imagemagick: memory allocate failure in AcquireQuantumPixels (quantum.c)

[Agostino Sarubbo <ago () gentoo org>]

Description:
imagemagick is a software suite to create, edit, compose, or convert bitmap 
images.

A fuzzing with the upstream security policy enabled revealed a memory allocate 
failure.

The complete ASan output:

# identify $FILE
==25084==WARNING: AddressSanitizer failed to allocate 0x46bf39483ac bytes                                               
                                                                                                                        
                               
==25084==AddressSanitizer's allocator is terminating the process instead of 
returning 0                                                                                                             
                                                                           
==25084==If you don't like this behavior set allocator_may_return_null=1                                                
                                                                                                                        
                               
==25084==AddressSanitizer CHECK failed: /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-
rt/lib/sanitizer_common/sanitizer_allocator.cc:147 "((0)) != (0)" (0x0, 0x0)                                            
                                
    #0 0x4c9f9d in AsanCheckFailed /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_rtl.cc:67                                                     
                                                                              
    #1 0x4d0ad3 in __sanitizer::CheckFailed(char const*, int, char const*, 
unsigned long long, unsigned long long) /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-
rt/lib/sanitizer_common/sanitizer_common.cc:159                              
    #2 0x4ce826 in __sanitizer::ReportAllocatorCannotReturnNull() 
/var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-
rt/lib/sanitizer_common/sanitizer_allocator.cc:147                                                                      
      
    #3 0x421bfc in 
__sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<105553116266496ul, 
4398046511104ul, 0ul, __sanitizer::SizeClassMap, 
__asan::AsanMapUnmapCallback>, 
__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 
4398046511104ul, 0ul, __sanitizer::SizeClassMap, __asan::AsanMapUnmapCallback> 
> , __sanitizer::LargeMmapAllocator >::ReturnNullOrDie() /var/tmp/portage/sys-
devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-
rt/lib/asan/../sanitizer_common/sanitizer_allocator.h:1317                                                              
                                                                                                                        
             
    #4 0x421bfc in __asan::Allocator::Allocate(unsigned long, unsigned long, 
__sanitizer::BufferedStackTrace*, __asan::AllocType, bool) 
/var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_allocator.cc:359                       
    #5 0x421bfc in __asan::asan_malloc(unsigned long, 
__sanitizer::BufferedStackTrace*) /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_allocator.cc:718                                              
                         
    #6 0x4c0661 in malloc /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:53                                            
                                                                                       
    #7 0x7f76c7533ff4 in AcquireQuantumPixels /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/quantum.c:175:47                                            
                                                                                      
    #8 0x7f76c7533ff4 in SetQuantumDepth /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/quantum.c:693                                               
                                                                                           
    #9 0x7f76c7532676 in AcquireQuantumInfo /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/quantum.c:125:10                                            
                                                                                        
    #10 0x7f76baf3607e in ReadTIFFImage /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/coders/tiff.c:1431:18                                                  
                                                                                            
    #11 0x7f76c7067b12 in ReadImage /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/constitute.c:496:13
    #12 0x7f76c77ff406 in ReadStream /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/stream.c:1012:9
    #13 0x7f76c70665ca in PingImage /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/constitute.c:226:9
    #14 0x7f76c7066e25 in PingImages /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/constitute.c:326:10
    #15 0x7f76c68ec4c3 in IdentifyImageCommand /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickWand/identify.c:319:18
    #16 0x7f76c698226a in MagickCommandGenesis /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickWand/mogrify.c:183:14
    #17 0x4f1fb5 in MagickMain /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/utilities/magick.c:145:10
    #18 0x4f1fb5 in main /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/utilities/magick.c:176
    #19 0x7f76c582661f in __libc_start_main /var/tmp/portage/sys-
libs/glibc-2.22-r4/work/glibc-2.22/csu/libc-start.c:289
    #20 0x419138 in _init (/usr/bin/magick+0x419138)

Affected version:
7.0.3.0

Fixed version:
7.0.3.1

Commit fix:
https://github.com/ImageMagick/ImageMagick/commit/6e48aa92ff4e6e95424300ecd52a9ea453c19c60

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
N/A

Timeline:
2016-09-14: bug discovered
2016-09-14: bug reported to upstream
2016-09-16: upstream released a patch
2016-09-21: upstream released 7.0.3.1
2016-10-07: blog post about the issue

Note:
This bug was found with American Fuzzy Lop.

Permalink:
https://blogs.gentoo.org/ago/2016/10/07/imagemagick-memory-allocate-failure-in-acquirequantumpixels-quantum-c/