libwmf: memory allocation failure in wmf_malloc (api.c)

[Agostino Sarubbo <ago () gentoo org>]

Description:
libwmf is a library for reading vector images in Microsøft’s native Windøws 
Metafile Format (WMF) and for either (a) displaying them in, e.g., an X 
window; or (b) converting them to more standard/open file formats such as, 
e.g., the W3C’s XML-based Scaleable Vector Graphic (SVG) format.

A fuzzing through imagemagick revealed a memory allocation failure. It was 
first reported to imagemagick developers(to double-check) which stated that 
the issue is in libwmf.
Since the libwmf project is dead the issue has not been reported elsewhere.

The complete ASan output:

# identify $FILE
==25497==ERROR: AddressSanitizer failed to allocate 0xfe769000 (4269182976) 
bytes of LargeMmapAllocator (error code: 12)                                                                            
                                                                           
==25497==Process memory map follows:                                                                                    
                                                                                                                        
                               
[..cut here..]
==25497==End of process memory map.
==25497==AddressSanitizer CHECK failed: /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-
rt/lib/sanitizer_common/sanitizer_common.cc:183 "((0 && "unable to mmap")) != 
(0)" (0x0, 0x0)
    #0 0x4c9f9d in AsanCheckFailed /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_rtl.cc:67
    #1 0x4d0ad3 in __sanitizer::CheckFailed(char const*, int, char const*, 
unsigned long long, unsigned long long) /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-
rt/lib/sanitizer_common/sanitizer_common.cc:159
    #2 0x4d0cc1 in __sanitizer::ReportMmapFailureAndDie(unsigned long, char 
const*, char const*, int, bool) /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-
rt/lib/sanitizer_common/sanitizer_common.cc:183
    #3 0x4d9cfa in __sanitizer::MmapOrDie(unsigned long, char const*, bool) 
/var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-
rt/lib/sanitizer_common/sanitizer_posix.cc:122
    #4 0x42208f in 
__sanitizer::LargeMmapAllocator::Allocate(__sanitizer::AllocatorStats*, 
unsigned long, unsigned long) /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-
rt/lib/asan/../sanitizer_common/sanitizer_allocator.h:1033
    #5 0x42208f in 
__sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64<105553116266496ul, 
4398046511104ul, 0ul, __sanitizer::SizeClassMap, 
__asan::AsanMapUnmapCallback>, 
__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 
4398046511104ul, 0ul, __sanitizer::SizeClassMap, __asan::AsanMapUnmapCallback> 
> , __sanitizer::LargeMmapAllocator 
> ::Allocate(__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64<105553116266496ul, 
4398046511104ul, 0ul, __sanitizer::SizeClassMap, __asan::AsanMapUnmapCallback> 
> *, unsigned long, unsigned long, bool, bool) /var/tmp/portage/sys-
devel/llvm-3.8.1-r2/work/llvm-3.8.1.src/projects/compiler-
rt/lib/asan/../sanitizer_common/sanitizer_allocator.h:1302
    #6 0x42208f in __asan::Allocator::Allocate(unsigned long, unsigned long, 
__sanitizer::BufferedStackTrace*, __asan::AllocType, bool) 
/var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_allocator.cc:368
    #7 0x42208f in __asan::asan_malloc(unsigned long, 
__sanitizer::BufferedStackTrace*) /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_allocator.cc:718
    #8 0x4c0661 in malloc /var/tmp/portage/sys-devel/llvm-3.8.1-
r2/work/llvm-3.8.1.src/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:53
    #9 0x7f7173b4d337 in wmf_malloc /tmp/portage/media-libs/libwmf-0.2.8.4-
r6/work/libwmf-0.2.8.4/src/api.c:482
    #10 0x7f7173b5d2f8 in wmf_scan /tmp/portage/media-libs/libwmf-0.2.8.4-
r6/work/libwmf-0.2.8.4/src/player.c:143
    #11 0x7f7173d6dcf7 in ReadWMFImage /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/coders/wmf.c:2675:13
    #12 0x7f717fde7b12 in ReadImage /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/constitute.c:496:13
    #13 0x7f718057f406 in ReadStream /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/stream.c:1012:9
    #14 0x7f717fde65ca in PingImage /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/constitute.c:226:9
    #15 0x7f717fde6e25 in PingImages /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickCore/constitute.c:326:10
    #16 0x7f717f66c4c3 in IdentifyImageCommand /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickWand/identify.c:319:18
    #17 0x7f717f70226a in MagickCommandGenesis /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/MagickWand/mogrify.c:183:14
    #18 0x4f1fb5 in MagickMain /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/utilities/magick.c:145:10
    #19 0x4f1fb5 in main /tmp/portage/media-
gfx/imagemagick-7.0.3.0/work/ImageMagick-7.0.3-0/utilities/magick.c:176
    #20 0x7f717e5a661f in __libc_start_main /var/tmp/portage/sys-
libs/glibc-2.22-r4/work/glibc-2.22/csu/libc-start.c:289
    #21 0x419138 in _init (/usr/bin/magick+0x419138)

Affected version:
0.2.8.4

Fixed version:
N/A

Commit fix:
N/A

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
N/A

Timeline:
2016-09-14: bug discovered
2016-10-18: blog post about the issue

Note:
This bug was found with American Fuzzy Lop.

Permalink:
https://blogs.gentoo.org/ago/2016/10/18/libwmf-memory-allocation-failure-in-wmf_malloc-api-c