openjpeg: memory allocation failure in opj_aligned_alloc_n (opj_malloc.c)

["Agostino Sarubbo" <ago () gentoo org>]

Description:
openjpeg is an open-source JPEG 2000 library.

The complete ASan output of the issue:

# opj_compress -n 1 -i $FILE -o null.j2c
==78690==ERROR: AddressSanitizer failed to allocate 0x5ea7983000 (406538694656) bytes of LargeMmapAllocator (error 
code: 12)
==78690==Process memory map follows:
        [..cut here..]
==78690==End of process memory map.
==78690==AddressSanitizer CHECK failed: 
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_common.cc:120
 "((0 && "unable to mmap")) != (0)" 
(0x0, 0x0)
    #0 0x4db60f in AsanCheckFailed 
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_rtl.cc:69
    #1 0x4f6375 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) 
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_termination.cc:79
    #2 0x4e59a2 in __sanitizer::ReportMmapFailureAndDie(unsigned long, char const*, char const*, int, bool) 
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_common.cc:120
    #3 0x4ef2a5 in __sanitizer::MmapOrDie(unsigned long, char const*, bool) 
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/sanitizer_common/sanitizer_posix.cc:132
    #4 0x426caa in __sanitizer::LargeMmapAllocator::Allocate(__sanitizer::AllocatorStats*, unsigned long, unsigned 
long) 
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/../sanitizer_common/sanitizer_allocator_secondary.h:41
    #5 0x426caa in __sanitizer::CombinedAllocator<__sanitizer::SizeClassAllocator64, 
__sanitizer::SizeClassAllocatorLocalcache<__sanitizer::SizeClassAllocator64 >, __sanitizer::LargeMmapAllocator 
> ::Allocate(__sanitizer::SizeClassAllocatorLocalCache<__sanitizer::SizeClassAllocator64 >*, unsigned long, unsigned 
long, bool, bool) 
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/../sanitizer_common/sanitizer_allocator_combined.h:70
    #6 0x426caa in __asan::Allocator::Allocate(unsigned long, unsigned long, __sanitizer::BufferedStackTrace*, 
__asan::AllocType, bool) 
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_allocator.cc:407
    #7 0x42138d in __asan::asan_posix_memalign(void**, unsigned long, unsigned long, __sanitizer::BufferedStackTrace*) 
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_allocator.cc:815
    #8 0x4d206d in __interceptor_posix_memalign 
/var/tmp/portage/sys-libs/compiler-rt-sanitizers-4.0.1/work/compiler-rt-4.0.1.src/lib/asan/asan_malloc_linux.cc:144
    #9 0x7f2627d95aa4 in opj_aligned_alloc_n 
/var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/opj_malloc.c:61:9
    #10 0x7f2627d95aa4 in opj_aligned_malloc 
/var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/opj_malloc.c:209
    #11 0x7f2627c79d09 in opj_image_create 
/var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/lib/openjp2/image.c:77:39
    #12 0x53437b in bmptoimage 
/var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/bin/jp2/convertbmp.c:768:13
    #13 0x50b635 in main 
/var/tmp/portage/media-libs/openjpeg-2.2.0/work/openjpeg-2.2.0/src/bin/jp2/opj_compress.c:1844:21
    #14 0x7f2626681680 in __libc_start_main 
/var/tmp/portage/sys-libs/glibc-2.23-r4/work/glibc-2.23/csu/../csu/libc-start.c:289
    #15 0x41bc78 in _start (/usr/bin/opj_compress+0x41bc78)

Affected version:
2.2.0

Fixed version:
N/A

Commit fix:
https://github.com/uclouvain/openjpeg/commit/baf0c1ad4572daa89caa3b12985bdd93530f0dd7

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
CVE-2017-12982

Reproducer:
https://github.com/asarubbo/poc/blob/master/00315-openjpeg-memallocfailure-opj_aligned_alloc_n

Timeline:
2017-08-14: bug discovered and reported to upstream
2017-08-14: blog post about the issue
2017-08-21: CVE assigned

Note:
This bug was found with American Fuzzy Lop.
This bug was identified with bare metal servers donated by Packet. This work is also supported by the Core 
Infrastructure Initiative.

Permalink:
https://blogs.gentoo.org/ago/2017/08/14/openjpeg-memory-allocation-failure-in-opj_aligned_alloc_n-opj_malloc-c/

--
Agostino Sarubbo
Gentoo Linux Developer