stb_image_resize2: ASAN heap-buffer-overflow

[ien646]

Using ASAN, a heap-buffer-overflow is reported when using stbir_resize with STBIR_2CHANNEL. The issue only happens when using specific source-destination image dimensions.

The issue only happens when compiling with -mavx2. ASAN message shows _mm_loadu_ps trying to load past the heap red leftzone. Apparently decode_buffer_size in stbir__alloc_internal_mem_and_build_samplers is not being correctly calculated for this particular case.

I've attached a minimum reproducible example that triggers the issue.
example.zip

==161245==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x522000004470 at pc 0x5a1bedc22660 bp 0x7ffe1dbc5590 sp 0x7ffe1dbc5580
READ of size 16 at 0x522000004470 thread T0
    #0 0x5a1bedc2265f in _mm_loadu_ps /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/include/xmmintrin.h:940
    #1 0x5a1bedc2265f in stbir__horizontal_gather_2_channels_with_9_coeffs /home/ien/Projects/stbir_repro/stb_image_resize2.h:10341
    #2 0x5a1bedc89893 in stbir__resample_horizontal_gather /home/ien/Projects/stbir_repro/stb_image_resize2.h:6158
    #3 0x5a1bedc89e25 in stbir__resample_vertical_gather /home/ien/Projects/stbir_repro/stb_image_resize2.h:6194
    #4 0x5a1bedc8a8d5 in stbir__vertical_gather_loop /home/ien/Projects/stbir_repro/stb_image_resize2.h:6276
    #5 0x5a1bedc91c85 in stbir__perform_resize /home/ien/Projects/stbir_repro/stb_image_resize2.h:7262
    #6 0x5a1bedc96674 in stbir_resize_extended /home/ien/Projects/stbir_repro/stb_image_resize2.h:7886
    #7 0x5a1bedc978ea in stbir_resize /home/ien/Projects/stbir_repro/stb_image_resize2.h:8056
    #8 0x5a1bedc97bdc in main /home/ien/Projects/stbir_repro/main.c:21
    #9 0x78e191e34e07  (/usr/lib/libc.so.6+0x25e07) (BuildId: 98b3d8e0b8c534c769cb871c438b4f8f3a8e4bf3)
    #10 0x78e191e34ecb in __libc_start_main (/usr/lib/libc.so.6+0x25ecb) (BuildId: 98b3d8e0b8c534c769cb871c438b4f8f3a8e4bf3)
    #11 0x5a1bedba3394 in _start (/home/ien/Projects/stbir_repro/main+0xa394) (BuildId: 6d33a73a79f0debd118c6c9367ad17c7dd630781)

0x52200000447c is located 0 bytes after 4988-byte region [0x522000003100,0x52200000447c)
allocated by thread T0 here:
    #0 0x78e1920fd891 in malloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:69
    #1 0x5a1bedc90063 in stbir__alloc_internal_mem_and_build_samplers /home/ien/Projects/stbir_repro/stb_image_resize2.h:7074
    #2 0x5a1bedc9601e in stbir__perform_build /home/ien/Projects/stbir_repro/stb_image_resize2.h:7801
    #3 0x5a1bedc963e9 in stbir_build_samplers_with_splits /home/ien/Projects/stbir_repro/stb_image_resize2.h:7841
    #4 0x5a1bedc9640f in stbir_build_samplers /home/ien/Projects/stbir_repro/stb_image_resize2.h:7851
    #5 0x5a1bedc96567 in stbir_resize_extended /home/ien/Projects/stbir_repro/stb_image_resize2.h:7868
    #6 0x5a1bedc978ea in stbir_resize /home/ien/Projects/stbir_repro/stb_image_resize2.h:8056
    #7 0x5a1bedc97bdc in main /home/ien/Projects/stbir_repro/main.c:21
    #8 0x78e191e34e07  (/usr/lib/libc.so.6+0x25e07) (BuildId: 98b3d8e0b8c534c769cb871c438b4f8f3a8e4bf3)
    #9 0x78e191e34ecb in __libc_start_main (/usr/lib/libc.so.6+0x25ecb) (BuildId: 98b3d8e0b8c534c769cb871c438b4f8f3a8e4bf3)
    #10 0x5a1bedba3394 in _start (/home/ien/Projects/stbir_repro/main+0xa394) (BuildId: 6d33a73a79f0debd118c6c9367ad17c7dd630781)

SUMMARY: AddressSanitizer: heap-buffer-overflow /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1/include/xmmintrin.h:940 in _mm_loadu_ps
Shadow bytes around the buggy address:
  0x522000004180: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x522000004200: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x522000004280: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x522000004300: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x522000004380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x522000004400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00[00]04
  0x522000004480: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x522000004500: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x522000004580: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x522000004600: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x522000004680: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==161245==ABORTING

[jeffatrad]

Can you tell me the command line you use to build this test app?

[jeffatrad]

Wait, nevermind, got it to happen in clang.