Loading...

Type: Bug
Resolution: Unresolved
Priority: Undefined
Fix Version/s: None
Affects Version/s: rhel-10.1
Component/s: compiler-rt
Labels:
None

Regression:
No
Severity:
None

AssignedTeam:
rhel-pt-go

Story Points:
None
Blocked:
False
Ready:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Sprint:
None

Preliminary Testing:
None
Test Coverage:
None

ProdDocsReview-CCS:
Unspecified
ProdDocsReview-Dev:
Unspecified
ProdDocsReview-QE:
Unspecified

Experience:

PX Impact Score:
SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

Hello,

In systemd we've been hit by a very strange stack-buffer-underflow when running our tests with LLVM's AddressSanitizer (it doesn't happen with GCC's libasan). The issue is believed to be a false positive by upstream (and after debugging it on RHEL 10.1 for some time I'm inclined to this resolution as well), but it still does seem to pop up there from time to time [0][1]. And with current systemd codebase in RHEL 10 I can reproduce the issue very reliably on both CentOS Stream 10 and latest Fedora Rawhide.

After some fiddling around I found an easy reproducer that doesn't require running a full test with systemd-udevd. Following steps work both on C10S and Fedora Rawhide; the output here is from Rawhide:

# dnf -y builddep systemd
# dnf -y install clang llvm compiler-rt
# rpm -q clang compiler-rt
clang-21.1.2-1.fc44.x86_64
compiler-rt-21.1.2-1.fc44.x86_64
# git clone https://github.com/redhat-plumbers/systemd-rhel10
# cd systemd-rhel10
# CC=clang CXX=clang++ meson setup build-test -Db_sanitize=address,undefined -Db_lundef=false -Dc_args=-fno-sanitize=function --optimization=2
# ninja -C build-test udevadm

# build-test/udevadm test-builtin blkid /dev/vda
Trying to open "/etc/systemd/hwdb/hwdb.bin"...
Trying to open "/etc/udev/hwdb.bin"...
=== trie on-disk ===
tool version:          258
file size:        13227366 bytes
header size             80 bytes
strings            2768342 bytes
nodes             10458944 bytes
Loading kernel module index.
Loaded 'libkmod.so.2' via dlopen()
Found container virtualization none.
Using default interface naming scheme 'rhel-10.1'.
Parsed configuration file "/usr/lib/systemd/network/99-default.link"
Parsed configuration file "/usr/lib/systemd/network/98-default-mac-none.link"
Parsed configuration file "/usr/lib/systemd/network/80-vm-vt.link"
Parsed configuration file "/usr/lib/systemd/network/80-namespace-ns.link"
Parsed configuration file "/usr/lib/systemd/network/80-namespace-ns-tun.link"
Parsed configuration file "/usr/lib/systemd/network/80-container-vz.link"
Parsed configuration file "/usr/lib/systemd/network/80-container-ve.link"
Parsed configuration file "/usr/lib/systemd/network/80-container-vb.link"
Parsed configuration file "/usr/lib/systemd/network/80-6rd-tunnel.link"
Created link configuration context.
vda: Probe /dev/vda with raid and offset=0
ID_PART_TABLE_UUID=f9b7a20c-b0db-41b5-ab6f-ff2f16d5fc9a
ID_PART_TABLE_TYPE=gpt
=================================================================
==5049==ERROR: AddressSanitizer: stack-buffer-underflow on address 0x7ffcba686620 at pc 0x0000005d0e05 bp 0x7ffcba686350 sp 0x7ffcba686348
READ of size 8 at 0x7ffcba686620 thread T0
    #0 0x0000005d0e04 in sd_id128_in_setv /root/systemd-rhel10/build-test/../src/systemd/sd-id128.h:139:32
    #1 0x0000005d0e04 in sd_id128_in_set_sentinel /root/systemd-rhel10/build-test/../src/systemd/sd-id128.h:154:13
    #2 0x0000005d06cf in find_gpt_root /root/systemd-rhel10/build-test/../src/udev/udev-builtin-blkid.c:173:21
    #3 0x0000005cf191 in builtin_blkid /root/systemd-rhel10/build-test/../src/udev/udev-builtin-blkid.c:441:17
    #4 0x00000053bffa in udev_builtin_run /root/systemd-rhel10/build-test/../src/udev/udev-builtin.c:137:16
    #5 0x0000005303af in builtin_main /root/systemd-rhel10/build-test/../src/udev/udevadm-test-builtin.c:118:13
    #6 0x7fcd17552f70 in dispatch_verb /root/systemd-rhel10/build-test/../src/shared/verbs.c:126:16
    #7 0x00000053a2e3 in udevadm_main /root/systemd-rhel10/build-test/../src/udev/udevadm.c:118:16
    #8 0x00000053a2e3 in run /root/systemd-rhel10/build-test/../src/udev/udevadm.c:138:16
    #9 0x00000053a2e3 in main /root/systemd-rhel10/build-test/../src/udev/udevadm.c:141:1
    #10 0x7fcd16eef5b4 in __libc_start_call_main (/lib64/libc.so.6+0x35b4) (BuildId: b58b7a876869476292db2687fbb7aea5544c0782)
    #11 0x7fcd16eef667 in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x3667) (BuildId: b58b7a876869476292db2687fbb7aea5544c0782)
    #12 0x0000004385d4 in _start (/root/systemd-rhel10/build-test/udevadm+0x4385d4) (BuildId: 289af63522edaf418acd2e5071a1a33f26e5b9e4)

Address 0x7ffcba686620 is located in stack of thread T0 at offset 0 in frame
    #0 0x0000005cdc7f in builtin_blkid /root/systemd-rhel10/build-test/../src/udev/udev-builtin-blkid.c:321

  This frame has 23 object(s):
    [32, 264) 'info' (line 256)
    [336, 344) 'name' (line 257)
    [368, 416) '_zzq_args' (line 289)
    [448, 704) 's' (line 38)
    [768, 800) '.compoundliteral' (line 38)
    [832, 976) 'st' (line 215)
    [1040, 1048) 'devnode' (line 323)
    [1072, 1080) 'root_partition' (line 323)
    [1104, 1112) 'data' (line 323)
    [1136, 1144) 'name' (line 323)
    [1168, 1176) 'offset' (line 330)
    [1200, 1208) '_sysname' (line 343)
    [1232, 1240) '_sysname' (line 358)
    [1264, 1272) '_sysname' (line 372)
    [1296, 1304) '_sysname' (line 374)
    [1328, 1336) '_sysname' (line 395)
    [1360, 1368) '_sysname' (line 400)
    [1392, 1400) '_sysname' (line 408)
    [1424, 1432) '_sysname' (line 410)
    [1456, 1464) '_sysname' (line 414)
    [1488, 1496) '_sysname' (line 422)
    [1520, 1528) '_sysname' (line 450)
    [1552, 1809) 'encoded' (line 459)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-underflow /root/systemd-rhel10/build-test/../src/systemd/sd-id128.h:139:32 in sd_id128_in_setv
Shadow bytes around the buggy address:
  0x7ffcba686380: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7ffcba686400: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7ffcba686480: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7ffcba686500: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x7ffcba686580: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x7ffcba686600: 00 00 00 00[f1]f1 f1 f1 f8 f8 f8 f8 f8 f8 f8 f8
  0x7ffcba686680: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
  0x7ffcba686700: f8 f8 f8 f8 f8 f2 f2 f2 f2 f2 f2 f2 f2 f2 f8 f2
  0x7ffcba686780: f2 f2 f8 f8 f8 f8 f8 f8 f2 f2 f2 f2 f8 f8 f8 f8
  0x7ffcba686800: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8
  0x7ffcba686880: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f2 f2 f2 f2
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==5049==ABORTING

I tried to isolate the issue into an independent reproducer several times but none of them reproduced the issue even though some of them used pretty much the same code as is in systemd, so I guess the issue depends on the stack state that's present in systemd (or udevadm in this case).

This seems to happen only with optimizations enabled (I could reproduce this with -O{1,2,3}, -O{0,s,g} work fine).

In my debugging I noticed that in RHEL 10's systemd this started happening once we backported https://github.com/redhat-plumbers/systemd-rhel10/commit/fc5978e7c5913187fba87ae5d71da5de3064f10c, but this patch doesn't really change anything, it's just refactoring. And as mentioned before, in upstream the issue disappeared even with this patch after moving to a different LLVM version.

If there's anything else I could provide to get to the core of this, please let me know.

[0] https://github.com/systemd/systemd/commit/9670922d4480af35bf6ab20ef5de3a41b4d48c4d
[1] https://github.com/systemd/systemd/commit/97940e77a9e347085a0b174bafa90a2823f43657

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates

Hide