Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-61945

'go test -exec valgrind' crashes with SIGILL on s390x

Linking RHIVOS CVEs to...Migration: Automation ...SWIFT: POC ConversionSync from "Extern...XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Normal Normal
    • rhel-10.0
    • rhel-10.0
    • valgrind
    • None
    • valgrind-3.24.0-1.el10
    • No
    • Low
    • 2
    • rhel-pt-perf-tools
    • ssg_platform_tools
    • 12
    • 14
    • 3
    • False
    • False
    • Hide

      None

      Show
      None
    • No
    • Perf Debug Sprint 12, Perf Debug Sprint 13
    • Unspecified Release Note Type - Unknown
    • s390x
    • None

      One of our testcases uses valgrind to detect possible memleaks on golang crypto functions but several tests fail with vex s390->IR: unimplemented insn: B92E 0024 and SIGILL: illegal instruction when running under go test like below:

      [root@ibm-z-522 x509]# GOLANG_FIPS=1 go test crypto/tls -timeout 50m -count=1 -v -run ^TestBoringClientHello$ -exec 'valgrind --trace-children=yes --track-origins=yes --leak-check=full --fair-sched=try'
      ==300499== Memcheck, a memory error detector
      ==300499== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al.
      ==300499== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info
      ==300499== Command: /tmp/go-build2837420072/b001/tls.test -test.paniconexit0 -test.timeout=50m0s -test.count=1 -test.v=true -test.run=^TestBoringClientHello$
      ==300499== 
      ==300499== Invalid write of size 8
      ==300499==    at 0x10925F0: ??? (in /tmp/go-build2837420072/b001/tls.test)
      ==300499==    by 0x48A337B: (below main) (in /usr/lib64/libc.so.6)
      ==300499==  Address 0x1ffefffa88 is on thread 1's stack
      ==300499==  8 bytes below stack pointer
      ==300499== 
      ==300499== Invalid write of size 8
      ==300499==    at 0x106652C: ??? (in /tmp/go-build2837420072/b001/tls.test)
      ==300499==    by 0x10926A7: ??? (in /tmp/go-build2837420072/b001/tls.test)
      ==300499==  Address 0x1ffefffa30 is on thread 1's stack
      ==300499==  64 bytes below stack pointer
      (...)
      ==300499== Invalid write of size 8
      ==300499==    at 0x11FA5D2: ??? (in /tmp/go-build2837420072/b001/tls.test)
      ==300499==    by 0x11F9CE5: ??? (in /tmp/go-build2837420072/b001/tls.test)
      ==300499==  Address 0xc000059940 is in a rw- anonymous segment
      ==300499== 
      ==300499== 
      ==300499== More than 1000 different errors detected.  I'm not reporting any more.
      ==300499== Final error counts will be inaccurate.  Go fix your program!
      ==300499== Rerun with --error-limit=no to disable this cutoff.  Note
      ==300499== that errors may occur in your program without prior warning from
      ==300499== Valgrind, because errors are no longer being displayed.
      ==300499== 
      vex s390->IR: unimplemented insn: B92E 0024
      ==300499== valgrind: Unrecognised instruction at address 0x12a02e8.
      ==300499==    at 0x12A02E8: ??? (in /tmp/go-build2837420072/b001/tls.test)
      ==300499==    by 0x129FB05: ??? (in /tmp/go-build2837420072/b001/tls.test)
      ==300499== Your program just tried to execute an instruction that Valgrind
      ==300499== did not recognise.  There are two possible reasons for this.
      ==300499== 1. Your program has a bug and erroneously jumped to a non-code
      ==300499==    location.  If you are running Memcheck and you just saw a
      ==300499==    warning about a bad jump, it's probably your program's fault.
      ==300499== 2. The instruction is legitimate but Valgrind doesn't handle it,
      ==300499==    i.e. it's Valgrind's fault.  If you think this is the case or
      ==300499==    you are not sure, please let us know and we'll try to fix it.
      ==300499== Either way, Valgrind will now raise a SIGILL signal which will
      ==300499== probably kill your program.
      SIGILL: illegal instruction
      PC=0x12a02ec m=0 sigcode=1
      instruction bytes: 0x7 0xfe 0x0 0x0 0xa7 0x9 0x0 0x0 0x41 0x1f 0x0 0x8 0xb9 0x2f 0x0 0x24
      
      goroutine 1 gp=0xc0000021c0 m=0 mp=0x16b30a0 [running, locked to thread]:
      vendor/golang.org/x/sys/cpu.kmQuery()
      	/usr/lib/golang/src/vendor/golang.org/x/sys/cpu/cpu_s390x.s:22 +0xc fp=0xc000059d30 sp=0xc000059d30 pc=0x12a02ec
      vendor/golang.org/x/sys/cpu.doinit()
      	/usr/lib/golang/src/vendor/golang.org/x/sys/cpu/cpu_s390x.go:144 +0x56 fp=0xc000059e08 sp=0xc000059d30 pc=0x129fb06
      vendor/golang.org/x/sys/cpu.archInit()
      	/usr/lib/golang/src/vendor/golang.org/x/sys/cpu/cpu_linux.go:13 +0x42 fp=0xc000059e20 sp=0xc000059e08 pc=0x129f6b2
      vendor/golang.org/x/sys/cpu.init.0()
      	/usr/lib/golang/src/vendor/golang.org/x/sys/cpu/cpu.go:202 +0x24 fp=0xc000059e28 sp=0xc000059e20 pc=0x129ef64
      runtime.doInit1(0x1680850)
      	/usr/lib/golang/src/runtime/proc.go:7176 +0xfc fp=0xc000059f68 sp=0xc000059e28 pc=0x1063c1c
      runtime.doInit(...)
      	/usr/lib/golang/src/runtime/proc.go:7143
      runtime.main()
      	/usr/lib/golang/src/runtime/proc.go:253 +0x3e2 fp=0xc000059fd8 sp=0xc000059f68 pc=0x1051242
      runtime.goexit({})
      	/usr/lib/golang/src/runtime/asm_s390x.s:774 +0x2 fp=0xc000059fd8 sp=0xc000059fd8 pc=0x1094c12
      
      goroutine 2 gp=0xc000002c40 m=nil [force gc (idle)]:
      runtime.gopark(0x1451f38, 0x16b0d00, 0x11, 0xa, 0x1)
      	/usr/lib/golang/src/runtime/proc.go:402 +0x136 fp=0xc000044fb0 sp=0xc000044f98 pc=0x1051706
      runtime.goparkunlock(...)
      	/usr/lib/golang/src/runtime/proc.go:408
      runtime.forcegchelper()
      	/usr/lib/golang/src/runtime/proc.go:326 +0xd2 fp=0xc000044fd8 sp=0xc000044fb0 pc=0x10514d2
      runtime.goexit({})
      	/usr/lib/golang/src/runtime/asm_s390x.s:774 +0x2 fp=0xc000044fd8 sp=0xc000044fd8 pc=0x1094c12
      created by runtime.init.5 in goroutine 1
      	/usr/lib/golang/src/runtime/proc.go:314 +0x30
      
      goroutine 3 gp=0xc000003180 m=nil [GC sweep wait]:
      runtime.gopark(0x1451f38, 0x16b11c0, 0xc, 0x9, 0x1)
      	/usr/lib/golang/src/runtime/proc.go:402 +0x136 fp=0xc000045790 sp=0xc000045778 pc=0x1051706
      runtime.goparkunlock(...)
      	/usr/lib/golang/src/runtime/proc.go:408
      runtime.bgsweep(0xc000026070)
      	/usr/lib/golang/src/runtime/mgcsweep.go:278 +0xaa fp=0xc0000457c8 sp=0xc000045790 pc=0x1035e0a
      runtime.gcenable.gowrap1()
      	/usr/lib/golang/src/runtime/mgc.go:203 +0x5e fp=0xc0000457d8 sp=0xc0000457c8 pc=0x1027b7e
      runtime.goexit({})
      	/usr/lib/golang/src/runtime/asm_s390x.s:774 +0x2 fp=0xc0000457d8 sp=0xc0000457d8 pc=0x1094c12
      created by runtime.gcenable in goroutine 1
      	/usr/lib/golang/src/runtime/mgc.go:203 +0xa8
      
      goroutine 4 gp=0xc000003340 m=nil [GC scavenge wait]:
      runtime.gopark(0x1451f38, 0x16b1be0, 0xd, 0xa, 0x2)
      	/usr/lib/golang/src/runtime/proc.go:402 +0x136 fp=0xc000045f80 sp=0xc000045f68 pc=0x1051706
      runtime.goparkunlock(...)
      	/usr/lib/golang/src/runtime/proc.go:408
      runtime.(*scavengerState).park(0x16b1be0)
      	/usr/lib/golang/src/runtime/mgcscavenge.go:425 +0x72 fp=0xc000045fa8 sp=0xc000045f80 pc=0x1033092
      runtime.bgscavenge(0xc000026070)
      	/usr/lib/golang/src/runtime/mgcscavenge.go:653 +0x5a fp=0xc000045fc8 sp=0xc000045fa8 pc=0x103374a
      runtime.gcenable.gowrap2()
      	/usr/lib/golang/src/runtime/mgc.go:204 +0x5e fp=0xc000045fd8 sp=0xc000045fc8 pc=0x1027afe
      runtime.goexit({})
      	/usr/lib/golang/src/runtime/asm_s390x.s:774 +0x2 fp=0xc000045fd8 sp=0xc000045fd8 pc=0x1094c12
      created by runtime.gcenable in goroutine 1
      	/usr/lib/golang/src/runtime/mgc.go:204 +0x10e
      
      goroutine 5 gp=0xc000003c00 m=nil [finalizer wait]:
      runtime.gopark(0x1451c78, 0x1721078, 0x10, 0xa, 0x1)
      	/usr/lib/golang/src/runtime/proc.go:402 +0x136 fp=0xc000044718 sp=0xc000044700 pc=0x1051706
      runtime.runfinq()
      	/usr/lib/golang/src/runtime/mfinal.go:194 +0x13a fp=0xc0000447d8 sp=0xc000044718 pc=0x102660a
      runtime.goexit({})
      	/usr/lib/golang/src/runtime/asm_s390x.s:774 +0x2 fp=0xc0000447d8 sp=0xc0000447d8 pc=0x1094c12
      created by runtime.createfing in goroutine 1
      	/usr/lib/golang/src/runtime/mfinal.go:164 +0x64
      
      r0   0x0	r1   0xc000059d38
      r2   0x1	r3   0xc0000593a0
      r4   0xc000059db0	r5   0xc0000fa150
      r6   0x0	r7   0x1
      r8   0xfffffffffffff001	r9   0x13a9280
      r10  0x17225c6	r11  0x4a5e720
      r12  0xc000059c98	r13  0xc0000021c0
      r14  0x129fb06	r15  0xc000059d30
      pc   0x12a02ec	link 0x129fb06
      ==300499== 
      ==300499== HEAP SUMMARY:
      ==300499==     in use at exit: 172,985 bytes in 3,854 blocks
      ==300499==   total heap usage: 25,990 allocs, 22,136 frees, 2,551,258 bytes allocated
      ==300499== 
      ==300499== 1,440 bytes in 5 blocks are possibly lost in loss record 886 of 911
      ==300499==    at 0x484111A: calloc (vg_replace_malloc.c:1675)
      ==300499==    by 0x401237B: UnknownInlinedFun (rtld-malloc.h:44)
      ==300499==    by 0x401237B: allocate_dtv (dl-tls.c:370)
      ==300499==    by 0x4012E87: _dl_allocate_tls (dl-tls.c:629)
      ==300499==    by 0x491BA73: pthread_create@@GLIBC_2.34 (in /usr/lib64/libc.so.6)
      ==300499==    by 0x138968B: ??? (in /tmp/go-build2837420072/b001/tls.test)
      ==300499==    by 0x1389821: ??? (in /tmp/go-build2837420072/b001/tls.test)
      ==300499==    by 0x1094997: ??? (in /tmp/go-build2837420072/b001/tls.test)
      ==300499== 
      ==300499== LEAK SUMMARY:
      ==300499==    definitely lost: 0 bytes in 0 blocks
      ==300499==    indirectly lost: 0 bytes in 0 blocks
      ==300499==      possibly lost: 1,440 bytes in 5 blocks
      ==300499==    still reachable: 171,545 bytes in 3,849 blocks
      ==300499==         suppressed: 0 bytes in 0 blocks
      ==300499== Reachable blocks (those to which a pointer was found) are not shown.
      ==300499== To see them, rerun with: --leak-check=full --show-leak-kinds=all
      ==300499== 
      ==300499== For lists of detected and suppressed errors, rerun with: -s
      ==300499== ERROR SUMMARY: 6373 errors from 1001 contexts (suppressed: 0 from 0)
      FAIL	crypto/tls	10.944s
      FAIL
      

      This is also reproducible on previous versions of RHEL, without GOLANG_FIPS=1 and even when using go test ...-exec 'valgrind' (with no additional parameters). The same test passes on x86_64 and aarch64.

      Please provide the package NVR for which the bug is seen:

      openssl-3.2.2-12.el10.s390x
      golang-1.23.1-2.el10.s390x
      valgrind-3.23.0-4.el10.s390x

      Steps to reproduce

      1. # yum install -y golang golang-tests valgrind
      2. # GOLANG_FIPS=1 go test crypto/tls -timeout 50m -count=1 -v -run ^TestBoringClientHello$ -exec 'valgrind --trace-children=yes --track-origins=yes --leak-check=full --fair-sched=try'

      Expected results

      Test passes

      Actual results

      Test crashes with SIGILL

              rhn-engineering-mjw Mark Wielaard
              rhn-support-emachado Edjunior Machado
              Mark Wielaard Mark Wielaard
              Martin Cermak Martin Cermak
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: