Loading...

Type: Bug
Resolution: Unresolved
Priority: Critical
Fix Version/s: rhel-9.5
Affects Version/s: CentOS Stream 9
Component/s: runc
Labels:
None

Fixed in Build:
runc-1.1.13-4.el9
Regression:
None
Severity:
Important
sprint_count:
1

Pool Team:

sst_container_tools

Story Points:
3
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Sprint:
RUN 256

Preliminary Testing:
Pass
Errata Link:
https://errata.engineering.redhat.com/advisory/133457
Test Coverage:
None

Experience:
Architecture:

All

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

The current runc shipped by the appstream CS9 repository is built with go1.22.3 (Red Hat 1.22.3-2.el9), that is affected by https://github.com/golang-fips/go/pull/207.

When a pod is created with a readinessProbe in a non-FIPS OKD cluster running on nodes with CentOS Stream CoreOS 9, kubelet communicates with runc through cri-o via gRPC to run the readiness probe, but runc is unable to load and exec the probe into the pod's container, responding with

opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version

runc version:

runc 4:1.1.13-1.el9 appstream
runc version 1.1.13
spec: 1.0.2-dev
go: go1.22.3 (Red Hat 1.22.3-2.el9)
libseccomp: 2.5.2

The issue doesn't reproduce if replacing the runc binary with the one built for RHCOS9 on go1.21.11 (not affected by the bug mentioned above).

Example pod leading to the failure in an OKD cluster installed from payload

registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-05-045104

:

apiVersion: v1
kind: Pod
metadata:
  name: example
  labels:
    app: httpd
  namespace: aleskandro
spec:
  securityContext:
    runAsNonRoot: true
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: httpd
      image: 'image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest'
      ports:
        - containerPort: 8080
      readinessProbe:
          exec:
            command:
              - "bash"
              - "-c"
              - "echo hello"
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL

Journal from a node:

Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:  > containerID="23d4e68b03449fd367931fcabdd9cfd00922fa994a875961e87d6f2876910f5d" cmd=["test","-f","/data/statefulset-continue"]
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: E0704 16:17:18.099796    2277 prober.go:104] "Probe errored" err=<
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         rpc error: code = Unknown desc = command error: panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 1 gp=0xc0000061c0 m=0 mp=0x55bfad0e11e0 [running]:
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         panic({0x55bfaccf5900?, 0xc00002ee10?})
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/panic.go:779 +0x158 fp=0xc000175ce8 sp=0xc000175c38 pc=0x55bfac831598
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         crypto/internal/backend.init.0()
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/crypto/internal/backend/openssl.go:50 +0x26c fp=0xc000175e20 sp=0xc000175ce8 pc=0x55bfaca3e46c
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.doInit1(0x55bfad0c9310)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:7176 +0xea fp=0xc000175f50 sp=0xc000175e20 pc=0x55bfac84352a
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.doInit(...)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:7143
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.main()
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:253 +0x357 fp=0xc000175fe0 sp=0xc000175f50 pc=0x55bfac834c77
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000175fe8 sp=0xc000175fe0 pc=0x55bfac867ec1
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004afa8 sp=0xc00004af88 pc=0x55bfac834fee
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goparkunlock(...)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:408
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.forcegchelper()
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:326 +0xb8 fp=0xc00004afe0 sp=0xc00004afa8 pc=0x55bfac834e78
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004afe8 sp=0xc00004afe0 pc=0x55bfac867ec1
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         created by runtime.init.7 in goroutine 1
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:314 +0x1a
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004b780 sp=0xc00004b760 pc=0x55bfac834fee
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goparkunlock(...)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:408
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.bgsweep(0xc00002a070)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00004b7c8 sp=0xc00004b780 pc=0x55bfac820774
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gcenable.gowrap1()
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgc.go:203 +0x25 fp=0xc00004b7e0 sp=0xc00004b7c8 pc=0x55bfac8150a5
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004b7e8 sp=0xc00004b7e0 pc=0x55bfac867ec1
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         created by runtime.gcenable in goroutine 1
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgc.go:203 +0x66
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0xc00002a070?, 0x55bfacca5d68?, 0x1?, 0x0?, 0xc000007340?)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004bf78 sp=0xc00004bf58 pc=0x55bfac834fee
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goparkunlock(...)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:408
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.(*scavengerState).park(0x55bfad0e0700)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00004bfa8 sp=0xc00004bf78 pc=0x55bfac81e169
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.bgscavenge(0xc00002a070)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00004bfc8 sp=0xc00004bfa8 pc=0x55bfac81e6fc
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gcenable.gowrap2()
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgc.go:204 +0x25 fp=0xc00004bfe0 sp=0xc00004bfc8 pc=0x55bfac815045
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004bfe8 sp=0xc00004bfe0 pc=0x55bfac867ec1
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         created by runtime.gcenable in goroutine 1
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgc.go:204 +0xa5
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]:
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0xc00004a648?, 0x55bfac808745?, 0xa8?, 0x1?, 0xc0000061c0?)
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004a620 sp=0xc00004a600 pc=0x55bfac834fee
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.runfinq()
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mfinal.go:194 +0x107 fp=0xc00004a7e0 sp=0xc00004a620 pc=0x55bfac8140e7
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004a7e8 sp=0xc00004a7e0 pc=0x55bfac867ec1
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         created by runtime.createfing in goroutine 1
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mfinal.go:164 +0x3d
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         time="2024-07-04T16:17:18Z" level=error msg="exec failed: unable to start container process: error writing config to pipe: write init-p: broken pipe"
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         , stdout: , stderr: , exit code -1
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:  > probeType="Readiness" pod="e2e-statefulset-1449/ss-0" podUID="1a76f966-e99a-4edc-a1cf-fcb31f04dfe6" containerName="webserver"
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: E0704 16:17:19.055154    2277 remote_runtime.go:496] "ExecSync cmd from runtime service failed" err=<
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         rpc error: code = Unknown desc = command error: panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 1 gp=0xc0000061c0 m=0 mp=0x55ff7183a1e0 [running]:
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         panic({0x55ff7144e900?, 0xc000098dd0?})
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/panic.go:779 +0x158 fp=0xc00019dce8 sp=0xc00019dc38 pc=0x55ff70f8a598
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         crypto/internal/backend.init.0()
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/crypto/internal/backend/openssl.go:50 +0x26c fp=0xc00019de20 sp=0xc00019dce8 pc=0x55ff7119746c
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.doInit1(0x55ff71822310)
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:7176 +0xea fp=0xc00019df50 sp=0xc00019de20 pc=0x55ff70f9c52a
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.doInit(...)
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:7143
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.main()
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:253 +0x357 fp=0xc00019dfe0 sp=0xc00019df50 pc=0x55ff70f8dc77
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00019dfe8 sp=0xc00019dfe0 pc=0x55ff70fc0ec1
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004afa8 sp=0xc00004af88 pc=0x55ff70f8dfee
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goparkunlock(...)
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:408
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.forcegchelper()
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:326 +0xb8 fp=0xc00004afe0 sp=0xc00004afa8 pc=0x55ff70f8de78
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004afe8 sp=0xc00004afe0 pc=0x55ff70fc0ec1
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         created by runtime.init.7 in goroutine 1
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:314 +0x1a
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004b780 sp=0xc00004b760 pc=0x55ff70f8dfee
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goparkunlock(...)
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:408
Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.bgsweep(0xc0000...

links to

RHBA-2024:133457 runc bug fix and enhancement update

Details

Description

Attachments

Issue Links

Activity

People

Dates