-
Bug
-
Resolution: Done-Errata
-
Critical
-
CentOS Stream 9
-
None
-
runc-1.1.13-4.el9
-
None
-
Important
-
1
-
rhel-sst-container-tools
-
3
-
False
-
-
None
-
RUN 256
-
-
All
-
None
The current runc shipped by the appstream CS9 repository is built with go1.22.3 (Red Hat 1.22.3-2.el9), that is affected by https://github.com/golang-fips/go/pull/207.
When a pod is created with a readinessProbe in a non-FIPS OKD cluster running on nodes with CentOS Stream CoreOS 9, kubelet communicates with runc through cri-o via gRPC to run the readiness probe, but runc is unable to load and exec the probe into the pod's container, responding with
opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version
runc version:
runc 4:1.1.13-1.el9 appstream runc version 1.1.13 spec: 1.0.2-dev go: go1.22.3 (Red Hat 1.22.3-2.el9) libseccomp: 2.5.2
The issue doesn't reproduce if replacing the runc binary with the one built for RHCOS9 on go1.21.11 (not affected by the bug mentioned above).
Example pod leading to the failure in an OKD cluster installed from payload
registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-05-045104
:
apiVersion: v1 kind: Pod metadata: name: example labels: app: httpd namespace: aleskandro spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault containers: - name: httpd image: 'image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest' ports: - containerPort: 8080 readinessProbe: exec: command: - "bash" - "-c" - "echo hello" securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL
Journal from a node:
Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: > containerID="23d4e68b03449fd367931fcabdd9cfd00922fa994a875961e87d6f2876910f5d" cmd=["test","-f","/data/statefulset-continue"] Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: E0704 16:17:18.099796 2277 prober.go:104] "Probe errored" err=< Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: rpc error: code = Unknown desc = command error: panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: goroutine 1 gp=0xc0000061c0 m=0 mp=0x55bfad0e11e0 [running]: Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: panic({0x55bfaccf5900?, 0xc00002ee10?}) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/panic.go:779 +0x158 fp=0xc000175ce8 sp=0xc000175c38 pc=0x55bfac831598 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: crypto/internal/backend.init.0() Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/crypto/internal/backend/openssl.go:50 +0x26c fp=0xc000175e20 sp=0xc000175ce8 pc=0x55bfaca3e46c Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.doInit1(0x55bfad0c9310) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:7176 +0xea fp=0xc000175f50 sp=0xc000175e20 pc=0x55bfac84352a Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.doInit(...) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:7143 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.main() Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:253 +0x357 fp=0xc000175fe0 sp=0xc000175f50 pc=0x55bfac834c77 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.goexit({}) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000175fe8 sp=0xc000175fe0 pc=0x55bfac867ec1 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]: Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004afa8 sp=0xc00004af88 pc=0x55bfac834fee Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.goparkunlock(...) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:408 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.forcegchelper() Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:326 +0xb8 fp=0xc00004afe0 sp=0xc00004afa8 pc=0x55bfac834e78 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.goexit({}) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004afe8 sp=0xc00004afe0 pc=0x55bfac867ec1 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: created by runtime.init.7 in goroutine 1 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:314 +0x1a Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]: Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004b780 sp=0xc00004b760 pc=0x55bfac834fee Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.goparkunlock(...) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:408 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.bgsweep(0xc00002a070) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00004b7c8 sp=0xc00004b780 pc=0x55bfac820774 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.gcenable.gowrap1() Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/mgc.go:203 +0x25 fp=0xc00004b7e0 sp=0xc00004b7c8 pc=0x55bfac8150a5 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.goexit({}) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004b7e8 sp=0xc00004b7e0 pc=0x55bfac867ec1 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: created by runtime.gcenable in goroutine 1 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/mgc.go:203 +0x66 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]: Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.gopark(0xc00002a070?, 0x55bfacca5d68?, 0x1?, 0x0?, 0xc000007340?) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004bf78 sp=0xc00004bf58 pc=0x55bfac834fee Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.goparkunlock(...) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:408 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.(*scavengerState).park(0x55bfad0e0700) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00004bfa8 sp=0xc00004bf78 pc=0x55bfac81e169 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.bgscavenge(0xc00002a070) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00004bfc8 sp=0xc00004bfa8 pc=0x55bfac81e6fc Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.gcenable.gowrap2() Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/mgc.go:204 +0x25 fp=0xc00004bfe0 sp=0xc00004bfc8 pc=0x55bfac815045 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.goexit({}) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004bfe8 sp=0xc00004bfe0 pc=0x55bfac867ec1 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: created by runtime.gcenable in goroutine 1 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/mgc.go:204 +0xa5 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]: Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.gopark(0xc00004a648?, 0x55bfac808745?, 0xa8?, 0x1?, 0xc0000061c0?) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004a620 sp=0xc00004a600 pc=0x55bfac834fee Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.runfinq() Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/mfinal.go:194 +0x107 fp=0xc00004a7e0 sp=0xc00004a620 pc=0x55bfac8140e7 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: runtime.goexit({}) Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004a7e8 sp=0xc00004a7e0 pc=0x55bfac867ec1 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: created by runtime.createfing in goroutine 1 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/mfinal.go:164 +0x3d Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: time="2024-07-04T16:17:18Z" level=error msg="exec failed: unable to start container process: error writing config to pipe: write init-p: broken pipe" Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: , stdout: , stderr: , exit code -1 Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: > probeType="Readiness" pod="e2e-statefulset-1449/ss-0" podUID="1a76f966-e99a-4edc-a1cf-fcb31f04dfe6" containerName="webserver" Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: E0704 16:17:19.055154 2277 remote_runtime.go:496] "ExecSync cmd from runtime service failed" err=< Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: rpc error: code = Unknown desc = command error: panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: goroutine 1 gp=0xc0000061c0 m=0 mp=0x55ff7183a1e0 [running]: Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: panic({0x55ff7144e900?, 0xc000098dd0?}) Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/panic.go:779 +0x158 fp=0xc00019dce8 sp=0xc00019dc38 pc=0x55ff70f8a598 Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: crypto/internal/backend.init.0() Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/crypto/internal/backend/openssl.go:50 +0x26c fp=0xc00019de20 sp=0xc00019dce8 pc=0x55ff7119746c Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.doInit1(0x55ff71822310) Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:7176 +0xea fp=0xc00019df50 sp=0xc00019de20 pc=0x55ff70f9c52a Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.doInit(...) Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:7143 Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.main() Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:253 +0x357 fp=0xc00019dfe0 sp=0xc00019df50 pc=0x55ff70f8dc77 Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.goexit({}) Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00019dfe8 sp=0xc00019dfe0 pc=0x55ff70fc0ec1 Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]: Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004afa8 sp=0xc00004af88 pc=0x55ff70f8dfee Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.goparkunlock(...) Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:408 Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.forcegchelper() Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:326 +0xb8 fp=0xc00004afe0 sp=0xc00004afa8 pc=0x55ff70f8de78 Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.goexit({}) Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004afe8 sp=0xc00004afe0 pc=0x55ff70fc0ec1 Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: created by runtime.init.7 in goroutine 1 Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:314 +0x1a Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]: Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?) Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004b780 sp=0xc00004b760 pc=0x55ff70f8dfee Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.goparkunlock(...) Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: /usr/lib/golang/src/runtime/proc.go:408 Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: runtime.bgsweep(0xc0000...
- links to
-
RHBA-2024:133457 runc bug fix and enhancement update