Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-46380

runc 1.1.13 in no FIPS OKD environments running on CentOS Stream CoreOS leads to opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Critical Critical
    • rhel-9.5
    • CentOS Stream 9
    • runc
    • None
    • runc-1.1.13-4.el9
    • None
    • Important
    • 1
    • rhel-sst-container-tools
    • 3
    • False
    • Hide

      None

      Show
      None
    • None
    • RUN 256
    • All
    • None

      The current runc shipped by the appstream CS9 repository is built with go1.22.3 (Red Hat 1.22.3-2.el9), that is affected by https://github.com/golang-fips/go/pull/207.

      When a pod is created with a readinessProbe in a non-FIPS OKD cluster running on nodes with CentOS Stream CoreOS 9, kubelet communicates with runc through cri-o via gRPC to run the readiness probe, but runc is unable to load and exec the probe into the pod's container, responding with

      opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version

       

      runc version:

      runc 4:1.1.13-1.el9 appstream
      runc version 1.1.13
      spec: 1.0.2-dev
      go: go1.22.3 (Red Hat 1.22.3-2.el9)
      libseccomp: 2.5.2
      

       

      The issue doesn't reproduce if replacing the runc binary with the one built for RHCOS9 on go1.21.11 (not affected by the bug mentioned above).

       

      Example pod leading to the failure in an OKD cluster installed from payload

      registry.ci.openshift.org/origin/release-scos:4.16.0-0.okd-scos-2024-07-05-045104
      

      :

      apiVersion: v1
      kind: Pod
      metadata:
        name: example
        labels:
          app: httpd
        namespace: aleskandro
      spec:
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
          - name: httpd
            image: 'image-registry.openshift-image-registry.svc:5000/openshift/httpd:latest'
            ports:
              - containerPort: 8080
            readinessProbe:
                exec:
                  command:
                    - "bash"
                    - "-c"
                    - "echo hello"
            securityContext:
              allowPrivilegeEscalation: false
              capabilities:
                drop:
                  - ALL
      

      Journal from a node:

      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:  > containerID="23d4e68b03449fd367931fcabdd9cfd00922fa994a875961e87d6f2876910f5d" cmd=["test","-f","/data/statefulset-continue"]
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]: E0704 16:17:18.099796    2277 prober.go:104] "Probe errored" err=<
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         rpc error: code = Unknown desc = command error: panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 1 gp=0xc0000061c0 m=0 mp=0x55bfad0e11e0 [running]:
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         panic({0x55bfaccf5900?, 0xc00002ee10?})
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/panic.go:779 +0x158 fp=0xc000175ce8 sp=0xc000175c38 pc=0x55bfac831598
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         crypto/internal/backend.init.0()
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/crypto/internal/backend/openssl.go:50 +0x26c fp=0xc000175e20 sp=0xc000175ce8 pc=0x55bfaca3e46c
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.doInit1(0x55bfad0c9310)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:7176 +0xea fp=0xc000175f50 sp=0xc000175e20 pc=0x55bfac84352a
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.doInit(...)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:7143
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.main()
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:253 +0x357 fp=0xc000175fe0 sp=0xc000175f50 pc=0x55bfac834c77
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc000175fe8 sp=0xc000175fe0 pc=0x55bfac867ec1
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004afa8 sp=0xc00004af88 pc=0x55bfac834fee
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goparkunlock(...)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:408
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.forcegchelper()
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:326 +0xb8 fp=0xc00004afe0 sp=0xc00004afa8 pc=0x55bfac834e78
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004afe8 sp=0xc00004afe0 pc=0x55bfac867ec1
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         created by runtime.init.7 in goroutine 1
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:314 +0x1a
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004b780 sp=0xc00004b760 pc=0x55bfac834fee
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goparkunlock(...)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:408
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.bgsweep(0xc00002a070)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgcsweep.go:278 +0x94 fp=0xc00004b7c8 sp=0xc00004b780 pc=0x55bfac820774
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gcenable.gowrap1()
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgc.go:203 +0x25 fp=0xc00004b7e0 sp=0xc00004b7c8 pc=0x55bfac8150a5
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004b7e8 sp=0xc00004b7e0 pc=0x55bfac867ec1
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         created by runtime.gcenable in goroutine 1
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgc.go:203 +0x66
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 4 gp=0xc000007340 m=nil [GC scavenge wait]:
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0xc00002a070?, 0x55bfacca5d68?, 0x1?, 0x0?, 0xc000007340?)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004bf78 sp=0xc00004bf58 pc=0x55bfac834fee
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goparkunlock(...)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:408
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.(*scavengerState).park(0x55bfad0e0700)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgcscavenge.go:425 +0x49 fp=0xc00004bfa8 sp=0xc00004bf78 pc=0x55bfac81e169
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.bgscavenge(0xc00002a070)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgcscavenge.go:653 +0x3c fp=0xc00004bfc8 sp=0xc00004bfa8 pc=0x55bfac81e6fc
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gcenable.gowrap2()
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgc.go:204 +0x25 fp=0xc00004bfe0 sp=0xc00004bfc8 pc=0x55bfac815045
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004bfe8 sp=0xc00004bfe0 pc=0x55bfac867ec1
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         created by runtime.gcenable in goroutine 1
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mgc.go:204 +0xa5
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 5 gp=0xc000007c00 m=nil [finalizer wait]:
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0xc00004a648?, 0x55bfac808745?, 0xa8?, 0x1?, 0xc0000061c0?)
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004a620 sp=0xc00004a600 pc=0x55bfac834fee
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.runfinq()
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mfinal.go:194 +0x107 fp=0xc00004a7e0 sp=0xc00004a620 pc=0x55bfac8140e7
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004a7e8 sp=0xc00004a7e0 pc=0x55bfac867ec1
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         created by runtime.createfing in goroutine 1
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/mfinal.go:164 +0x3d
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         time="2024-07-04T16:17:18Z" level=error msg="exec failed: unable to start container process: error writing config to pipe: write init-p: broken pipe"
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:         , stdout: , stderr: , exit code -1
      Jul 04 16:17:18 ip-10-0-57-23 kubenswrapper[2277]:  > probeType="Readiness" pod="e2e-statefulset-1449/ss-0" podUID="1a76f966-e99a-4edc-a1cf-fcb31f04dfe6" containerName="webserver"
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]: E0704 16:17:19.055154    2277 remote_runtime.go:496] "ExecSync cmd from runtime service failed" err=<
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         rpc error: code = Unknown desc = command error: panic: opensslcrypto: can't initialize OpenSSL : openssl: can't retrieve OpenSSL version
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 1 gp=0xc0000061c0 m=0 mp=0x55ff7183a1e0 [running]:
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         panic({0x55ff7144e900?, 0xc000098dd0?})
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/panic.go:779 +0x158 fp=0xc00019dce8 sp=0xc00019dc38 pc=0x55ff70f8a598
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         crypto/internal/backend.init.0()
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/crypto/internal/backend/openssl.go:50 +0x26c fp=0xc00019de20 sp=0xc00019dce8 pc=0x55ff7119746c
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.doInit1(0x55ff71822310)
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:7176 +0xea fp=0xc00019df50 sp=0xc00019de20 pc=0x55ff70f9c52a
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.doInit(...)
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:7143
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.main()
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:253 +0x357 fp=0xc00019dfe0 sp=0xc00019df50 pc=0x55ff70f8dc77
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00019dfe8 sp=0xc00019dfe0 pc=0x55ff70fc0ec1
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 2 gp=0xc000006c40 m=nil [force gc (idle)]:
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004afa8 sp=0xc00004af88 pc=0x55ff70f8dfee
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goparkunlock(...)
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:408
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.forcegchelper()
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:326 +0xb8 fp=0xc00004afe0 sp=0xc00004afa8 pc=0x55ff70f8de78
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goexit({})
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc00004afe8 sp=0xc00004afe0 pc=0x55ff70fc0ec1
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         created by runtime.init.7 in goroutine 1
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:314 +0x1a
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         goroutine 3 gp=0xc000007180 m=nil [GC sweep wait]:
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:402 +0xce fp=0xc00004b780 sp=0xc00004b760 pc=0x55ff70f8dfee
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.goparkunlock(...)
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:                 /usr/lib/golang/src/runtime/proc.go:408
      Jul 04 16:17:19 ip-10-0-57-23 kubenswrapper[2277]:         runtime.bgsweep(0xc0000...
      

              rhn-support-jnovy Jindrich Novy
              rhn-support-adistefa Alessandro Di Stefano
              Container Runtime Eng Bot Container Runtime Eng Bot
              Alex Jia Alex Jia
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: