Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-59773

HKDF-Extract invocation failed unexpectedly (oc-mirror) FIPS

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Normal Normal
    • None
    • 4.18, 4.19, 4.20
    • oc-mirror
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Critical
    • None
    • x86_64
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Problem:

      oc-mirror, when ran on a FIPS enabled EL host, throws a showstopping error:

      HKDF-Extract invocation failed unexpectedly

       

      Expected Behavior:

      To run normally while on a FIPS-enabled EL host.

      Steps to Reproduce:

      • Build a vanilla Centos Stream 9 machine
      • enable fips:
        • sudo fips-mode-setup --enable
        • allow machine to reboot after generating new boot artifacts
      • Download oc and oc-mirror  from console.redhat.com/openshift/downloads
        • note: oc-mirror does not have a RHEL9 (FIPS) build- use the Linux x86_64 arch release
      • Download your auth pull secret from the same portal mentioned in the previous step (name it something like pull-secret.txt
      • Place oc , kubectl , and oc-mirror in user's $HOME/.local/bin directory
      • Write an isc.yml config file with the following content:

      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v2alpha1
      mirror:
        platform:
          channels:
            - name: stable-4.19
          graph: true

        * Note the api version is v2alpha1 which corresponds with oc-mirror --v2 workflows

      • Copy the contents of the pull secret (pull-secret.txt) to /run/user/$(id -u)/containers/auth.json so the oc-mirror tool can authenticate to our registries and pull content
      • Run oc mirror -config isc.yml file://foo -v2 to initiate a mirrorToDisk workflow
      • The resultant error in this report should be displayed (HKDF-Extract invocation failed unexpectedly)
         
        oc-mirror can also be called directly as an alternative, as apposed to calling it as a plugin to oc  (via oc mirror instead of oc-mirror)
         
        Host Information:

      [admin@host ~]$ oc mirror version
      W0724 15:34:01.884269   24676 mirror.go:86]

      ⚠️  oc-mirror v1 is deprecated (starting in 4.18 release) and will be removed in a future release - please migrate to oc-mirror --v2

      WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
      Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.19.0-202507081507.p0.gf364aec.assembly.stream.el9-f364aec", GitCommit:"f364aec3fc13fad183ff85ad40b6c861964c483e", GitTreeState:"clean", BuildDate:"2025-07-09T00:32:22Z", GoVersion:"go1.23.9 (Red Hat 1.23.9-1.el9_6) X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}

      [admin@host ~]$ cat /etc/os-release
      NAME="CentOS Stream"
      VERSION="9"
      ID="centos"
      ID_LIKE="rhel fedora"
      VERSION_ID="9"
      PLATFORM_ID="platform:el9"
      PRETTY_NAME="CentOS Stream 9"
      ANSI_COLOR="0;31"
      LOGO="fedora-logo-icon"
      CPE_NAME="cpe:/o:centos:centos:9"
      HOME_URL="https://centos.org/"
      BUG_REPORT_URL="https://issues.redhat.com/"
      REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
      REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

      [admin@host ~]$ sudo dnf info openssl
      Updating Subscription Management repositories.
      Unable to read consumer identity

      This system is not registered with an entitlement server. You can use "rhc" or "subscription-manager" to register.

      Last metadata expiration check: 0:37:18 ago on Wed 23 Jul 2025 12:35:06 PM PDT.
      Installed Packages
      Name         : openssl
      Epoch        : 1
      Version      : 3.5.1
      Release      : 2.el9
      Architecture : x86_64
      Size         : 2.2 M
      Source       : openssl-3.5.1-2.el9.src.rpm
      Repository   : @System
      From repo    : baseos
      Summary      : Utilities from the general purpose cryptography library with TLS implementation
      URL          : http://www.openssl.org/
      License      : Apache-2.0
      Description  : The OpenSSL toolkit provides support for secure communications between
                   : machines. OpenSSL includes a certificate management tool and shared
                   : libraries which provide various cryptographic algorithms and
                   : protocols.

      Output:

      2025/07/23 13:08:04  [INFO]   : 👋 Hello, welcome to oc-mirror
      2025/07/23 13:08:04  [INFO]   : ⚙️  setting up the environment for you...
      2025/07/23 13:08:04  [INFO]   : 🔀 workflow mode: mirrorToDisk
      2025/07/23 13:08:04  [INFO]   : 🕵  going to discover the necessary images...
      2025/07/23 13:08:04  [INFO]   : 🔍 collecting release images...
      panic: tls: HKDF-Extract invocation failed unexpectedly

      goroutine 36 [running]:
      crypto/tls.(*cipherSuiteTLS13).extract(0x593ee40, {0x0?, 0x0?, 0x0?}, {0x0, 0x0, 0x0})
              /usr/lib/golang/src/crypto/tls/key_schedule.go:99 +0x187
      crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys(0xc000733c50)
              /usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:518 +0x2ce
      crypto/tls.(*clientHandshakeStateTLS13).handshake(0xc000733c50)
              /usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:136 +0x785
      crypto/tls.(*Conn).clientHandshake(0xc00053e708, {0x4193ac8, 0xc0006b6190})
              /usr/lib/golang/src/crypto/tls/handshake_client.go:372 +0x845
      crypto/tls.(*Conn).handshakeContext(0xc00053e708, {0x4193ac8, 0xc00052ea50})
              /usr/lib/golang/src/crypto/tls/conn.go:1568 +0x3a6
      crypto/tls.(*Conn).HandshakeContext(...)
              /usr/lib/golang/src/crypto/tls/conn.go:1508
      net/http.(*persistConn).addTLS.func2()
              /usr/lib/golang/src/net/http/transport.go:1651 +0x6e
      created by net/http.(*persistConn).addTLS in goroutine 33
              /usr/lib/golang/src/net/http/transport.go:1647 +0x309

      Possible Related:
      https://github.com/golang-fips/go/issues/205

      Related Chat Threads:
      https://redhat-internal.slack.com/archives/C02JW6VCYS1/p1753301343349389

              luzuccar@redhat.com Luigi Mario Zuccarelli
              rhn-gps-ckuperst Christoffer Kuperstein (Inactive)
              None
              None
              Nidan Gavali Nidan Gavali
              None
              Votes:
              6 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: