Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-55403

OLMv1 cannot get the trust CA: got the x509 error

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Undefined Undefined
    • None
    • 4.19.0
    • OLM
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • Rejected
    • None
    • Done
    • Release Note Not Required
    • Hide
      N/A
      Show
      N/A
    • None
    • None
    • None
    • None

      Description of problem:

      Got the below error when using the custom internal image registry.

      jiazha-mac:~ jiazha$ oc get clustercatalog cc-redhat-operator-index-v4-17 -o yaml
      apiVersion: olm.operatorframework.io/v1
      kind: ClusterCatalog
      metadata:
        creationTimestamp: "2025-04-27T07:21:06Z"
        finalizers:
        - olm.operatorframework.io/delete-server-cache
        generation: 1
        labels:
          olm.operatorframework.io/metadata.name: cc-redhat-operator-index-v4-17
        name: cc-redhat-operator-index-v4-17
        resourceVersion: "87633"
        uid: 747f3cb8-2d52-45ed-aa9b-cce82cc7dc71
      spec:
        availabilityMode: Available
        priority: 0
        source:
          image:
            ref: my-route-jian.apps.heli-419.qe.devcluster.openshift.com/redhat/redhat-operator-index:v4.17
          type: Image
      status:
        conditions:
        - lastTransitionTime: "2025-04-27T07:21:06Z"
          message: 'source catalog content: error creating image source: pinging container
            registry my-route-jian.apps.heli-419.qe.devcluster.openshift.com: Get "https://my-route-jian.apps.heli-419.qe.devcluster.openshift.com/v2/":
            tls: failed to verify certificate: x509: certificate signed by unknown authority'
          observedGeneration: 1
          reason: Retrying
          status: "True"
          type: Progressing

      Version-Release number of selected component (if applicable):

      The OLMv1 versions as follows,

          4.19.0-0.nightly-2025-04-24-005837
      
      jiazha-mac:~ jiazha$ oc adm release info -a .dockerconfigjson --commits registry.ci.openshift.org/ocp/release:4.19.0-0.nightly-2025-04-24-005837 |grep olm 
        cluster-olm-operator                           https://github.com/openshift/cluster-olm-operator                           6a517418e9a01d880226959b06c6adcaf0f4d504
        olm-catalogd                                   https://github.com/openshift/operator-framework-operator-controller         355dcf40d5fcb4d9e93ab312a376c4e80d46ba76
        olm-operator-controller                        https://github.com/openshift/operator-framework-operator-controller         355dcf40d5fcb4d9e93ab312a376c4e80d46ba76

      How reproducible:

          always

      Steps to Reproduce:

          1. create an image registry,
      jiazha-mac:~ jiazha$ oc  new-app --image quay.io/openshifttest/registry@sha256:1106aedc1b2e386520bc2fb797d9a7af47d651db31d8e7ab472f2352da37d1b3 REGISTRY_STORAGE_DELETE_ENABLED=true --import-mode=PreserveOriginal
      --> Found container image 1106aed (1 second old) from quay.io for "quay.io/openshifttest/registry@sha256:1106aedc1b2e386520bc2fb797d9a7af47d651db31d8e7ab472f2352da37d1b3"    * An image stream tag will be created as "registry:latest" that will track this image--> Creating resources ...
          imagestream.image.openshift.io "registry" created
          deployment.apps "registry" created
          service "registry" created
      --> Success
          Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
           'oc expose service/registry' 
          Run 'oc status' to view your app.
      
      jiazha-mac:~ jiazha$ oc get svc 
      NAME       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
      registry   ClusterIP   172.30.197.19   <none>        5000/TCP   10s
      jiazha-mac:~ jiazha$ 
      jiazha-mac:~ jiazha$ oc create route edge my-route --service=registry
      route.route.openshift.io/my-route created
      
      jiazha-mac:~ jiazha$ oc get route 
      NAME       HOST/PORT                                                 PATH   SERVICES   PORT       TERMINATION   WILDCARD
      my-route   my-route-jian.apps.heli-419.qe.devcluster.openshift.com          registry   5000-tcp   edge          None
      jiazha-mac:~ jiazha$ 
      jiazha-mac:~ jiazha$ oc set volume deploy registry --add -t pvc  --claim-size=30G -m /var/lib/registry --overwrite
      deployment.apps/registry volume updated
      
      2, update the trust CA.
      
      jiazha-mac:~ jiazha$ oc extract secret/router-ca -n openshift-ingress-operator --to=/tmp --confirm
      /tmp/tls.crt
      /tmp/tls.key
      jiazha-mac:~ jiazha$ oc create -n openshift-config configmap trusted-ca-73124 --from-file=my-route-jian.apps.heli-419.qe.devcluster.openshift.com=/tmp/tls.crt --from-file=updateservice-registry=/tmp/tls.crt 
      configmap/trusted-ca-73124 created
      jiazha-mac:~ jiazha$ 
      jiazha-mac:~ jiazha$ oc  patch image.config.openshift.io/cluster -p '{"spec": {"additionalTrustedCA": {"name": "trusted-ca-73124"}}}' --type=merge
      image.config.openshift.io/cluster patched
      
      3. create a ClusterCatalog.
      jiazha-mac:~ jiazha$ cat cc-internal-registry.yaml 
      apiVersion: olm.operatorframework.io/v1
      kind: ClusterCatalog
      metadata:
        name: cc-redhat-operator-index-v4-17
      spec:
        availabilityMode: Available
        priority: 0
        source:
          image:
            ref: my-route-jian.apps.heli-419.qe.devcluster.openshift.com/redhat/redhat-operator-index:v4.17
          type: Image
      
      jiazha-mac:~ jiazha$ oc create -f cc-internal-registry.yaml 
      clustercatalog.olm.operatorframework.io/cc-redhat-operator-index-v4-17 created
           

      Actual results:

      Got the `tls: failed to verify certificate: x509: certificate signed by unknown authority'` error. 

          jiazha-mac:~ jiazha$ oc get clustercatalog cc-redhat-operator-index-v4-17 -o yaml
      apiVersion: olm.operatorframework.io/v1
      kind: ClusterCatalog
      metadata:
        creationTimestamp: "2025-04-27T07:21:06Z"
        finalizers:
        - olm.operatorframework.io/delete-server-cache
        generation: 1
        labels:
          olm.operatorframework.io/metadata.name: cc-redhat-operator-index-v4-17
        name: cc-redhat-operator-index-v4-17
        resourceVersion: "87633"
        uid: 747f3cb8-2d52-45ed-aa9b-cce82cc7dc71
      spec:
        availabilityMode: Available
        priority: 0
        source:
          image:
            ref: my-route-jian.apps.heli-419.qe.devcluster.openshift.com/redhat/redhat-operator-index:v4.17
          type: Image
      status:
        conditions:
        - lastTransitionTime: "2025-04-27T07:21:06Z"
          message: 'source catalog content: error creating image source: pinging container
            registry my-route-jian.apps.heli-419.qe.devcluster.openshift.com: Get "https://my-route-jian.apps.heli-419.qe.devcluster.openshift.com/v2/":
            tls: failed to verify certificate: x509: certificate signed by unknown authority'
          observedGeneration: 1
          reason: Retrying
          status: "True"
          type: Progressing
      
      

      Expected results:

      The ClusterCatalog that uses the custom internal image registry works well.

      Additional info:

      The CAs under /etc/docker/ look good.

      jiazha-mac:~ jiazha$ oc debug node/ip-10-0-10-189.us-east-2.compute.internal
      Temporary namespace openshift-debug-l5njz is created for debugging node...
      Starting pod/ip-10-0-10-189us-east-2computeinternal-debug-mdsv7 ...
      To use host binaries, run `chroot /host`
      ...
      sh-5.1# ls -R /etc/docker/
      /etc/docker/:
      certs.d
      
      
      /etc/docker/certs.d:
      image-registry.openshift-image-registry.svc.cluster.local:5000	my-route-jian.apps.heli-419.qe.devcluster.openshift.com
      image-registry.openshift-image-registry.svc:5000		updateservice-registry
      
      
      '/etc/docker/certs.d/image-registry.openshift-image-registry.svc.cluster.local:5000':
      ca.crt
      
      
      '/etc/docker/certs.d/image-registry.openshift-image-registry.svc:5000':
      ca.crt
      
      
      /etc/docker/certs.d/my-route-jian.apps.heli-419.qe.devcluster.openshift.com:
      ca.crt
      
      
      /etc/docker/certs.d/updateservice-registry:
      ca.crt
      
      

      This issue was also found in https://qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/view/gs/qe-private-deck/logs/periodic-ci-o[...]d-release-gcp-ipi-f999/1916290620670873600

              rhn-support-jiazha Jian Zhang
              rhn-support-jiazha Jian Zhang
              None
              None
              Jian Zhang Jian Zhang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: