Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-51140

[4.18][catalogd] olmv1 reads wrong image URL when controller pod does not restart after updating registry.conf

XMLWordPrintable

    • Important
    • None
    • Glaceon OLM Sprint 267
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the Operator Controller would not accept live updates to registries with a proxy configuration. Unless the controller pods were restarted, this issue caused OLM v1 to read the wrong image URL. With this release, a fix to the Operator Controller means that it accepts live updates to registries with a proxy configuration and the controller pod no longer needs to be restarted. (link:https://issues.redhat.com/browse/OCPBUGS-51140[*OCPBUGS-51140*])
      Show
      * Previously, the Operator Controller would not accept live updates to registries with a proxy configuration. Unless the controller pods were restarted, this issue caused OLM v1 to read the wrong image URL. With this release, a fix to the Operator Controller means that it accepts live updates to registries with a proxy configuration and the controller pod no longer needs to be restarted. (link: https://issues.redhat.com/browse/OCPBUGS-51140 [* OCPBUGS-51140 *])
    • Bug Fix
    • Done

      Description of problem:

          we have epic https://issues.redhat.com/browse/OPRUN-3460 to support it, and it worked when testing new feature. recently our automation case fails, and we found it does not work now after checking.
      
          by the way, we can still find the updated /etc/containers/registries.conf in pod catalogd-controller-manager and operator-controller-controller-manager.
      But when we tested the new feature, we found they restart when we apply ICSP, ImageDigestMirrorSet or ImageTagMirrorSet. While now it seems they does not restart. I am not sure it is reason why it does not work although we can see /etc/containers/registries.conf updated.
      
         please refer to the following step to reproduce the issue.

      Version-Release number of selected component (if applicable):

      [root@preserve-olm-env2 OCP-76843]# oc get clusterversion
      NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
      version   4.18.0-0.nightly-2024-11-27-162407   True        False         4h4m    Cluster version is 4.18.0-0.nightly-2024-11-27-162407    

      How reproducible:

          always

      Steps to Reproduce:

      [root@preserve-olm-env2 OCP-76843]# cat icsp.yaml 
      apiVersion: operator.openshift.io/v1alpha1
      kind: ImageContentSourcePolicy
      metadata:
        name: iscp-76843
      spec:
        repositoryDigestMirrors:
        - mirrors:
          - quay.io/olmqe
          source: qe76843.myregistry.io/olmqe[root@preserve-olm-env2 OCP-76843]# oc apply -f icsp.yaml 
      imagecontentsourcepolicy.operator.openshift.io/iscp-76843 created
      [root@preserve-olm-env2 OCP-76843]# oc get imagecontentsourcepolicy.operator.openshift.io/iscp-76843 -o yaml
      apiVersion: operator.openshift.io/v1alpha1
      kind: ImageContentSourcePolicy
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"operator.openshift.io/v1alpha1","kind":"ImageContentSourcePolicy","metadata":{"annotations":{},"name":"iscp-76843"},"spec":{"repositoryDigestMirrors":[{"mirrors":["quay.io/olmqe"],"source":"qe76843.myregistry.io/olmqe"}]}}
        creationTimestamp: "2024-11-29T05:22:40Z"
        generation: 1
        name: iscp-76843
        resourceVersion: "140255"
        uid: ed93abf4-1fb5-415c-a36d-c31540e3cf4b
      spec:
        repositoryDigestMirrors:
        - mirrors:
          - quay.io/olmqe
          source: qe76843.myregistry.io/olmqe
      [root@preserve-olm-env2 OCP-76843]# oc get mcp
      NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master   rendered-master-cad53c4acbcdfa70f5ec101b02b3ce64   False     True       False      3              2                   2                     0                      4h32m
      worker   rendered-worker-73b828a90420f12c221aec5e1ff61a4c   False     True       False      3              2                   2                     0                      4h32m
      [root@preserve-olm-env2 OCP-76843]# oc get mcp
      NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
      master   rendered-master-e794e5893ba064c2f45cda3ca68af6b8   True      False      False      3              3                   3                     0                      4h33m
      worker   rendered-worker-09c87bcab52d6bb46a047c9c288f99a1   True      False      False      3              3                   3                     0                      4h33m
      [root@preserve-olm-env2 OCP-76843]# oc -n openshift-catalogd get pod
      NAME                                          READY   STATUS    RESTARTS   AGE
      catalogd-controller-manager-6554d794d-dpt6w   2/2     Running   0          159m
      [root@preserve-olm-env2 OCP-76843]# oc -n openshift-catalogd rsh catalogd-controller-manager-6554d794d-dpt6w
      sh-5.1$ cat /etc/containers/registries.conf
      unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]
      short-name-mode = ""[[registry]]
        prefix = ""
        location = "qe76843.myregistry.io/olmqe"  [[registry.mirror]]
          location = "quay.io/olmqe"
          pull-from-mirror = "digest-only"[[registry]]
        prefix = ""
        location = "registry-proxy.engineering.redhat.com"  [[registry.mirror]]
          location = "brew.registry.redhat.io"
          pull-from-mirror = "digest-only"[[registry]]
        prefix = ""
        location = "registry.redhat.io"  [[registry.mirror]]
          location = "brew.registry.redhat.io"
          pull-from-mirror = "digest-only"[[registry]]
        prefix = ""
        location = "registry.stage.redhat.io"  [[registry.mirror]]
          location = "brew.registry.redhat.io"
          pull-from-mirror = "digest-only"
      sh-5.1$ exit
      exit
      [root@preserve-olm-env2 OCP-76843]# 
      [root@preserve-olm-env2 OCP-76843]# oc get clustercatalog.olm.operatorframework.io/clustercatalog-76843 -o yaml
      apiVersion: olm.operatorframework.io/v1
      kind: ClusterCatalog
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"olm.operatorframework.io/v1","kind":"ClusterCatalog","metadata":{"annotations":{},"name":"clustercatalog-76843"},"spec":{"source":{"image":{"ref":"qe76843.myregistry.io/olmqe/nginx-ok-index@sha256:c613ddd68b74575d823c6f370c0941b051ea500aa4449224489f7f2cc716e712"},"type":"Image"}}}
        creationTimestamp: "2024-11-29T05:30:33Z"
        finalizers:
        - olm.operatorframework.io/delete-server-cache
        generation: 1
        labels:
          olm.operatorframework.io/metadata.name: clustercatalog-76843
        name: clustercatalog-76843
        resourceVersion: "143831"
        uid: 5b8a7b27-79de-456d-9389-7a4440340706
      spec:
        availabilityMode: Available
        priority: 0
        source:
          image:
            ref: qe76843.myregistry.io/olmqe/nginx-ok-index@sha256:c613ddd68b74575d823c6f370c0941b051ea500aa4449224489f7f2cc716e712
          type: Image
      status:
        conditions:
        - lastTransitionTime: "2024-11-29T05:31:33Z"
          message: 'source catalog content: error copying image: initializing source docker://qe76843.myregistry.io/olmqe/nginx-ok-index@sha256:c613ddd68b74575d823c6f370c0941b051ea500aa4449224489f7f2cc716e712:
            pinging container registry qe76843.myregistry.io: Get "https://qe76843.myregistry.io/v2/":
            dial tcp 37.120.182.60:443: i/o timeout'
          observedGeneration: 1
          reason: Retrying
          status: "True"
          type: Progressing
      [root@preserve-olm-env2 OCP-76843]#      

      Actual results:

          can not install clustercalog.

      Expected results:

          can install clustercalog. 

      Additional info:

          

              rh-ee-dfranz Daniel Franz
              rhn-support-kuiwang Kui Wang
              Kui Wang Kui Wang
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: