Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-13809

OVN image pre-puller pod uses `imagePullPolicy: Always` and blocks upgrade when there is no registry

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Normal
    • None
    • 4.12
    • Networking / ovn-kubernetes
    • None
    • No
    • False
    • Hide

      None

      Show
      None

    Description

      This is a clone of issue OCPBUGS-13219. The following is the description of the original issue:
      —
      Description of problem:

      OVN image pre-puller blocks upgrades in environments where the images have already been pulled but the registry server is not available.

      Version-Release number of selected component (if applicable):

      4.12

      How reproducible:

      Always

      Steps to Reproduce:

      1. Create a cluster in a disconnected environment.

      2. Manually pre-pull all the images required for the upgrade. For example, get the list of images needed:

      # oc adm release info quay.io/openshift-release-dev/ocp-release:4.12.10-x86_64 -o json > release-info.json

      And then pull them in all the nodes of the cluster:

      # crio pull $(cat release-info.json | jq -r '.references.spec.tags[].from.name')

      3. Stop or somehow make the registry unreachable, then trigger the upgrade.

      Actual results:

      The upgrade blocks with the following error reported by the cluster version operator:

      # oc get clusterversion; oc get co network
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.12.10   True        True          62m     Working towards 4.12.11: 483 of 830 done (58% complete), waiting on network
NAME      VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
network   4.12.10   True        True          False      133m    DaemonSet "/openshift-ovn-kubernetes/ovnkube-upgrades-prepuller" is not available (awaiting 1 nodes)

      The reason for that is that the `ovnkube-upgrades-prepuller-...` pod uses `imagePullPolicy: Always` and that fails if there is no registry, even if the image has already been pulled:

      # oc get pods -n openshift-ovn-kubernetes ovnkube-upgrades-prepuller-5s2cn
NAME                               READY   STATUS             RESTARTS   AGE
ovnkube-upgrades-prepuller-5s2cn   0/1     ImagePullBackOff   0          44m

# oc get events -n openshift-ovn-kubernetes --field-selector involvedObject.kind=Pod,involvedObject.name=ovnkube-upgrades-prepuller-5s2cn,reason=Failed
LAST SEEN   TYPE      REASON   OBJECT                                 MESSAGE
43m         Warning   Failed   pod/ovnkube-upgrades-prepuller-5s2cn   Failed to pull image "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52f189797a83cae8769f1a4dc6dfd46d586914575ee99de6566fc23c77282071": rpc error: code = Unknown desc = (Mirrors also failed: [server.home.arpa:8443/openshift/release@sha256:52f189797a83cae8769f1a4dc6dfd46d586914575ee99de6566fc23c77282071: pinging container registry server.home.arpa:8443: Get "https://server.home.arpa:8443/v2/": dial tcp 192.168.100.1:8443: connect: connection refused]): quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52f189797a83cae8769f1a4dc6dfd46d586914575ee99de6566fc23c77282071: pinging container registry quay.io: Get "https://quay.io/v2/": dial tcp: lookup quay.io on 192.168.100.1:53: server misbehaving
43m         Warning   Failed   pod/ovnkube-upgrades-prepuller-5s2cn   Error: ErrImagePull
43m         Warning   Failed   pod/ovnkube-upgrades-prepuller-5s2cn   Error: ImagePullBackOff

# oc get pod -n openshift-ovn-kubernetes ovnkube-upgrades-prepuller-5s2cn -o json | jq -r '.spec.containers[] | .imagePullPolicy + " " + .image'
Always quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:52f189797a83cae8769f1a4dc6dfd46d586914575ee99de6566fc23c77282071

      Expected results:

      The upgrade should not block.

      Additional info:

      We detected this in a situation where we want to be able to perform upgrades in a disconnected environment and without the registry server running. See MGMT-13733 for details.

      Attachments

        Issue Links

          clones

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-13219 OVN image pre-puller pod uses `imagePullPolicy: Always` and blocks upgrade when there is no registry

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          is blocked by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-13219 OVN image pre-puller pod uses `imagePullPolicy: Always` and blocks upgrade when there is no registry

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          links to

          GitHub openshift/cluster-network-operator#1813: [release-4.13] OCPBUGS-13809: Use `IfNotPresent` instead of `Always` in OVNK upgrades pre-puller

          Activity

            People

              jhernand-rh Juan Hernández
              openshift-crt-jira-prow OpenShift Prow Bot
              Michael Fiedler Michael Fiedler
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: