Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-41552

ROSA HCP Nodepool versions unexpectedly do not match Node versions

    • Moderate
    • None
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, a {hcp} cluster that used mirroring release images might result in existing node pools to use the hosted cluster's operating system version instead of the `NodePool` version. With this release, a fix ensures that node pools use their own versions. (link:https://issues.redhat.com/browse/OCPBUGS-41552[*OCPBUGS-41552*])
      Show
      * Previously, a {hcp} cluster that used mirroring release images might result in existing node pools to use the hosted cluster's operating system version instead of the `NodePool` version. With this release, a fix ensures that node pools use their own versions. (link: https://issues.redhat.com/browse/OCPBUGS-41552 [* OCPBUGS-41552 *])
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-39420. The following is the description of the original issue:

      Description of problem:

      ROSA HCP allows customers to select hostedcluster and nodepool OCP z-stream versions, respecting version skew requirements. E.g.:

      • A 4.15.28 hostedcluster with
      • A 4.15.28 nodepool
      • A 4.15.25 nodepool

      Version-Release number of selected component (if applicable):

      Reproducible on 4.14-4.16.z, this bug report demonstrates it for a 4.15.28 hostedcluster with a 4.15.25 nodepool

      How reproducible:

      100%    

      Steps to Reproduce:

          1. Create a ROSA HCP cluster, which comes with a 2-replica nodepool with the same z-stream version (4.15.28)
          2. Create an additional nodepool at a different version (4.15.25)
          

      Actual results:

      Observe that while nodepool objects report the different version (4.15.25), the resulting kernel version of the node is that of the hostedcluster (4.15.28)
      
      ❯ k get nodepool -n ocm-staging-2didt6btjtl55vo3k9hckju8eeiffli8                                                                                    
      NAME                     CLUSTER       DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION   UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
      mshen-hyper-np-4-15-25   mshen-hyper   1               1               False         True         4.15.25   False             False            
      mshen-hyper-workers      mshen-hyper   2               2               False         True         4.15.28   False             False  
      
      
      ❯ k get no -owide                                            
      NAME                                         STATUS   ROLES    AGE   VERSION            INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                       KERNEL-VERSION                  CONTAINER-RUNTIME
      ip-10-0-129-139.us-west-2.compute.internal   Ready    worker   24m   v1.28.12+396c881   10.0.129.139   <none>        Red Hat Enterprise Linux CoreOS 415.92.202408100433-0 (Plow)   5.14.0-284.79.1.el9_2.aarch64   cri-o://1.28.9-5.rhaos4.15.git674ed4c.el9
      ip-10-0-129-165.us-west-2.compute.internal   Ready    worker   98s   v1.28.12+396c881   10.0.129.165   <none>        Red Hat Enterprise Linux CoreOS 415.92.202408100433-0 (Plow)   5.14.0-284.79.1.el9_2.aarch64   cri-o://1.28.9-5.rhaos4.15.git674ed4c.el9
      ip-10-0-132-50.us-west-2.compute.internal    Ready    worker   30m   v1.28.12+396c881   10.0.132.50    <none>        Red Hat Enterprise Linux CoreOS 415.92.202408100433-0 (Plow)   5.14.0-284.79.1.el9_2.aarch64   cri-o://1.28.9-5.rhaos4.15.git674ed4c.el9

      Expected results:

          

      Additional info:

       

            [OCPBUGS-41552] ROSA HCP Nodepool versions unexpectedly do not match Node versions

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Important: OpenShift Container Platform 4.17.2 bug fix and security update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2024:8229

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Important: OpenShift Container Platform 4.17.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:8229

            He Liu added a comment -

            test passed

            $ oc get hc -A
            oc get nNAMESPACE   NAME        VERSION                              KUBECONFIG                   PROGRESS    AVAILABLE   PROGRESSING   MESSAGE
            clusters    heli-test   4.17.0-0.nightly-2024-10-07-214907   heli-test-admin-kubeconfig   Completed   True        False         The hosted control plane is available
            
            $ oc get np -A
            NAMESPACE   NAME                   CLUSTER     DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION                              UPDATINGVERSION   UPDATINGCONFIG   MESSAGE
            clusters    heli-test-02           heli-test   1               1               False         False        4.16.0-0.nightly-2024-10-07-143832   False             False            
            clusters    heli-test-us-east-2a   heli-test   1               1               False         False        4.17.0-0.nightly-2024-10-07-214907   False             False 
            
            $ oc get pod -n clusters-heli-test ignition-server-67d4889b8b-zzjqc -oyaml | grep -E "OPENSHIFT_IMG_OVERRIDES|MIRRORED_RELEASE_IMAGE" -a5
                - name: MY_NAMESPACE
                  valueFrom:
                    fieldRef:
                      apiVersion: v1
                      fieldPath: metadata.namespace
                - name: OPENSHIFT_IMG_OVERRIDES
                  value: quay.io/openshift-release-dev/ocp-v4.0-art-dev=preservehypershiftaks.azurecr.io/ocp-image,registry-proxy.engineering.redhat.com=brew.registry.redhat.io,registry.ci.openshift.org/ocp/release=preservehypershiftaks.azurecr.io/ocp-image,registry.redhat.io=brew.registry.redhat.io,registry.stage.redhat.io=brew.registry.redhat.io
                - name: MIRRORED_RELEASE_IMAGE
                  value: preservehypershiftaks.azurecr.io/ocp-image@sha256:bc2b469692835520b264a4520ddb62273c60e69ded56fd79890fb932673f050d
                image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7039796d59a0a41b06a9e686808eac54d72993e9acc1858639e681db4f202c3a
                imagePullPolicy: IfNotPresent
                livenessProbe:
                  failureThreshold: 6
            
            $ oc --kubeconfig=/Users/harry/Downloads/hc.kubeconfig get no -owide
            NAME                                         STATUS   ROLES    AGE     VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                KERNEL-VERSION                 CONTAINER-RUNTIME
            ip-10-0-138-101.us-east-2.compute.internal   Ready    worker   5m31s   v1.29.8+632b078   10.0.138.101   <none>        Red Hat Enterprise Linux CoreOS 416.94.202410031653-0   5.14.0-427.37.1.el9_4.x86_64   cri-o://1.29.9-2.rhaos4.16.git933bdd2.el9
            ip-10-0-140-149.us-east-2.compute.internal   Ready    worker   69m     v1.30.4           10.0.140.149   <none>        Red Hat Enterprise Linux CoreOS 417.94.202410032347-0   5.14.0-427.37.1.el9_4.x86_64   cri-o://1.30.6-2.rhaos4.17.git747ac86.el9 

            He Liu added a comment - test passed $ oc get hc -A oc get nNAMESPACE   NAME        VERSION                              KUBECONFIG                   PROGRESS    AVAILABLE   PROGRESSING   MESSAGE clusters    heli-test   4.17.0-0.nightly-2024-10-07-214907   heli-test-admin-kubeconfig   Completed   True        False         The hosted control plane is available $ oc get np -A NAMESPACE   NAME                   CLUSTER     DESIRED NODES   CURRENT NODES   AUTOSCALING   AUTOREPAIR   VERSION                              UPDATINGVERSION   UPDATINGCONFIG   MESSAGE clusters    heli-test-02           heli-test   1               1               False         False        4.16.0-0.nightly-2024-10-07-143832   False             False             clusters    heli-test-us-east-2a   heli-test   1               1               False         False        4.17.0-0.nightly-2024-10-07-214907   False             False $ oc get pod -n clusters-heli-test ignition-server-67d4889b8b-zzjqc -oyaml | grep -E "OPENSHIFT_IMG_OVERRIDES|MIRRORED_RELEASE_IMAGE" -a5     - name: MY_NAMESPACE       valueFrom:         fieldRef:           apiVersion: v1           fieldPath: metadata.namespace     - name: OPENSHIFT_IMG_OVERRIDES       value: quay.io/openshift-release-dev/ocp-v4.0-art-dev=preservehypershiftaks.azurecr.io/ocp-image,registry-proxy.engineering.redhat.com=brew.registry.redhat.io,registry.ci.openshift.org/ocp/release=preservehypershiftaks.azurecr.io/ocp-image,registry.redhat.io=brew.registry.redhat.io,registry.stage.redhat.io=brew.registry.redhat.io     - name: MIRRORED_RELEASE_IMAGE       value: preservehypershiftaks.azurecr.io/ocp-image@sha256:bc2b469692835520b264a4520ddb62273c60e69ded56fd79890fb932673f050d     image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7039796d59a0a41b06a9e686808eac54d72993e9acc1858639e681db4f202c3a     imagePullPolicy: IfNotPresent     livenessProbe:       failureThreshold: 6 $ oc --kubeconfig=/Users/harry/Downloads/hc.kubeconfig get no -owide NAME                                         STATUS   ROLES    AGE     VERSION           INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                                                KERNEL-VERSION                 CONTAINER-RUNTIME ip-10-0-138-101.us-east-2.compute.internal   Ready    worker   5m31s   v1.29.8+632b078   10.0.138.101   <none>        Red Hat Enterprise Linux CoreOS 416.94.202410031653-0   5.14.0-427.37.1.el9_4.x86_64   cri-o: //1.29.9-2.rhaos4.16.git933bdd2.el9 ip-10-0-140-149.us-east-2.compute.internal   Ready    worker   69m     v1.30.4           10.0.140.149   <none>        Red Hat Enterprise Linux CoreOS 417.94.202410032347-0   5.14.0-427.37.1.el9_4.x86_64   cri-o: //1.30.6-2.rhaos4.17.git747ac86.el9

            Hi trking,

            Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

            OpenShift Jira Bot added a comment - Hi trking , Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

              agarcial@redhat.com Alberto Garcia Lamela
              openshift-crt-jira-prow OpenShift Prow Bot
              He Liu He Liu
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: