Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-23106

[4.13] Remove z-upgrades from UpgradeBackupController

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Major Major
    • 4.15.0
    • 4.13, 4.12, 4.11, 4.10, 4.14, 4.15
    • Etcd
    • None
    • Moderate
    • No
    • 3
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • N/A
    • Release Note Not Required

      Description of problem:

      Continuous triggers on the UpgradeBackupController on every z-upgrades caused out of space issues for some customers. 
      
      We decided in a meeting to remove the (undocumented) z-stream upgrade triggers. 
      In addition we're going to add a retention mechanism by deleting all files and folders in /etc/kubernetes/cluster-backup before taking a y-stream upgrade backup.

      Version-Release number of selected component (if applicable):

      4.10 -> 4.14

      How reproducible:

      always

      Steps to Reproduce / expected results:

      * do a z-upgrade
      * observe that no backup should be taken anymore
      
      * do a y-upgrade
      * observe that a backup is still taken
      * note that any previous backups should not exist anymore in /etc/kubernetes/cluster-backup

      Additional info:

       

       

       

            [OCPBUGS-23106] [4.13] Remove z-upgrades from UpgradeBackupController

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Important: OpenShift Container Platform 4.13.32 bug fix and security update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2024:0660

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Important: OpenShift Container Platform 4.13.32 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:0660

            Ge Liu added a comment -

            Yes, sandeep, you may check all 3 master nodes to verify there is not cluster-backup dir created, I checked all master nodes, there is not backup dir created, so let's close this bug now.

            Ge Liu added a comment - Yes, sandeep, you may check all 3 master nodes to verify there is not cluster-backup dir created, I checked all master nodes, there is not backup dir created, so let's close this bug now.

            Sandeep Kundu added a comment - - edited

            Please find the below logs.

            before upgrade:

            oc get clusterversion
            NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
            version   4.13.31   True        False         125m    Cluster version is 4.13.31
             

            upgrade in progress.

            oc get clusterversion
            NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
            version   4.13.31   True        True          26s     Working towards 4.13.32: 106 of 841 done (12% complete), waiting on etcd, kube-apiserver
             

            successful upgrade completion.

            oc get clusterversion
            NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
            version   4.13.32   True        False         34m     Cluster version is 4.13.32 

            verify that upgrade backup files are not present in node.

            oc debug no/ip-10-0-136-198.us-east-2.compute.internal
            sh-4.4# chroot /host
            cd /etc/kubernetes/
            sh-5.1# ls
            apiserver-url.env  ca.crt  cloud.conf  cni  kubeconfig    kubelet-ca.crt    kubelet-plugins  kubelet.conf  manifests  static-pod-resources
             
            Backup directory is not present in any of the 3 master nodes. 

             

             

            Sandeep Kundu added a comment - - edited Please find the below logs. before upgrade: oc get clusterversion NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS version   4.13.31   True        False         125m    Cluster version is 4.13.31 upgrade in progress. oc get clusterversion NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS version   4.13.31   True        True          26s     Working towards 4.13.32: 106 of 841 done (12% complete), waiting on etcd, kube-apiserver successful upgrade completion. oc get clusterversion NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS version   4.13.32   True        False         34m     Cluster version is 4.13.32 verify that upgrade backup files are not present in node. oc debug no/ip-10-0-136-198.us-east-2.compute.internal sh-4.4# chroot /host cd /etc/kubernetes/ sh-5.1# ls apiserver-url.env  ca.crt  cloud.conf  cni  kubeconfig    kubelet-ca.crt    kubelet-plugins  kubelet.conf  manifests   static -pod-resources Backup directory is not present in any of the 3 master nodes.    

            Thomas Jungblut added a comment - geliu this change is not included in 4.13.30: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4-stable/release/4.13.30 it went into the following z-release: https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/releasestream/4-stable/release/4.13.31 So this should work from 4.13.31 -> 4.13.32.

            Thanks Ge, I was just about to get to that ticket Let me quickly check myself.

            Thomas Jungblut added a comment - Thanks Ge, I was just about to get to that ticket Let me quickly check myself.

            Ge Liu added a comment -

            There is not latest z stable release ready for test, we will verify it after latest release ready.

            Ge Liu added a comment - There is not latest z stable release ready for test, we will verify it after latest release ready.

            Looks like this bug is far enough along in the workflow that a code fix is ready. Customers and support need to know the backport plan. Please complete the "Target Backport Versions" field to indicate which version(s) will receive the fix.

            OpenShift Jira Bot added a comment - Looks like this bug is far enough along in the workflow that a code fix is ready. Customers and support need to know the backport plan. Please complete the " Target Backport Versions " field to indicate which version(s) will receive the fix.

              dwest@redhat.com Dean West
              tjungblu@redhat.com Thomas Jungblut
              Sandeep Kundu Sandeep Kundu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: