Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-15997

Machine with AZ and rootVolume but no volume AZ can't be created if Cinder AZs != Nova AZs

    • +
    • No
    • ShiftStack Sprint 238, ShiftStack Sprint 239, ShiftStack Sprint 240
    • 3
    • False
    • Hide

      None

      Show
      None
    • Hide
      * For clusters that run on {rh-openstack} and that you upgrade to version 4.14, when compute zones are configured with at least one zone, root volumes must now also be configured with at least one zone. If this configuration change does not occur, a control plane machine set cannot be generated for your cluster.

      To make the required change, follow the instructions in link:https://access.redhat.com/solutions/7024383[OpenShift on OpenStack with compute Availability Zones: Missing rootVolume availability zone].

      (link:https://issues.redhat.com/browse/OCPBUGS-15997[*OCPBUGS-15997*])
      Show
      * For clusters that run on {rh-openstack} and that you upgrade to version 4.14, when compute zones are configured with at least one zone, root volumes must now also be configured with at least one zone. If this configuration change does not occur, a control plane machine set cannot be generated for your cluster. To make the required change, follow the instructions in link: https://access.redhat.com/solutions/7024383 [OpenShift on OpenStack with compute Availability Zones: Missing rootVolume availability zone]. (link: https://issues.redhat.com/browse/OCPBUGS-15997 [* OCPBUGS-15997 *])
    • Known Issue
    • Done

      Description of problem:

      When a machine is created with a compute availability zone (defined via mpool.zones) and a storage root volume (defined as mpool.rootVolume) and that rootVolume has no specified zones, CAPO will use the compute AZ for the volume AZ.
      
      This can be problematic if the AZ doesn't exist in Cinder.
      Source:
      
      https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/9d183bd479fe9aed4f6e7ac3d5eee46681c518e7/pkg/cloud/services/compute/instance.go#L439-L442

      Version-Release number of selected component (if applicable):

      All versions supporting rootVolume AZ.

      Steps to Reproduce:

      1. In install-config.yaml, add "zones" with valid Nova AZs, and a rootVolume without "zones". Your OpenStack cloud must not have Cinder AZs (only Nova AZs)
      2. Day 1 deployment will go fine, Terraform will create the machines with no AZ.
      3. Day 2 operation on machines will fail since CAPO tries to use the Nova AZ for the root volume if no volume AZ is provided, but since the AZ don't match between Cinder & Nova, the machine will never be created
      

      Actual results:

      Machine not created

      Expected results:

      Machine created in the right AZ for both Nova & Cinder

            [OCPBUGS-15997] Machine with AZ and rootVolume but no volume AZ can't be created if Cinder AZs != Nova AZs

            Errata Tool added a comment -

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Important: OpenShift Container Platform 4.14.0 bug fix and security update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2023:5006

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Important: OpenShift Container Platform 4.14.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:5006

            like the other bug, please mention that needs to happen after an upgrade on 4.14 too, because we were not enforcing it before and we are now. Otherwise LGTM. Thanks.

            Emilien Macchi added a comment - like the other bug, please mention that needs to happen after an upgrade on 4.14 too, because we were not enforcing it before and we are now. Otherwise LGTM. Thanks.

            Max Bridges added a comment -

            emacchi@redhat.com Is this the gist of it?

            For clusters that run on {rh-openstack}, when compute zones are configured with at least one zone, root volumes must now also be configured with at least one zone. If this configuration change does not occur, a control plane machine set cannot be generated for your cluster.

            To make the required change, follow the instructions in link:https://access.redhat.com/solutions/7024383[OpenShift on OpenStack with compute Availability Zones: Missing rootVolume availability zone].

            Max Bridges added a comment - emacchi@redhat.com Is this the gist of it? For clusters that run on {rh-openstack}, when compute zones are configured with at least one zone, root volumes must now also be configured with at least one zone. If this configuration change does not occur, a control plane machine set cannot be generated for your cluster. To make the required change, follow the instructions in link: https://access.redhat.com/solutions/7024383[OpenShift on OpenStack with compute Availability Zones: Missing rootVolume availability zone].

            mbridges@redhat.com FYI the release note on this one is very important.

            Emilien Macchi added a comment - mbridges@redhat.com FYI the release note on this one is very important.

            Jon Uriarte added a comment -

            Verified in 4.14.0-0.nightly-2023-08-11-055332.

            $ openstack availability zone list --compute
            +-----------+-------------+
            | Zone Name | Zone Status |
            +-----------+-------------+
            | internal  | available   |
            | AZ0       | available   |
            | AZ1       | available   |
            | AZ2       | available   |
            +-----------+-------------+
            
            $ openstack availability zone list --volume
            +-----------+-------------+
            | Zone Name | Zone Status |
            +-----------+-------------+
            | nova      | available   |
            +-----------+-------------+
            
            

            **

            Master node's rootVolume AZ validation
            Node AZ is set but no rootVolume AZ (from install-config.yaml):

            compute:
            - name: worker
              platform:
                openstack:
                  zones: []
                  additionalNetworkIDs: []
                  type: "m4.xlarge"
              replicas: 2
            controlPlane:
              name: master
              platform:
                openstack:
                  zones: ['AZ0', 'AZ1', 'AZ2']
                  additionalNetworkIDs: []
                  type: "m4.xlarge"
                  rootVolume:
                    size: 100
                    type: tripleo
                    zones: []
              replicas: 3
            

            The installer raises the validation error:

            DEBUG OpenShift Installer 4.14.0-0.nightly-2023-08-11-055332
            DEBUG Built from commit 0be5579f9c69d275db0600f41fac1686068a62c8
            DEBUG Fetching Metadata...
            DEBUG Loading Metadata...
            DEBUG Loading Cluster ID...
            DEBUG Loading Install Config...
            DEBUG Loading SSH Key...
            DEBUG Loading Base Domain...
            DEBUG Loading Platform...
            DEBUG Loading Cluster Name...
            DEBUG Loading Base Domain...
            DEBUG Loading Platform...
            DEBUG Loading Networking...
            DEBUG Loading Platform...
            DEBUG Loading Pull Secret...
            DEBUG Loading Platform...
            ERROR failed to fetch Metadata: failed to load asset "Install Config": failed to create install config: invalid "install-config.yaml" file: controlPlane.platform.openstack.rootVolume.zones: Required value: root volume availability zones must be specified when compute availability zones are specified
            

             

            Worker node's rootVolume AZ validation
            Node AZ is set but no rootVolume AZ (from install-config.yaml):

            compute:
            - name: worker
              platform:
                openstack:
                  zones: ['AZ0', 'AZ1', 'AZ2']
                  additionalNetworkIDs: []
                  type: "m4.xlarge"
                  rootVolume:
                    size: 100
                    type: tripleo
                    zones: []
              replicas: 2
            controlPlane:
              name: master
              platform:
                openstack:
                  zones: []
                  additionalNetworkIDs: []
                  type: "m4.xlarge"
              replicas: 3
            

            The installer raises the validation error:

            DEBUG OpenShift Installer 4.14.0-0.nightly-2023-08-11-055332
            DEBUG Built from commit 0be5579f9c69d275db0600f41fac1686068a62c8
            DEBUG Fetching Metadata...
            DEBUG Loading Metadata...
            DEBUG Loading Cluster ID...
            DEBUG Loading Install Config...
            DEBUG Loading SSH Key...
            DEBUG Loading Base Domain...
            DEBUG Loading Platform...
            DEBUG Loading Cluster Name...
            DEBUG Loading Base Domain...
            DEBUG Loading Platform...
            DEBUG Loading Networking...
            DEBUG Loading Platform...
            DEBUG Loading Pull Secret...
            DEBUG Loading Platform...
            ERROR failed to fetch Metadata: failed to load asset "Install Config": failed to create install config: invalid "install-config.yaml" file: compute[0].platform.openstack.rootVolume.zones: Required value: root volume availability zones must be specified when compute availability zones are specified
            

            Jon Uriarte added a comment - Verified in 4.14.0-0.nightly-2023-08-11-055332. $ openstack availability zone list --compute +-----------+-------------+ | Zone Name | Zone Status | +-----------+-------------+ | internal | available | | AZ0 | available | | AZ1 | available | | AZ2 | available | +-----------+-------------+ $ openstack availability zone list --volume +-----------+-------------+ | Zone Name | Zone Status | +-----------+-------------+ | nova | available | +-----------+-------------+ ** Master node's rootVolume AZ validation Node AZ is set but no rootVolume AZ (from install-config.yaml): compute: - name: worker platform: openstack: zones: [] additionalNetworkIDs: [] type: "m4.xlarge" replicas: 2 controlPlane: name: master platform: openstack: zones: [ 'AZ0' , 'AZ1' , 'AZ2' ] additionalNetworkIDs: [] type: "m4.xlarge" rootVolume: size: 100 type: tripleo zones: [] replicas: 3 The installer raises the validation error: DEBUG OpenShift Installer 4.14.0-0.nightly-2023-08-11-055332 DEBUG Built from commit 0be5579f9c69d275db0600f41fac1686068a62c8 DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Networking... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... ERROR failed to fetch Metadata: failed to load asset "Install Config" : failed to create install config: invalid "install-config.yaml" file: controlPlane.platform.openstack.rootVolume.zones: Required value: root volume availability zones must be specified when compute availability zones are specified   Worker node's rootVolume AZ validation Node AZ is set but no rootVolume AZ (from install-config.yaml): compute: - name: worker platform: openstack: zones: [ 'AZ0' , 'AZ1' , 'AZ2' ] additionalNetworkIDs: [] type: "m4.xlarge" rootVolume: size: 100 type: tripleo zones: [] replicas: 2 controlPlane: name: master platform: openstack: zones: [] additionalNetworkIDs: [] type: "m4.xlarge" replicas: 3 The installer raises the validation error: DEBUG OpenShift Installer 4.14.0-0.nightly-2023-08-11-055332 DEBUG Built from commit 0be5579f9c69d275db0600f41fac1686068a62c8 DEBUG Fetching Metadata... DEBUG Loading Metadata... DEBUG Loading Cluster ID... DEBUG Loading Install Config... DEBUG Loading SSH Key... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Cluster Name... DEBUG Loading Base Domain... DEBUG Loading Platform... DEBUG Loading Networking... DEBUG Loading Platform... DEBUG Loading Pull Secret... DEBUG Loading Platform... ERROR failed to fetch Metadata: failed to load asset "Install Config" : failed to create install config: invalid "install-config.yaml" file: compute[0].platform.openstack.rootVolume.zones: Required value: root volume availability zones must be specified when compute availability zones are specified

              emacchi@redhat.com Emilien Macchi
              emacchi@redhat.com Emilien Macchi
              Jon Uriarte Jon Uriarte
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: