Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-30171

/redfish/v1/Managers/1/VirtualMedia has gone from Lenovo servers

    • No
    • 3
    • Metal Platform 252
    • 1
    • False
    • Hide

      * Previously, newer versions of Redfish used Manager resources to deprecate the Uniform Resource Identifier (URI) for the RedFish Virtual Media API. This caused any hardware that used the newer Redfish URI for Virtual Media to not be provisioned. With this release, the Ironic API identifies the correct Redfish URI to deploy for the RedFish Virtual Media API so that hardware relying on either the deprectaed or newer URI could be provisioned. (link: https://issues.redhat.com/browse/OCPBUGS-30171[*OCPBUGS-30171*]
      Show
      * Previously, newer versions of Redfish used Manager resources to deprecate the Uniform Resource Identifier (URI) for the RedFish Virtual Media API. This caused any hardware that used the newer Redfish URI for Virtual Media to not be provisioned. With this release, the Ironic API identifies the correct Redfish URI to deploy for the RedFish Virtual Media API so that hardware relying on either the deprectaed or newer URI could be provisioned. (link: https://issues.redhat.com/browse/OCPBUGS-30171 [* OCPBUGS-30171 *]
    • Bug Fix
    • Done

      Description of problem:

      Attempting to add Lenovo ThinkSystem SR675 V3 server to NodePool for future add it into Hosted Control PLanes Openshift node pool. ACM attempts to reboot server and checks if VirtualMedia ejected. Ironic checks if /redfish/v1/Managers/1 has VirtualMedia attribute and crashes.
      
      2024-03-01 10:04:23.812 1 DEBUG sushy.connector [None req-1dc15b4e-24c3-49ff-a93f-5dfa55351560 - - - - - -] HTTP response for GET https://129.40.92.61:443/redfish/v1/Managers/1: status code: 200 _op /usr/lib/python3.9/site-packages/sushy/connector.py:2832024-03-01 10:04:23.812 1 DEBUG sushy.resources.base [None req-1dc15b4e-24c3-49ff-a93f-5dfa55351560 - - - - - -] Received representation of Manager /redfish/v1/Managers/1: {'_actions': {'reset': {'allowed_values': ['GracefulRestart', 'ForceRestart'], 'operation_apply_time_support': None, 'target_uri': '/redfish/v1/Managers/1/Actions/Manager.Reset'}}, '_oem_vendors': ['Lenovo'], 'auto_dst_enabled': False, 'command_shell': {'connect_types_supported': ['SSH'], 'max_concurrent_sessions': 2, 'service_enabled': True}, 'description': 'This resource is used to represent a management subsystem for a Redfish implementation.', 'firmware_version': 'QGX314J 3.10 2023-09-15', 'graphical_console': {'connect_types_supported': ['KVMIP'], 'max_concurrent_sessions': 6, 'service_enabled': True}, 'identity': '1', 'links': {'oem_vendors': None}, 'manager_type': <ManagerType.BMC: 'BMC'>, 'model': 'Lenovo XClarity Controller 2', 'name': 'Manager', 'serial_console': {'connect_types_supported': ['IPMI', 'SSH'], 'max_concurrent_sessions': 2, 'service_enabled': True}, 'uuid': '6F1D76DE-BE9E-11EE-8CC4-0A3A88FFF8E0'} refresh /usr/lib/python3.9/site-packages/sushy/resources/base.py:6942024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager [None req-1dc15b4e-24c3-49ff-a93f-5dfa55351560 - - - - - -] Error in tear_down of node a16830d1-6ce2-44fa-ae6b-5a6fcad5ca14: The attribute VirtualMedia is missing from the resource /redfish/v1/Managers/1: sushy.exceptions.MissingAttributeError: The attribute VirtualMedia is missing from the resource /redfish/v1/Managers/12024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager Traceback (most recent call last):2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager   File "/usr/lib/python3.9/site-packages/ironic/conductor/manager.py", line 1083, in _do_node_tear_down2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager     task.driver.deploy.clean_up(task)2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager   File "/usr/lib/python3.9/site-packages/ironic_lib/metrics.py", line 60, in wrapped2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager     result = f(*args, **kwargs)2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/agent_base.py", line 773, in clean_up2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager     task.driver.boot.clean_up_ramdisk(task)2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/redfish/boot.py", line 638, in clean_up_ramdisk2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager     self._eject_all(task)2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/redfish/boot.py", line 728, in _eject_all2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager     _eject_vmedia(task, managers, sushy.VIRTUAL_MEDIA_CD)2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager   File "/usr/lib/python3.9/site-packages/ironic/drivers/modules/redfish/boot.py", line 267, in _eject_vmedia2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager     for v_media in manager.virtual_media.get_members():2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager   File "/usr/lib/python3.9/site-packages/sushy/utils.py", line 233, in func_wrapper2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager     cache_attr_val = res_accessor_method(res_selfie)2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager   File "/usr/lib/python3.9/site-packages/sushy/resources/manager/manager.py", line 196, in virtual_media2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager     self._conn, utils.get_sub_resource_path_by(self, 'VirtualMedia'),2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager   File "/usr/lib/python3.9/site-packages/sushy/utils.py", line 105, in get_sub_resource_path_by2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager     raise exceptions.MissingAttributeError(2024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager sushy.exceptions.MissingAttributeError: The attribute VirtualMedia is missing from the resource /redfish/v1/Managers/12024-03-01 10:04:23.813 1 ERROR ironic.conductor.manager 

      Version-Release number of selected component (if applicable):

          OCP 4.14

      How reproducible:

          Everytime

      Steps to Reproduce:

          1. Add Lenovo ThinkSystem SR675 V3 / BMC Version 3.10 (Build ID: QGX314J) into the NodePool
          2.
          3.
          

      Actual results:

          Host Inventory UI changes node state in Provisioning fisrt and then - into Error state 

      Expected results:

          

      Additional info:

          Host is Provisioned

            [OCPBUGS-30171] /redfish/v1/Managers/1/VirtualMedia has gone from Lenovo servers

            Since the problem described in this issue should be resolved in a recent advisory, it has been closed.

            For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below.

            If the solution does not work for you, open a new bug report.
            https://access.redhat.com/errata/RHSA-2024:0041

            Errata Tool added a comment - Since the problem described in this issue should be resolved in a recent advisory, it has been closed. For information on the advisory (Critical: OpenShift Container Platform 4.16.0 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:0041

            Verified based on code presence and No regression:

            oc get clusterversion
            NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
            version   4.16.0-0.nightly-2024-04-16-195622   True        False         27m     Cluster version is 4.16.0-0.nightly-2024-04-16-195622

            oc get nodes
            NAME                                       STATUS   ROLES                  AGE   VERSION
            openshift-qe-013.arm.eng.rdu2.redhat.com   Ready    control-plane,master   88m   v1.29.3+5fa1806
            openshift-qe-014.arm.eng.rdu2.redhat.com   Ready    control-plane,master   87m   v1.29.3+5fa1806
            openshift-qe-015.arm.eng.rdu2.redhat.com   Ready    control-plane,master   88m   v1.29.3+5fa1806
            openshift-qe-016.arm.eng.rdu2.redhat.com   Ready    worker                 36m   v1.29.3+5fa1806
            openshift-qe-023.arm.eng.rdu2.redhat.com   Ready    worker                 38m   v1.29.3+5fa1806

            Jad Haj Yahya added a comment - Verified based on code presence and No regression: oc get clusterversion NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS version   4.16.0-0.nightly-2024-04-16-195622   True        False         27m     Cluster version is 4.16.0-0.nightly-2024-04-16-195622 oc get nodes NAME                                       STATUS   ROLES                  AGE   VERSION openshift-qe-013.arm.eng.rdu2.redhat.com   Ready    control-plane,master   88m   v1.29.3+5fa1806 openshift-qe-014.arm.eng.rdu2.redhat.com   Ready    control-plane,master   87m   v1.29.3+5fa1806 openshift-qe-015.arm.eng.rdu2.redhat.com   Ready    control-plane,master   88m   v1.29.3+5fa1806 openshift-qe-016.arm.eng.rdu2.redhat.com   Ready    worker                 36m   v1.29.3+5fa1806 openshift-qe-023.arm.eng.rdu2.redhat.com   Ready    worker                 38m   v1.29.3+5fa1806

            jadha you check check the metal3-ironic container in the metal3 pod. It must contain both changes in /usr/lib/python3.9/site-packages.

            Dmitry Tantsur added a comment - jadha you check check the metal3-ironic container in the metal3 pod. It must contain both changes in /usr/lib/python3.9/site-packages.

            jadha I don't remember atm,
            but it would something similar on how you did the verification of https://issues.redhat.com/browse/OCPBUGS-15235

            Iury Gregory Melo Ferreira added a comment - jadha I don't remember atm, but it would something similar on how you did the verification of https://issues.redhat.com/browse/OCPBUGS-15235

            Hi imelofer After having a deployment with 4.16.0-0.nightly-2024-04-05-052913, where should I check code verification, is it in metal3 related pods

            or maybe by spawning sushy-tools/ironic container and checking there. if such, what are the digests to be used ?

            Jad Haj Yahya added a comment - Hi imelofer After having a deployment with 4.16.0-0.nightly-2024-04-05-052913, where should I check code verification, is it in metal3 related pods or maybe by spawning sushy-tools/ironic container and checking there. if such, what are the digests to be used ?

            jadha Hi, can you help verify this bug? We will do a code verification only (unless you have access to the hardware Lenovo ThinkSystem SR675 V3 / BMC Version 3.10 (Build ID: QGX314J) )
            The nightly build 4.16.0-0.nightly-2024-04-05-052913 should already contain the fixes

            Thanks!

            Iury Gregory Melo Ferreira added a comment - jadha Hi, can you help verify this bug? We will do a code verification only (unless you have access to the hardware Lenovo ThinkSystem SR675 V3 / BMC Version 3.10 (Build ID: QGX314J) ) The nightly build 4.16.0-0.nightly-2024-04-05-052913 should already contain the fixes ironic needs to include the code from this commit https://github.com/openshift/openstack-ironic/commit/6d046ad7b403f3ea9c6b0fcdde52968794d4cb3a sushy needs to include the code from this commit https://github.com/openshift/openstack-sushy/commit/a9c4a8e7c2e8ed242f038f0c56476d4a578a88f4 Thanks!

            Hi imelofer,

            Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

            OpenShift Jira Bot added a comment - Hi imelofer , Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.

            Marking as triaged, Winicius (my mentee) is working on the ironic change upstream.
            I will start looking at the sushy backports in the meantime.

            Iury Gregory Melo Ferreira added a comment - Marking as triaged, Winicius (my mentee) is working on the ironic change upstream. I will start looking at the sushy backports in the meantime.

              imelofer Iury Gregory Melo Ferreira
              dmitry.zhukovski@ibm.com Dmitry Zhukovski
              Jad Haj Yahya Jad Haj Yahya
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: