In OpenShift 4.10 (and backported to 4.9, I think 4.9.13?) we introduced some new changes to sushy in an attempt to maximise compatibility with newer hardware platforms via Redfish with better eTag management, however we accidentally broke compatibility with older hardware platforms, namely HP Gen9 via iLO4 systems, and I believe some other HPE hardware has been affected also. This has caused a regression and has resulted in customers no longer being able to manage or provision new clusters with these systems.

We've validated that a test patch (https://review.opendev.org/c/openstack/sushy/+/856123) is able to workaround this problem with a failback to no eTag validation and should allow customers to continue to utilise their systems.

4.9.13+ we believe.

Consistently

1. Attempt an openshift-install with 4.9.13+ via redfish
2. Validate that nodes cannot be provisioned due to 412 errors (eTag)
3. Alternatively bring a HP Gen9 system into management on a pre-installed cluster, or upgrade to 4.9.13+ on a previously working environment with such hardware.

2022-07-10 09:25:03.872 1 WARNING sushy.exceptions [req-363d0a13-99d4-44c2-8e7c-928b325f9b75 ironic-user - - - -] Error response from PATCH https://<ip-address>/redfish/v1/Systems/1/ with status code 412 has no JSON body: simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2022-07-10 09:25:03.873 1 DEBUG sushy.exceptions [req-363d0a13-99d4-44c2-8e7c-928b325f9b75 ironic-user - - - -] HTTP response for PATCH https://<ip-address>/redfish/v1/Systems/1/: status code: 412, error: unknown error, extended: none __init__ /usr/lib/python3.6/site-packages/sushy/exceptions.py:122
2022-07-10 09:25:03.873 1 ERROR ironic.drivers.modules.redfish.management [req-363d0a13-99d4-44c2-8e7c-928b325f9b75 ironic-user - - - -] Setting boot mode to uefi failed for node 5abf3a2c-c662-48c3-a509-613e1d47606b. Error: HTTP PATCH https://<ip-address>/redfish/v1/Systems/1/ returned code 412. unknown error Extended information: none: sushy.exceptions.HTTPError: HTTP PATCH https://<ip-address>/redfish/v1/Systems/1/ returned code 412. unknown error Extended information: none
2022-07-10 09:25:03.874 1 INFO ironic.drivers.modules.redfish.management [req-363d0a13-99d4-44c2-8e7c-928b325f9b75 ironic-user - - - -] Attempt to set boot mode on node 5abf3a2c-c662-48c3-a509-613e1d47606b failed to set boot mode as the node does not appear to support overriding the boot mode. Possibly partial Redfish implementation?

Node can be managed just fine :-)

Potential fix: https://review.opendev.org/c/openstack/sushy/+/856123

Similar/linked issues: https://bugzilla.redhat.com/show_bug.cgi?id=2084059, https://bugzilla.redhat.com/show_bug.cgi?id=2103710, https://issues.redhat.com/browse/OCPBUGS-602, and https://issues.redhat.com/browse/METAL-343