-
Bug
-
Resolution: Cannot Reproduce
-
Normal
-
rhos-18.0 Beta
-
None
It looks like there is a race condition that happens quite rarely - the ironic-operator get's "stuck" in a inspector reconcile loop and never completes.
The following log entries are repeated over and over in the controller manager log:
2024-06-29T11:02:18Z INFO Controllers.IronicInspector Applied new databasehostname openstack.openstack.svc to MariaDBDatabase ironic-inspector {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector" , "IronicInspector": {"name":"ironic-inspector","namespace":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "6afd1271-0676-4a36-93bc-c136c1d2c3a5"} 2024-06-29T11:02:18Z INFO Controllers.IronicInspector Reconciling Ironic Inspector init {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector", "IronicInspector": {"name":"ironic-inspector","namespa ce":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "6afd1271-0676-4a36-93bc-c136c1d2c3a5"} 2024-06-29T11:02:18Z INFO Controllers.IronicInspector Service ironic-inspector-public - updated {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector", "IronicInspector": {"name":"ironic-inspector", "namespace":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "6afd1271-0676-4a36-93bc-c136c1d2c3a5"} 2024-06-29T11:02:18Z INFO Controllers.IronicInspector Service ironic-inspector-internal - updated {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector", "IronicInspector": {"name":"ironic-inspector", "namespace":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "6afd1271-0676-4a36-93bc-c136c1d2c3a5"} 2024-06-29T11:02:18Z INFO Controllers.IronicInspector Reconciled Ironic Inspector init successfully {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector", "IronicInspector": {"name":"ironic-inspector", "namespace":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "6afd1271-0676-4a36-93bc-c136c1d2c3a5"} 2024-06-29T11:02:18Z INFO Controllers.IronicInspector Reconciling Ironic Inspector Service update {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector", "IronicInspector": {"name":"ironic-inspector", "namespace":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "6afd1271-0676-4a36-93bc-c136c1d2c3a5"} 2024-06-29T11:02:18Z INFO Controllers.IronicInspector Reconciling Ironic Inspector Service upgrade {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector", "IronicInspector": {"name":"ironic-inspector", "namespace":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "6afd1271-0676-4a36-93bc-c136c1d2c3a5"} 2024-06-29T11:02:18Z INFO Controllers.IronicInspector Found ingress domain: apps.ocp.openstack.lab {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector", "IronicInspector": {"name":"ironic-inspector", "namespace":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "6afd1271-0676-4a36-93bc-c136c1d2c3a5"} 2024-06-29T11:02:18Z INFO Controllers.IronicInspector StatefulSet ironic-inspector - updated {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector", "IronicInspector": {"name":"ironic-inspector","namespa ce":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "6afd1271-0676-4a36-93bc-c136c1d2c3a5"} 2024-06-29T11:02:28Z INFO Controllers.IronicInspector Reconciling Ironic Inspector {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector", "IronicInspector": {"name":"ironic-inspector","namespace":"ope nstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "27119178-2af8-430b-8796-baa27447ccec"} 2024-06-29T11:02:28Z INFO Controllers.IronicInspector Successfully ensured MariaDBAccount ironic-inspector exists; database username is ironic_inspector_55d5 {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicIn spector", "IronicInspector": {"name":"ironic-inspector","namespace":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "27119178-2af8-430b-8796-baa27447ccec", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectN ame": "ironic-inspector"}
In the attached log file it enters this state of endlessly looping the same reconcile tasks at timestamp: 2024-06-29T10:24:00Z
Prior to that timestamp it is also repeating the same tasks in Controllers.IronicInspector, but there is also things executing in Controllers.Ironic.
A workaround is to delete the ironic-inspector resource, this causes the operators to first delete inspector and then it is re-applied - and successfully reconciles.
oc delete -n openstack ironicinspectors.ironic.openstack.org ironic-inspector
2024-06-29T11:02:29Z INFO Controllers.IronicInspector Reconciling Ironic Inspector delete {"controller": "ironicinspector", "controllerGroup": "ironic.openstack.org", "controllerKind": "IronicInspector", "IronicInspector": {"name":"ironic-inspector","namespace":"openstack"}, "namespace": "openstack", "name": "ironic-inspector", "reconcileID": "5ba7c63e-5b29-4184-8218-1f4e9194e90f"}
2024-06-29T11:02:29Z INFO Controllers.Ironic Reconciling Service {"controller": "ironic", "controllerGroup": "ironic.openstack.org", "controllerKind": "Ironic", "Ironic": {"name":"ironic","namespace":"openstack"}, "namespace": "openstack", "name": "ironic", "reconcileID": "658a416c-0500-4509-b455-5686339d8d57"}
