-
Bug
-
Resolution: Unresolved
-
Minor
-
rhos-18.0.0
-
None
-
0
-
False
-
-
False
-
?
-
?
-
?
-
?
-
None
-
-
-
2
-
CloudOps 2024 Sprint 21
-
Low
Telemetry can't make progress, yet operator status is "in progress" and no event nor status update on OpenSTackControlPlane CR
# oc get openstackcontrolplane -n openstack NAME STATUS MESSAGE openstack-control-plane False OpenStackControlPlane Telemetry in progress
[root@jumphost 05_control_plane]# oc get events -n openstack | grep -i telem [root@jumphost 05_control_plane]#
# oc describe openstackcontrolplanes.core.openstack.org -A | less
(...)
Message: OpenStackControlPlane Telemetry in progress
Reason: Requested
Severity: Info
Status: False
Type: OpenStackControlPlaneTelemetryReady
Yet, when one checks the logs of the telemetry operator, one can see that this fails:
2024-09-08T15:42:01Z INFO Controllers.MetricStorage Reconciling Service 'metric-storage' {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "c339d5df-b3f7-4f9e-9e7e-91712f3291fb"} 2024-09-08T15:42:01Z INFO Controllers.MetricStorage Can't own MonitoringStack resource {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "c339d5df-b3f7-4f9e-9e7e-91712f3291fb"} 2024-09-08T15:43:01Z INFO Controllers.MetricStorage Reconciling Service 'metric-storage' {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "bbf7a02e-22af-4559-9aaf-136475dcb01d"} 2024-09-08T15:43:01Z INFO Controllers.MetricStorage Can't own MonitoringStack resource {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "bbf7a02e-22af-4559-9aaf-136475dcb01d"} 2024-09-08T15:44:01Z INFO Controllers.MetricStorage Reconciling Service 'metric-storage' {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "d78d6e58-72fe-484f-922f-a58a2bd1437e"} 2024-09-08T15:44:01Z INFO Controllers.MetricStorage Can't own MonitoringStack resource {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "d78d6e58-72fe-484f-922f-a58a2bd1437e"} 2024-09-08T15:45:01Z INFO Controllers.MetricStorage Reconciling Service 'metric-storage' {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "c3481658-79bd-4752-8772-f0d14b012866"} 2024-09-08T15:45:01Z INFO Controllers.MetricStorage Can't own MonitoringStack resource {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "c3481658-79bd-4752-8772-f0d14b012866"} 2024-09-08T15:46:01Z INFO Controllers.MetricStorage Reconciling Service 'metric-storage' {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "d16eeb65-5dbc-44f9-b367-0f234aa7d1a6"} 2024-09-08T15:46:01Z INFO Controllers.MetricStorage Can't own MonitoringStack resource {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "d16eeb65-5dbc-44f9-b367-0f234aa7d1a6"} 2024-09-08T15:47:01Z INFO Controllers.MetricStorage Reconciling Service 'metric-storage' {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "8e6adaa0-4a9d-4c92-a89c-bb2aebb63b9f"} 2024-09-08T15:47:01Z INFO Controllers.MetricStorage Can't own MonitoringStack resource {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "8e6adaa0-4a9d-4c92-a89c-bb2aebb63b9f"} 2024-09-08T15:47:57Z INFO Controllers.Ceilometer Reconciling Service 'ceilometer' {"controller": "ceilometer", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Ceilometer", "Ceilometer": {"name":"ceilometer","namespace":"openstack"}, "namespace": "openstack", "name": "ceilometer", "reconcileID": "4ca19310-fb8f-4949-ad3b-39484bb1125c"} 2024-09-08T15:47:57Z ERROR Reconciler error {"controller": "ceilometer", "controllerGroup": "telemetry.openstack.org", "controllerKind": "Ceilometer", "Ceilometer": {"name":"ceilometer","namespace":"openstack"}, "namespace": "openstack", "name": "ceilometer", "reconcileID": "4ca19310-fb8f-4949-ad3b-39484bb1125c", "error": "internal endpoint not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.6/pkg/internal/controller/controller.go:329 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.6/pkg/internal/controller/controller.go:266 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2 /remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.6/pkg/internal/controller/controller.go:227 2024-09-08T15:48:01Z INFO Controllers.MetricStorage Reconciling Service 'metric-storage' {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "3b8aac26-ff09-4305-a235-dc4832c09ddc"} 2024-09-08T15:48:01Z INFO Controllers.MetricStorage Can't own MonitoringStack resource {"controller": "metricstorage", "controllerGroup": "telemetry.openstack.org", "controllerKind": "MetricStorage", "MetricStorage": {"name":"metric-storage","namespace":"openstack"}, "namespace": "openstack", "name": "metric-storage", "reconcileID": "3b8aac26-ff09-4305-a235-dc4832c09ddc"}
Please make sure to update the OpenStackControlPlane CR with this failure, and please generate events as well. Forcing users to check the various controller logs and not generating events nor updating the CR status is an anti-pattern in OpenShift
Also, let's make sure that these error messages are understandable by human operators and that they clearly state an action that the operator has to do to remediate the issue
The same comment is true for any other of the openstack operators.
Thanks a bunch!
- is related to
-
OSPRH-10084 Add events for Operators
-
- New
-
- links to
