Uploaded image for project: 'Cluster Observability Operator'
  1. Cluster Observability Operator
  2. COO-1051

After upgrade, Content of UIplugin monitoring disappeared

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Critical Critical
    • 1.3.0 RC
    • 1.2.1
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • 1
    • None
    • None
    • None
    • Sprint 276
    • None
    • None
    • None

      Install COO 1.2.0
      Create UIplugin as below following doc https://github.com/rhobs/observability-operator/blob/main/docs/user-guides/observability-ui-plugins.md#monitoring and all the dashboards, deploy logging, tracing
      Install ACM following doc https://docs.google.com/document/d/1BqWYq907iNHJYqDGsNROSglBvIhFQgUXrHIhMk7df9c/edit?tab=t.0 

      % oc -n open-cluster-management get csv
      NAME                                    DISPLAY                                      VERSION   REPLACES                                PHASE
      advanced-cluster-management.v2.13.3     Advanced Cluster Management for Kubernetes   2.13.3    advanced-cluster-management.v2.13.2     Succeeded
      cluster-observability-operator.v1.2.1   Cluster Observability Operator               1.2.1     cluster-observability-operator.v1.2.0   Succeeded 
      apiVersion: observability.openshift.io/v1alpha1
      kind: UIPlugin
      metadata:
        name: monitoring
      spec:
        type: Monitoring
        monitoring:
          acm:
            enabled: true
            alertmanager:
              url: 'https://alertmanager.open-cluster-management-observability.svc:9095'
            thanosQuerier:
              url: 'https://rbac-query-proxy.open-cluster-management-observability.svc:8443'
          perses:
            enabled: true
          incidents:
            enabled: true 

      Perform upgrade to COO 1.2.1, upgrade successfully

       % oc get csv
      NAME                                    DISPLAY                          VERSION     REPLACES                                PHASE
      cluster-logging.v6.2.3                  Red Hat OpenShift Logging        6.2.3       cluster-logging.v6.2.2                  Succeeded
      cluster-observability-operator.v1.2.1   Cluster Observability Operator   1.2.1       cluster-observability-operator.v1.2.0   Succeeded
      loki-operator.v6.2.3                    Loki Operator                    6.2.3       loki-operator.v6.2.2                    Succeeded
      opentelemetry-operator.v0.127.0-1       Red Hat build of OpenTelemetry   0.127.0-1   opentelemetry-operator.v0.119.0-2       Succeeded
      tempo-operator.v0.16.0-1                Tempo Operator                   0.16.0-1    tempo-operator.v0.15.4-1                Succeeded 

       

      After upgrade, Content of UIplugin monitoring disppeared

      oc get uiplugin monitoring -oyaml
      apiVersion: observability.openshift.io/v1alpha1
      kind: UIPlugin
      metadata:
        annotations:
          kubectl.kubernetes.io/last-applied-configuration: |
            {"apiVersion":"observability.openshift.io/v1alpha1","kind":"UIPlugin","metadata":{"annotations":{"observability.openshift.io/api-support":"TechPreview"},"name":"monitoring"},"spec":{"type":"Monitoring"}}
          observability.openshift.io/api-support: TechPreview
        creationTimestamp: "2025-07-08T03:31:17Z"
        finalizers:
        - uiplugin.observability.openshift.io/finalizer
        generation: 2
        name: monitoring
        resourceVersion: "192156"
        uid: 30cdb585-413c-42d8-8b84-6759e25cc918
      spec:
        type: Monitoring 

      All the pods, such as perses, health-analyzer and troubleshooting are running

      % oc get pod
      NAME                                                         READY   STATUS    RESTARTS   AGE
      distributed-tracing-6cff77b7d-tcf7f                          1/1     Running   0          22m
      health-analyzer-6f499ffcf5-pslzt                             2/2     Running   0          22m
      korrel8r-564cdc6b99-6ln48                                    1/1     Running   0          22m
      logging-d87c468c7-nxrqh                                      1/1     Running   0          22m
      monitoring-5f7f69bb65-cnjh6                                  1/1     Running   0          22m
      obo-prometheus-operator-56bd84ffcc-kgxl4                     1/1     Running   0          22m
      obo-prometheus-operator-admission-webhook-6b95f87dd4-qjr6p   1/1     Running   0          22m
      obo-prometheus-operator-admission-webhook-6b95f87dd4-xs7k5   1/1     Running   0          22m
      observability-operator-c6f65ccb9-jj7jj                       1/1     Running   0          22m
      observability-ui-dashboards-55fd9c95cb-tzq4c                 1/1     Running   0          22m
      perses-0                                                     1/1     Running   0          3h46m
      perses-operator-69548c9d84-44xd8                             1/1     Running   0          22m
      troubleshooting-panel-55774cb887-sdlck                       1/1     Running   0          22m 

      After upgrade, see Error in logs of pod observability-operator 'observability-ui failed to reconcile plugin'

      oc logs observability-operator-c6f65ccb9-jj7jj
      ......
      2025-07-08T07:06:42Z ERROR observability-ui failed to reconcile plugin {"plugin": {"name":"monitoring"}, "error": "monitoring configuration can not be empty for plugin type Monitoring"}
      github.com/rhobs/observability-operator/pkg/controllers/uiplugin.resourceManager.Reconcile
       /workspace/pkg/controllers/uiplugin/controller.go:263
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
       /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:119
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
       /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:334
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
       /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
       /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255
      2025-07-08T07:06:42Z ERROR Reconciler error {"controller": "uiplugin", "controllerGroup": "observability.openshift.io", "controllerKind": "UIPlugin", "UIPlugin": {"name":"monitoring"}, "namespace": "", "name": "monitoring", "reconcileID": "c16d7cb3-4403-42fc-8d93-b7b52aee17eb", "error": "monitoring configuration can not be empty for plugin type Monitoring"}
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
       /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:347
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
       /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:294
      sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.2
       /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.20.4/pkg/internal/controller/controller.go:255
      

       

              rh-ee-coquadro Coleen Iona Quadros
              hongyli@redhat.com Hongyan Li
              None
              Evelyn Murasaki
              None
              None
              None
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: