Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-17679

"Operation cannot be fulfilled on *.monitoring.coreos.com" error logs

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Minor Minor
    • None
    • 4.14.0
    • Monitoring
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Low
    • No
    • None
    • None
    • None
    • MON Sprint 243, MON Sprint 245, MON Sprint 248
    • 3
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      4.14.0-0.nightly-2023-08-11-055332 cluster, enable UWM, then will see error log "Operation cannot be fulfilled on thanosrulers.monitoring.coreos.com \"user-workload\"" in UWM prometheus-operator, the error does not affect the function

      $ oc logs deployment/prometheus-operator -c prometheus-operator -n openshift-user-workload-monitoring
      level=info ts=2023-08-14T02:47:16.060727889Z caller=main.go:239 msg="Starting Prometheus Operator" version="(version=0.67.1, branch=rhaos-4.14-rhel-8, revision=f717460)"
      level=info ts=2023-08-14T02:47:16.060802209Z caller=main.go:240 build_context="(go=go1.20.5 X:strictfipsruntime, platform=linux/arm64, user=root, date=20230809-05:28:40, tags=strictfipsruntime)"
      level=warn ts=2023-08-14T02:47:16.081655528Z caller=main.go:107 msg="scrapeconfigs CRD not supported" reason="unmet prerequisites: scrapeconfigs resource (apiGroup: \"monitoring.coreos.com/v1alpha1\") not installed"
      level=info ts=2023-08-14T02:47:16.083679328Z caller=operator.go:356 component=prometheusoperator msg="Kubernetes API capabilities" endpointslices=true
      level=warn ts=2023-08-14T02:47:16.084642488Z caller=main.go:107 msg="prometheusagents CRD not supported" reason="unmet prerequisites: prometheusagents resource (apiGroup: \"monitoring.coreos.com/v1alpha1\") not installed"
      level=info ts=2023-08-14T02:47:16.086469088Z caller=main.go:119 msg="Starting insecure server on 127.0.0.1:8080"
      level=info ts=2023-08-14T02:47:16.087361888Z caller=operator.go:313 component=thanosoperator msg="connection established" cluster-version=v1.27.4+deb2c60
      level=info ts=2023-08-14T02:47:16.087387448Z caller=operator.go:322 component=thanosoperator msg="CRD API endpoints ready"
      level=info ts=2023-08-14T02:47:16.087435408Z caller=operator.go:481 component=prometheusoperator msg="connection established" cluster-version=v1.27.4+deb2c60
      level=info ts=2023-08-14T02:47:16.087438928Z caller=operator.go:476 component=alertmanageroperator msg="connection established" cluster-version=v1.27.4+deb2c60
      level=info ts=2023-08-14T02:47:16.087448608Z caller=operator.go:490 component=prometheusoperator msg="CRD API endpoints ready"
      level=info ts=2023-08-14T02:47:16.087456528Z caller=operator.go:485 component=alertmanageroperator msg="CRD API endpoints ready"
      level=info ts=2023-08-14T02:47:16.187822123Z caller=operator.go:274 component=thanosoperator msg="successfully synced all caches"
      level=info ts=2023-08-14T02:47:16.188243283Z caller=operator.go:413 component=prometheusoperator msg="successfully synced all caches"
      level=info ts=2023-08-14T02:47:16.201367603Z caller=operator.go:307 component=alertmanageroperator msg="successfully synced all caches"
      level=info ts=2023-08-14T02:47:18.442283267Z caller=operator.go:1189 component=prometheusoperator key=openshift-user-workload-monitoring/user-workload msg="sync prometheus"
      level=info ts=2023-08-14T02:47:18.528044223Z caller=operator.go:1189 component=prometheusoperator key=openshift-user-workload-monitoring/user-workload msg="sync prometheus"
      level=info ts=2023-08-14T02:47:18.646051018Z caller=operator.go:1189 component=prometheusoperator key=openshift-user-workload-monitoring/user-workload msg="sync prometheus"
      level=info ts=2023-08-14T02:47:18.851978809Z caller=operator.go:1189 component=prometheusoperator key=openshift-user-workload-monitoring/user-workload msg="sync prometheus"
      level=info ts=2023-08-14T02:47:18.913692487Z caller=operator.go:552 component=thanosoperator key=openshift-user-workload-monitoring/user-workload msg="sync thanos-ruler"
      level=info ts=2023-08-14T02:47:18.952845965Z caller=operator.go:659 component=thanosoperator msg="StatefulSet not found" key=openshift-user-workload-monitoring/thanos-ruler-user-workload
      level=info ts=2023-08-14T02:47:18.965643124Z caller=operator.go:552 component=thanosoperator key=openshift-user-workload-monitoring/user-workload msg="sync thanos-ruler"
      level=info ts=2023-08-14T02:47:18.966071804Z caller=operator.go:659 component=thanosoperator msg="StatefulSet not found" key=openshift-user-workload-monitoring/thanos-ruler-user-workload
      level=info ts=2023-08-14T02:47:19.016365442Z caller=operator.go:552 component=thanosoperator key=openshift-user-workload-monitoring/user-workload msg="sync thanos-ruler"
      level=error ts=2023-08-14T02:47:19.116527918Z caller=klog.go:116 component=k8s_client_runtime func=ErrorDepth msg="status \"openshift-user-workload-monitoring/user-workload\" failed: failed to update status subresource: Operation cannot be fulfilled on thanosrulers.monitoring.coreos.com \"user-workload\": the object has been modified; please apply your changes to the latest version and try again"
      level=info ts=2023-08-14T02:47:28.931009098Z caller=operator.go:1189 component=prometheusoperator key=openshift-user-workload-monitoring/user-workload msg="sync prometheus"
      level=info ts=2023-08-14T02:47:28.953216857Z caller=operator.go:552 component=thanosoperator key=openshift-user-workload-monitoring/user-workload msg="sync thanos-ruler"
      level=info ts=2023-08-14T02:47:29.006615294Z caller=operator.go:552 component=thanosoperator key=openshift-user-workload-monitoring/user-workload msg="sync thanos-ruler"
      level=info ts=2023-08-14T02:47:29.017325574Z caller=operator.go:1189 component=prometheusoperator key=openshift-user-workload-monitoring/user-workload msg="sync prometheus"
      level=info ts=2023-08-14T02:47:38.465331049Z caller=operator.go:1189 component=prometheusoperator key=openshift-user-workload-monitoring/user-workload msg="sync prometheus"
      level=info ts=2023-08-14T02:47:38.556070685Z caller=operator.go:1189 component=prometheusoperator key=openshift-user-workload-monitoring/user-workload msg="sync prometheus"
      
      
      $ oc get thanosrulers --all-namespaces
      NAMESPACE                            NAME            VERSION   REPLICAS   READY   RECONCILED   AVAILABLE   AGE
      openshift-user-workload-monitoring   user-workload             2          2       True         True        4h12m

      Version-Release number of selected component (if applicable):

      4.14.0-0.nightly-2023-08-11-055332

      How reproducible:

      always

              mariofer@redhat.com Mario Fernandez Herrero
              juzhao@redhat.com Junqi Zhao
              None
              Simon Pasquier
              Junqi Zhao Junqi Zhao
              None
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: