-
Bug
-
Resolution: Done
-
Undefined
-
None
-
1.4.0
-
False
-
-
False
-
-
Description of the problem:
workflow metrics targets down after enabling metrics for sonataflow
All targets are up before enabling sonataflow metrics.
Update the orchestrator CR sonataflowPlatform.monitoring.enabled == true:
How reproducible: 100%
Steps to reproduce:
1. Deploy rhdh with orchestrator 1.4 rc9
2. This is the output from the steps in this document: https://sonataflow.org/serverlessworkflow/main/cloud/operator/monitoring-workflows.html
[root@ocp-edge73 ~]# cat << EOF | oc apply -f - > apiVersion: v1 > kind: ConfigMap > metadata: > name: cluster-monitoring-config > namespace: openshift-monitoring > data: > config.yaml: | > enableUserWorkload: true > EOFconfigmap/cluster-monitoring-config created [root@ocp-edge73 ~]# [root@ocp-edge73 ~]# oc new-project grafana-operator Now using project "grafana-operator" on server "https://api.ocp-edge73-0.qe.lab.redhat.com:6443".You can add applications to this project with the 'new-app' command. For example, try: oc new-app rails-postgresql-exampleto build a new example application in Ruby. Or use kubectl to deploy a simple Kubernetes application: kubectl create deployment hello-node --image=registry.k8s.io/e2e-test-images/agnhost:2.43 -- /agnhost serve-hostname[root@ocp-edge73 ~]# cat << EOF | oc create -f - > apiVersion: operators.coreos.com/v1 > kind: OperatorGroup > metadata: > generateName: grafana-operator- > namespace: grafana-operator > spec: > targetNamespaces: > - grafana-operator > --- > apiVersion: operators.coreos.com/v1alpha1 > kind: Subscription > metadata: > generateName: grafana-operator- > namespace: grafana-operator > spec: > channel: v5 > name: grafana-operator > installPlanApproval: Automatic > source: community-operators > sourceNamespace: openshift-marketplace > EOFoperatorgroup.operators.coreos.com/grafana-operator-knjzk created subscription.operators.coreos.com/grafana-operator-gq9gc created [root@ocp-edge73 ~]# [root@ocp-edge73 ~]# oc -n grafana-operator rollout status \ > deployment grafana-operator-controller-manager-v5 deployment "grafana-operator-controller-manager-v5" successfully rolled out [root@ocp-edge73 ~]# cat << EOF | oc create -f - > apiVersion: grafana.integreatly.org/v1beta1 > kind: Grafana > metadata: > name: grafana > labels: > dashboards: "grafana" > spec: > config: > security: > admin_user: root > admin_password: secret > EOFgrafana.grafana.integreatly.org/grafana created [root@ocp-edge73 ~]# [root@ocp-edge73 ~]# oc adm policy add-cluster-role-to-user cluster-monitoring-view -z grafana-sa clusterrole.rbac.authorization.k8s.io/cluster-monitoring-view added: "grafana-sa" [root@ocp-edge73 ~]# TOKEN=`oc sa new-token grafana-sa` Command "new-token" is deprecated, and will be removed in the future version. Use oc create token instead. [root@ocp-edge73 ~]#
3. At this point the metrics targets for the workflow show up in the OCP UI and have 'Down' status with error:
Get "http://10.128.0.113:8080/q/metrics": context deadline exceeded
Greeting Service Monitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
resourceVersion: '1351810'
name: greeting
uid: 15fbd689-e215-4c26-be15-76e14250241c
creationTimestamp: '2025-02-05T15:55:08Z'
generation: 1
managedFields:
- apiVersion: monitoring.coreos.com/v1
fieldsType: FieldsV1
fieldsV1:
'f:metadata':
'f:labels':
.: {}
'f:app': {}
'f:app.kubernetes.io/component': {}
'f:app.kubernetes.io/managed-by': {}
'f:app.kubernetes.io/name': {}
'f:app.kubernetes.io/part-of': {}
'f:sonataflow.org/workflow-app': {}
'f:sonataflow.org/workflow-namespace': {}
'f:ownerReferences':
.: {}
'k:{"uid":"3b7c1e7c-ad43-4bd3-bdde-48efcc30da3b"}': {}
'f:spec':
.: {}
'f:endpoints': {}
'f:namespaceSelector': {}
'f:selector': {}
manager: manager
operation: Update
time: '2025-02-05T15:55:08Z'
namespace: sonataflow-infra
ownerReferences:
- apiVersion: sonataflow.org/v1alpha08
blockOwnerDeletion: true
controller: true
kind: SonataFlow
name: greeting
uid: 3b7c1e7c-ad43-4bd3-bdde-48efcc30da3b
labels:
app: greeting
app.kubernetes.io/component: serverless-workflow
app.kubernetes.io/managed-by: sonataflow-operator
app.kubernetes.io/name: greeting
app.kubernetes.io/part-of: sonataflow-platform
sonataflow.org/workflow-app: greeting
sonataflow.org/workflow-namespace: sonataflow-infra
spec:
endpoints:
- bearerTokenSecret:
key: ''
path: /q/metrics
port: web
namespaceSelector: {}
selector:
matchLabels:
sonataflow.org/workflow-app: greeting
sonataflow.org/workflow-namespace: sonataflow-infra
Actual results: Get "http://10.128.0.113:8080/q/metrics": context deadline exceeded
Expected results: Metrics links are in Up status
- links to