-
Bug
-
Resolution: Done-Errata
-
Major
-
None
-
False
-
None
-
False
-
NEW
-
OBSDA-115 - Create alerting rules based on logs
-
VERIFIED
-
Before this update, the LokiStack ruler did not restart caused when the administrator change the RulerConfig custom resource. With this update, the Loki Operator restarts the Ruler pods on RulerConfig changes resolves the issue.
-
Bug Fix
-
-
-
Log Storage - Sprint 236, Log Storage - Sprint 237, Log Storage - Sprint 238
Description:
Already firing user alerts with openshift-monitoring AM (AlertManager) are not redirected to user-workload-monitoring(UWM) AM URL once user-workload-monitoring is enabled. Need to manually restart the ruler for alerts to be routed to user-workload AM.
MyApplication1LogVolumeIsHigh and MyApplication2LogVolumeIsHigh are the 2 user alerts created for tenant: application
Logs:
ALERT_MANAGER=$(oc get route alertmanager-main -n openshift-monitoring -o jsonpath='{@.spec.host}') $ curl -k -H "Authorization: Bearer $(oc whoami -t)" https://$ALERT_MANAGER/api/v1/alerts | jq '.data[].labels.alertname' % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6641 0 6641 0 0 21989 0 -::- -::- -::- 22063 "InsightsRecommendationActive" "CannotRetrieveUpdates" "MyApplication1LogVolumeIsHigh" "MyApplication2LogVolumeIsHigh" "Watchdog" "AlertmanagerReceiversNotConfigured"
$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 – curl -k -H "Authorization: Bearer $(oc whoami -t)" 'https://alertmanager-user-workload.openshift-user-workload-monitoring.svc:9095/api/v2/alerts' | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 3 100 3 0 0 111 0 -::- -::- -::- 111 []
runtime-config/override:
— overrides: application: ruler_alertmanager_config: alertmanager_url: https://_web._tcp.alertmanager-operated.openshift-user-workload-monitoring.svc enable_alertmanager_v2: true enable_alertmanager_discovery: true alertmanager_refresh_interval: 1m alertmanager_client: tls_ca_path: /var/run/ca/alertmanager/service-ca.crt tls_server_name: alertmanager-user-workload.openshift-user-workload-monitoring.svc.cluster.local type: Bearer credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
Version: Logging 5.7.0 with OCP 4.13
Steps to reproduce:
1) Deploy CLO and LO
2) Forward logs to Loki
3) Create Alertingrules for app tenant
4) Query active/firing alerts on openshift-monitoring AM
5) Enable User-workload-monitoring
6) Validate runtime-config for overrides.
7) Wait for 5-10 minutes
8) Query user-workload AM for alerts
9) Query cluster-monitoring AM for alerts.
How reproducible: Always
Expected Result: User alerts can be queried through user-workload AM once UWM is enabled.
Actual Result: Query with user-workload AM returns empty. Query with openshift-monitoring AM returns user alerts after UWM is enabled.
- links to
-
RHBA-2023:6139
Logging Subsystem 5.8.0 - Red Hat OpenShift
- mentioned on
