-
Bug
-
Resolution: Duplicate
-
Normal
-
None
-
4.12.0
-
Quality / Stability / Reliability
-
False
-
-
None
-
Moderate
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
-
None
Description of problem:
as we filed 4.11 bug for kube-state-metrics: https://bugzilla.redhat.com/show_bug.cgi?id=2069543
the same issue exists in 4.12 and triggered APIRemovedInNextEUSReleaseInUse/APIRemovedInNextReleaseInUse alerts
# oc -n openshift-monitoring logs -c kube-state-metrics $(oc -n openshift-monitoring get pod --no-headers | grep kube-state-metrics | awk '{print $1}') | grep HorizontalPodAutoscaler
W1019 07:18:50.931381 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
W1019 07:18:50.936268 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
W1019 07:27:45.938541 1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
...
# token=`oc create token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname=~"APIRemovedInNextEUSReleaseInUse|APIRemovedInNextReleaseInUse",alertstate="firing"}' | jq
{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{
"metric": {
"__name__": "ALERTS",
"alertname": "APIRemovedInNextEUSReleaseInUse",
"alertstate": "firing",
"group": "autoscaling",
"namespace": "openshift-kube-apiserver",
"removed_release": "1.26",
"resource": "horizontalpodautoscalers",
"severity": "info",
"version": "v2beta2"
},
"value": [
1666167509.532,
"1"
]
},
{
"metric": {
"__name__": "ALERTS",
"alertname": "APIRemovedInNextReleaseInUse",
"alertstate": "firing",
"group": "autoscaling",
"namespace": "openshift-kube-apiserver",
"removed_release": "1.26",
"resource": "horizontalpodautoscalers",
"severity": "info",
"version": "v2beta2"
},
"value": [
1666167509.532,
"1"
]
}
]
}
}
and HorizontalPodAutoscaler version is autoscaling/v2
# oc explain horizontalpodautoscalers KIND: HorizontalPodAutoscaler VERSION: autoscaling/v2
Version-Release number of selected component (if applicable):
4.12.0-0.nightly-2022-10-18-192348 kube-state-metrics 2.6.0
How reproducible:
always
Steps to Reproduce:
1. check the alert for the cluster 2. 3.
Actual results:
APIRemovedInNextEUSReleaseInUse/APIRemovedInNextReleaseInUse alerts triggered for horizontalpodautoscalers/v2beta2
Expected results:
no alert
Additional info:
APIRemovedInNextEUSReleaseInUse/APIRemovedInNextReleaseInUse alerts defined in
# oc -n openshift-kube-apiserver get prometheusrules api-usage -oyaml
...
spec:
groups:
- name: pre-release-lifecycle
rules:
- alert: APIRemovedInNextReleaseInUse
annotations:
description: Deprecated API that will be removed in the next version is being
used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version
}}/{{ $labels.resource }} API might be necessary for a successful upgrade
to the next cluster version with Kubernetes {{ $labels.removed_release }}.
Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version
}}.{{ $labels.group }} -o yaml` to identify the workload.
summary: Deprecated API that will be removed in the next version is being
used.
expr: group by (group,version,resource,removed_release) (apiserver_requested_deprecated_apis{removed_release="1.26"})
* on (group,version,resource) group_left () sum by (group,version,resource)
( rate(apiserver_request_total{system_client!="kube-controller-manager",system_client!="cluster-policy-controller"}[4h])
) > 0
for: 1h
labels:
namespace: openshift-kube-apiserver
severity: info
- alert: APIRemovedInNextEUSReleaseInUse
annotations:
description: Deprecated API that will be removed in the next EUS version is
being used. Removing the workload that is using the {{ $labels.group }}.{{
$labels.version }}/{{ $labels.resource }} API might be necessary for a successful
upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release
}}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version
}}.{{ $labels.group }} -o yaml` to identify the workload.
summary: Deprecated API that will be removed in the next EUS version is being
used.
expr: group by (group,version,resource,removed_release) (apiserver_requested_deprecated_apis{removed_release=~"1[.]2[67]"})
* on (group,version,resource) group_left () sum by (group,version,resource)
( rate(apiserver_request_total{system_client!="kube-controller-manager",system_client!="cluster-policy-controller"}[4h])
) > 0
for: 1h
labels:
namespace: openshift-kube-apiserver
severity: info
- links to
