Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-2566

APIRemovedInNextEUSReleaseInUse/APIRemovedInNextReleaseInUse alerts triggered for horizontalpodautoscalers/v2beta2

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Normal
    • None
    • 4.12.0
    • Monitoring
    • Moderate
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      as we filed 4.11 bug for kube-state-metrics: https://bugzilla.redhat.com/show_bug.cgi?id=2069543

      the same issue exists in 4.12 and triggered APIRemovedInNextEUSReleaseInUse/APIRemovedInNextReleaseInUse alerts

      # oc -n openshift-monitoring logs -c kube-state-metrics $(oc -n openshift-monitoring get pod --no-headers | grep kube-state-metrics | awk '{print $1}') | grep HorizontalPodAutoscaler
      W1019 07:18:50.931381       1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
      W1019 07:18:50.936268       1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
      W1019 07:27:45.938541       1 warnings.go:70] autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
      ...
      
      # token=`oc create token prometheus-k8s -n openshift-monitoring` 
      # oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=ALERTS{alertname=~"APIRemovedInNextEUSReleaseInUse|APIRemovedInNextReleaseInUse",alertstate="firing"}' | jq
      {
        "status": "success",
        "data": {
          "resultType": "vector",
          "result": [
            {
              "metric": {
                "__name__": "ALERTS",
                "alertname": "APIRemovedInNextEUSReleaseInUse",
                "alertstate": "firing",
                "group": "autoscaling",
                "namespace": "openshift-kube-apiserver",
                "removed_release": "1.26",
                "resource": "horizontalpodautoscalers",
                "severity": "info",
                "version": "v2beta2"
              },
              "value": [
                1666167509.532,
                "1"
              ]
            },
            {
              "metric": {
                "__name__": "ALERTS",
                "alertname": "APIRemovedInNextReleaseInUse",
                "alertstate": "firing",
                "group": "autoscaling",
                "namespace": "openshift-kube-apiserver",
                "removed_release": "1.26",
                "resource": "horizontalpodautoscalers",
                "severity": "info",
                "version": "v2beta2"
              },
              "value": [
                1666167509.532,
                "1"
              ]
            }
          ]
        }
      }

      and HorizontalPodAutoscaler version is autoscaling/v2

      # oc explain horizontalpodautoscalers
      KIND:     HorizontalPodAutoscaler
      VERSION:  autoscaling/v2 

      Version-Release number of selected component (if applicable):

      4.12.0-0.nightly-2022-10-18-192348
      kube-state-metrics 2.6.0

       

      How reproducible:

      always

      Steps to Reproduce:

      1. check the alert for the cluster
      2.
      3.
      

      Actual results:

      APIRemovedInNextEUSReleaseInUse/APIRemovedInNextReleaseInUse alerts triggered for horizontalpodautoscalers/v2beta2

      Expected results:

      no alert

      Additional info:

      APIRemovedInNextEUSReleaseInUse/APIRemovedInNextReleaseInUse alerts defined in

      # oc -n openshift-kube-apiserver get prometheusrules api-usage -oyaml
      ...
      spec:
        groups:
        - name: pre-release-lifecycle
          rules:
          - alert: APIRemovedInNextReleaseInUse
            annotations:
              description: Deprecated API that will be removed in the next version is being
                used. Removing the workload that is using the {{ $labels.group }}.{{ $labels.version
                }}/{{ $labels.resource }} API might be necessary for a successful upgrade
                to the next cluster version with Kubernetes {{ $labels.removed_release }}.
                Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version
                }}.{{ $labels.group }} -o yaml` to identify the workload.
              summary: Deprecated API that will be removed in the next version is being
                used.
            expr: group by (group,version,resource,removed_release) (apiserver_requested_deprecated_apis{removed_release="1.26"})
              * on (group,version,resource) group_left () sum by (group,version,resource)
              ( rate(apiserver_request_total{system_client!="kube-controller-manager",system_client!="cluster-policy-controller"}[4h])
              ) > 0
            for: 1h
            labels:
              namespace: openshift-kube-apiserver
              severity: info
          - alert: APIRemovedInNextEUSReleaseInUse
            annotations:
              description: Deprecated API that will be removed in the next EUS version is
                being used. Removing the workload that is using the {{ $labels.group }}.{{
                $labels.version }}/{{ $labels.resource }} API might be necessary for a successful
                upgrade to the next EUS cluster version with Kubernetes {{ $labels.removed_release
                }}. Refer to `oc get apirequestcounts {{ $labels.resource }}.{{ $labels.version
                }}.{{ $labels.group }} -o yaml` to identify the workload.
              summary: Deprecated API that will be removed in the next EUS version is being
                used.
            expr: group by (group,version,resource,removed_release) (apiserver_requested_deprecated_apis{removed_release=~"1[.]2[67]"})
              * on (group,version,resource) group_left () sum by (group,version,resource)
              ( rate(apiserver_request_total{system_client!="kube-controller-manager",system_client!="cluster-policy-controller"}[4h])
              ) > 0
            for: 1h
            labels:
              namespace: openshift-kube-apiserver
              severity: info
      

      Attachments

        Issue Links

          Activity

            People

              jmarcal@redhat.com Joao Marcal
              juzhao@redhat.com Junqi Zhao
              Junqi Zhao Junqi Zhao
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: