Uploaded image for project: 'Knative Serving'
  1. Knative Serving
  2. SRVKS-186

Custom metrics API stops working after a few hours

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Minor Minor
    • None
    • v0.7.0
    • None
    • Serverless Sprint 169

      The autoscaler starts logging the following error repeatedly after several hours of runtime

      E0717 13:27:28.176500       1 authentication.go:62] Unable to authenticate the request due to an error: x509: certificate signed by unknown authority
      

      The following errors can be seen in the OpenShift apiserver logs:

      E0717 13:29:16.338954       1 memcache.go:141] couldn't get resource list for custom.metrics.k8s.io/v1beta1: Unauthorized
      

      custom.metrics.k8s.io being:

      apiVersion: apiregistration.k8s.io/v1
      kind: APIService
      metadata:
        annotations:
          manifestival: new
        creationTimestamp: "2019-07-16T13:50:44Z"
        labels:
          autoscaling.knative.dev/metric-provider: custom-metrics
          serving.knative.dev/release: devel
        name: v1beta1.custom.metrics.k8s.io
        resourceVersion: "19269"
        selfLink: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.custom.metrics.k8s.io
        uid: b52198cd-a7d0-11e9-b451-066cad9c3470
      spec:
        group: custom.metrics.k8s.io
        groupPriorityMinimum: 100
        insecureSkipTLSVerify: true
        service:
          name: autoscaler
          namespace: knative-serving
        version: v1beta1
        versionPriority: 100
      status:
        conditions:
        - lastTransitionTime: "2019-07-16T13:51:08Z"
          message: all checks passed
          reason: Passed
          status: "True"
          type: Available
      

      While the autoscaler apiservice appears to be available, it apparently isn't functioning as am apiservice, as namespaces are being stuck in Terminating state on the cluster.

              Unassigned Unassigned
              maschmid@redhat.com Marek Schmidt
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: