Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-61110

metrics-server pods are CrashLoopBackOff in 4.15 TechPreview cluster for panic $root.info is missing required property: version

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.15.z
    • kube-apiserver
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      installer QE found in 4.15 TechPreview IBMCloud cluster, metrics-server pods are CrashLoopBackOff due to panic

      unable to get openapi models: [3,11] $root.info is missing required property: version 

      see from must-gather: https://gcsweb-qe-private-deck-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/qe-private-deck/logs/periodic-ci-openshift-openshift-tests-private-release-4.15-amd64-nightly-ibmcloud-ipi-tp-f28/1960016175710605312/artifacts/ibmcloud-ipi-tp-f28/gather-must-gather/artifacts/

      $ omc get infrastructure/cluster -o jsonpath={.spec.platformSpec.type}
      IBMCloud
      
      $ omc get clusterversion
      NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
      version             False       True          57m     Unable to apply 4.15.0-0.nightly-2025-08-20-015949: the cluster operator monitoring is not available
      
      $ omc get featuregate/cluster -oyaml
      apiVersion: config.openshift.io/v1
      kind: FeatureGate
      ...
      spec:
        featureSet: TechPreviewNoUpgrade
      status:
        featureGates:
        ...
          enabled:
        ...
          - name: MetricsServer
        ...
      
      $ omc -n openshift-monitoring get pod | grep metrics-server
      metrics-server-5558cb4954-dlvps                         0/1     CrashLoopBackOff   15         57m
      metrics-server-5558cb4954-rlxl4                         0/1     CrashLoopBackOff   15         57m
      
      $ omc -n openshift-monitoring logs metrics-server-5558cb4954-dlvps
      2025-08-25T18:08:50.906990714Z Error: unable to get openapi models: [3,11] $root.info is missing required property: version
      2025-08-25T18:08:50.907435574Z Usage:
      2025-08-25T18:08:50.907435574Z    [flags]
      2025-08-25T18:08:50.907435574Z 
      2025-08-25T18:08:50.907435574Z Metrics server flags:
      2025-08-25T18:08:50.907435574Z 
      2025-08-25T18:08:50.907435574Z       --kubeconfig string            The path to the kubeconfig used to connect to the Kubernetes API server and the Kubelets (defaults to in-cluster config)
      2025-08-25T18:08:50.907435574Z       --metric-resolution duration   The resolution at which metrics-server will retain metrics, must set value at least 10s. (default 1m0s)
      2025-08-25T18:08:50.907435574Z       --version                      Show version
      2025-08-25T18:08:50.907435574Z 
      2025-08-25T18:08:50.907435574Z Kubelet client flags:
      2025-08-25T18:08:50.907435574Z 
      2025-08-25T18:08:50.907435574Z       --deprecated-kubelet-completely-insecure    DEPRECATED: Do not use any encryption, authorization, or authentication when communicating with the Kubelet. This is rarely the right option, since it leaves kubelet communication completely insecure.  If you encounter auth errors, make sure you've enabled token webhook auth on the Kubelet, and if you're in a test cluster with self-signed Kubelet certificates, consider using kubelet-insecure-tls instead.
      2025-08-25T18:08:50.907435574Z       --kubelet-certificate-authority string      Path to the CA to use to validate the Kubelet's serving certificates.
      2025-08-25T18:08:50.907435574Z       --kubelet-client-certificate string         Path to a client cert file for TLS.
      2025-08-25T18:08:50.907435574Z       --kubelet-client-key string                 Path to a client key file for TLS.
      2025-08-25T18:08:50.907435574Z       --kubelet-insecure-tls                      Do not verify CA of serving certificates presented by Kubelets.  For testing purposes only.
      2025-08-25T18:08:50.907435574Z       --kubelet-port int                          The port to use to connect to Kubelets. (default 10250)
      2025-08-25T18:08:50.907435574Z       --kubelet-preferred-address-types strings   The priority of node address types to use when determining which address to use to connect to a particular node (default [Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP])
      2025-08-25T18:08:50.907435574Z       --kubelet-use-node-status-port              Use the port in the node status. Takes precedence over --kubelet-port flag.
      2025-08-25T18:08:50.907435574Z 
      2025-08-25T18:08:50.907435574Z Apiserver secure serving flags:
      2025-08-25T18:08:50.907435574Z 
      2025-08-25T18:08:50.907435574Z       --bind-address ip                        The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces will be used. (default 0.0.0.0)
      2025-08-25T18:08:50.907435574Z       --cert-dir string                        The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
      2025-08-25T18:08:50.907435574Z       --http2-max-streams-per-connection int   The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.
      2025-08-25T18:08:50.907435574Z       --permit-address-sharing                 If true, SO_REUSEADDR will be used when binding the port. This allows binding to wildcard IPs like 0.0.0.0 and specific IPs in parallel, and it avoids waiting for the kernel to release sockets in TIME_WAIT state. [default=false]
      2025-08-25T18:08:50.907435574Z       --permit-port-sharing                    If true, SO_REUSEPORT will be used when binding the port, which allows more than one instance to bind on the same address and port. [default=false]
      2025-08-25T18:08:50.907435574Z       --secure-port int                        The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 443)
      2025-08-25T18:08:50.907435574Z       --tls-cert-file string                   File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
      2025-08-25T18:08:50.907435574Z       --tls-cipher-suites strings              Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used. 
      2025-08-25T18:08:50.907435574Z                                                Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384. 
      2025-08-25T18:08:50.907435574Z                                                Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.
      2025-08-25T18:08:50.907435574Z       --tls-min-version string                 Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13
      2025-08-25T18:08:50.907435574Z       --tls-private-key-file string            File containing the default x509 private key matching --tls-cert-file.
      2025-08-25T18:08:50.907435574Z       --tls-sni-cert-key namedCertKey          A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
      2025-08-25T18:08:50.907435574Z 
      2025-08-25T18:08:50.907435574Z Apiserver authentication flags:
      2025-08-25T18:08:50.907435574Z 
      2025-08-25T18:08:50.907435574Z       --authentication-kubeconfig string                  kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenreviews.authentication.k8s.io.
      2025-08-25T18:08:50.907435574Z       --authentication-skip-lookup                        If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
      2025-08-25T18:08:50.907435574Z       --authentication-token-webhook-cache-ttl duration   The duration to cache responses from the webhook token authenticator. (default 10s)
      2025-08-25T18:08:50.907435574Z       --authentication-tolerate-lookup-failure            If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.
      2025-08-25T18:08:50.907435574Z       --client-ca-file string                             If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
      2025-08-25T18:08:50.907435574Z       --requestheader-allowed-names strings               List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
      2025-08-25T18:08:50.907435574Z       --requestheader-client-ca-file string               Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.
      2025-08-25T18:08:50.907435574Z       --requestheader-extra-headers-prefix strings        List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
      2025-08-25T18:08:50.907435574Z       --requestheader-group-headers strings               List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
      2025-08-25T18:08:50.907435574Z       --requestheader-username-headers strings            List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
      2025-08-25T18:08:50.907435574Z 
      2025-08-25T18:08:50.907435574Z Apiserver authorization flags:
      2025-08-25T18:08:50.907435574Z 
      2025-08-25T18:08:50.907435574Z       --authorization-always-allow-paths strings                A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server. (default [/healthz,/readyz,/livez])
      2025-08-25T18:08:50.907435574Z       --authorization-kubeconfig string                         kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io.
      2025-08-25T18:08:50.907490519Z       --authorization-webhook-cache-authorized-ttl duration     The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
      2025-08-25T18:08:50.907490519Z       --authorization-webhook-cache-unauthorized-ttl duration   The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
      2025-08-25T18:08:50.907490519Z 
      2025-08-25T18:08:50.907490519Z Apiserver audit log flags:
      2025-08-25T18:08:50.907490519Z 
      2025-08-25T18:08:50.907490519Z       --audit-log-batch-buffer-size int             The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
      2025-08-25T18:08:50.907490519Z       --audit-log-batch-max-size int                The maximum size of a batch. Only used in batch mode. (default 1)
      2025-08-25T18:08:50.907490519Z       --audit-log-batch-max-wait duration           The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode.
      2025-08-25T18:08:50.907490519Z       --audit-log-batch-throttle-burst int          Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.
      2025-08-25T18:08:50.907490519Z       --audit-log-batch-throttle-enable             Whether batching throttling is enabled. Only used in batch mode.
      2025-08-25T18:08:50.907490519Z       --audit-log-batch-throttle-qps float32        Maximum average number of batches per second. Only used in batch mode.
      2025-08-25T18:08:50.907490519Z       --audit-log-compress                          If set, the rotated log files will be compressed using gzip.
      2025-08-25T18:08:50.907490519Z       --audit-log-format string                     Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json format. Known formats are legacy,json. (default "json")
      2025-08-25T18:08:50.907490519Z       --audit-log-maxage int                        The maximum number of days to retain old audit log files based on the timestamp encoded in their filename.
      2025-08-25T18:08:50.907490519Z       --audit-log-maxbackup int                     The maximum number of old audit log files to retain. Setting a value of 0 will mean there's no restriction on the number of files.
      2025-08-25T18:08:50.907490519Z       --audit-log-maxsize int                       The maximum size in megabytes of the audit log file before it gets rotated.
      2025-08-25T18:08:50.907490519Z       --audit-log-mode string                       Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict. (default "blocking")
      2025-08-25T18:08:50.907490519Z       --audit-log-path string                       If set, all requests coming to the apiserver will be logged to this file.  '-' means standard out.
      2025-08-25T18:08:50.907490519Z       --audit-log-truncate-enabled                  Whether event and batch truncating is enabled.
      2025-08-25T18:08:50.907490519Z       --audit-log-truncate-max-batch-size int       Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size. (default 10485760)
      2025-08-25T18:08:50.907490519Z       --audit-log-truncate-max-event-size int       Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded. (default 102400)
      2025-08-25T18:08:50.907490519Z       --audit-log-version string                    API group and version used for serializing audit events written to log. (default "audit.k8s.io/v1")
      2025-08-25T18:08:50.907490519Z       --audit-policy-file string                    Path to the file that defines the audit policy configuration.
      2025-08-25T18:08:50.907490519Z       --audit-webhook-batch-buffer-size int         The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
      2025-08-25T18:08:50.907490519Z       --audit-webhook-batch-max-size int            The maximum size of a batch. Only used in batch mode. (default 400)
      2025-08-25T18:08:50.907490519Z       --audit-webhook-batch-max-wait duration       The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode. (default 30s)
      2025-08-25T18:08:50.907490519Z       --audit-webhook-batch-throttle-burst int      Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode. (default 15)
      2025-08-25T18:08:50.907490519Z       --audit-webhook-batch-throttle-enable         Whether batching throttling is enabled. Only used in batch mode. (default true)
      2025-08-25T18:08:50.907490519Z       --audit-webhook-batch-throttle-qps float32    Maximum average number of batches per second. Only used in batch mode. (default 10)
      2025-08-25T18:08:50.907490519Z       --audit-webhook-config-file string            Path to a kubeconfig formatted file that defines the audit webhook configuration.
      2025-08-25T18:08:50.907490519Z       --audit-webhook-initial-backoff duration      The amount of time to wait before retrying the first failed request. (default 10s)
      2025-08-25T18:08:50.907490519Z       --audit-webhook-mode string                   Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking,blocking-strict. (default "batch")
      2025-08-25T18:08:50.907490519Z       --audit-webhook-truncate-enabled              Whether event and batch truncating is enabled.
      2025-08-25T18:08:50.907490519Z       --audit-webhook-truncate-max-batch-size int   Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes greater. If a batch exceeds this limit, it is split into several batches of smaller size. (default 10485760)
      2025-08-25T18:08:50.907490519Z       --audit-webhook-truncate-max-event-size int   Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number, first request and response are removed, and if this doesn't reduce the size enough, event is discarded. (default 102400)
      2025-08-25T18:08:50.907490519Z       --audit-webhook-version string                API group and version used for serializing audit events written to webhook. (default "audit.k8s.io/v1")
      2025-08-25T18:08:50.907490519Z 
      2025-08-25T18:08:50.907490519Z Features flags:
      2025-08-25T18:08:50.907490519Z 
      2025-08-25T18:08:50.907490519Z       --contention-profiling   Enable lock contention profiling, if profiling is enabled
      2025-08-25T18:08:50.907490519Z       --profiling              Enable profiling via web interface host:port/debug/pprof/ (default true)
      2025-08-25T18:08:50.907490519Z 
      2025-08-25T18:08:50.907490519Z Logging flags:
      2025-08-25T18:08:50.907490519Z 
      2025-08-25T18:08:50.907490519Z       --add_dir_header                   If true, adds the file directory to the header of the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
      2025-08-25T18:08:50.907490519Z       --alsologtostderr                  log to standard error as well as files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
      2025-08-25T18:08:50.907490519Z       --log-flush-frequency duration     Maximum number of seconds between log flushes (default 5s)
      2025-08-25T18:08:50.907490519Z       --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default :0)
      2025-08-25T18:08:50.907490519Z       --log_dir string                   If non-empty, write log files in this directory (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
      2025-08-25T18:08:50.907490519Z       --log_file string                  If non-empty, use this log file (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
      2025-08-25T18:08:50.907490519Z       --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default 1800)
      2025-08-25T18:08:50.907490519Z       --logtostderr                      log to standard error instead of files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default true)
      2025-08-25T18:08:50.907490519Z       --one_output                       If true, only write logs to their native severity level (vs also writing to each lower severity level) (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
      2025-08-25T18:08:50.907537436Z       --skip_headers                     If true, avoid header prefixes in the log messages (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
      2025-08-25T18:08:50.907537436Z       --skip_log_headers                 If true, avoid headers when opening log files (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components)
      2025-08-25T18:08:50.907537436Z       --stderrthreshold severity         logs at or above this threshold go to stderr (DEPRECATED: will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components) (default 2)
      2025-08-25T18:08:50.907537436Z   -v, --v Level                          number for the log level verbosity
      2025-08-25T18:08:50.907537436Z       --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging
      2025-08-25T18:08:50.907537436Z 
      2025-08-25T18:08:50.909557778Z panic: unable to get openapi models: [3,11] $root.info is missing required property: version
      2025-08-25T18:08:50.909557778Z 
      2025-08-25T18:08:50.909557778Z goroutine 1 [running]:
      2025-08-25T18:08:50.909557778Z main.main()
      2025-08-25T18:08:50.909557778Z     /go/src/sigs.k8s.io/metrics-server/cmd/metrics-server/metrics-server.go:37 +0x9a

      Version-Release number of selected component (if applicable):

      4.15.0-0.nightly-2025-08-20-015949

      How reproducible:

      not sure

      Steps to Reproduce:

      1. see description
          

      Actual results:

      metrics-server pods are CrashLoopBackOff

      Expected results:

      no error

      Additional info:

          

              Unassigned Unassigned
              juzhao@redhat.com Junqi Zhao
              None
              None
              Junqi Zhao Junqi Zhao
              None
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated: