Sometimes the kubelet does not return metrics in under 30 seconds, and in some cases using the proxy interface to the kubelet the metrics are not returned within 2 minutes. These are likely two different issues.

The upstream https://github.com/kubernetes/kubernetes/pull/133392 PR fixes the upstream [test|https://github.com/kubernetes/kubernetes/blob/544dfee60a99eaec9962c3853674dc1e4d7f0c8d/test/e2e/instrumentation/metrics.go#L56] by deflaking the  HTTP request call to the proxy. It's unclear why the upstream test called out why the test is flaking in this regards, so further study will be needed.

Additionally, we see metrics taking over 30s to be returned from cadvisor. This is likely a separate issue that needs more investigation on why it happens.  [~abukashem] has some thoughts with improvements in this area.

    

    1.
    2.
    3.