The metrics scraped from kubelet are inconsistent.

Kubelet logs report below messages indefinitely:
[1]
Apr 05 09:30:26 uk3posw3404 kubenswrapper[4149]: W0405 09:30:26.676270    4149 manager.go:694] Error getting data for container /kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462b0978_2821_4450_bf16_6d347bbfd065.slice/crio-conmon-24d90b01977d3ade52289e79629d0fa31e291de02f9c3030e895ca9a6f7b7558.scope because of race condition

[2] 
Apr 05 10:23:30 uk3posw3404 kubenswrapper[4149]: E0405 10:23:30.478772    4149 cadvisor_stats_provider.go:442] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d383d5_f4bf_4313_9b95_e40f6aa89805.slice/crio-conmon-476a344d46f31187487013e48952be26e881f78cd94477f560ccd66385abe4a1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d383d5_f4bf_4313_9b95_e40f6aa89805.slice/crio-conmon-30005fa64629cc38d96068ff197ab19d1df3b41bbacc1f8285562290e41a3281.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d383d5_f4bf_4313_9b95_e40f6aa89805.slice/crio-30005fa64629cc38d96068ff197ab19d1df3b41bbacc1f8285562290e41a3281.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod2cbb80c8_2e3a_4447_8526_57cf5088b222.slice/crio-9422cf74197c63fd0044b0cd5276e80c1fd975ac1d2d68933e45d6d266873c86.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58e4e81_088d_4393_a959_b36a182b10e2.slice/crio-conmon-91e019cb13394f314541d1ed2e6be3a6b73fc24ad9872d82ee4fdb770f620c10.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod72dfdd97_081b_42e8_ae92_80d546f7f310.slice/crio-conmon-5083f11d081db79c66c3563ca29f76e60e2f2f2f2b246da26565d2aa38e80a08.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod2cbb80c8_2e3a_4447_8526_57cf5088b222.slice/crio-conmon-bf79a45d353f00c45bdc623af5a4d57908662f8104165991fe3625861fc27dbd.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d383d5_f4bf_4313_9b95_e40f6aa89805.slice/crio-476a344d46f31187487013e48952be26e881f78cd94477f560ccd66385abe4a1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d383d5_f4bf_4313_9b95_e40f6aa89805.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d383d5_f4bf_4313_9b95_e40f6aa89805.slice/crio-677b3740e7d792dff0cebb328392021e3cbba76fbeaa09e06028addd3faa2eb5\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod2cbb80c8_2e3a_4447_8526_57cf5088b222.slice/crio-7d81b364b1437e8fa977f1835c58fbb3cb0e6018ea45c36744325682335fd7c7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod462b0978_2821_4450_bf16_6d347bbfd065.slice/crio-conmon-1773d45f65fb1cc8faa277b3310fc7415c34d84fac63e447d7586cdfd0dac49b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-pod2cbb80c8_2e3a_4447_8526_57cf5088b222.slice/crio-726ff883036ce7e0014eea86c18e667377b4e61be5282c4cb030ec8ab56e5dd0.scope\": RecentStats: unable to find data in memory cache]"

RHOCP 4.14.41 and 4.16.33

This is customer specific issue and there are no definite steps to reproduce it.

    1.
    2.
    3.
    

When checking the metrics scraped from kubelet, inconsistency was found.

The metrics from kubelet should be consistent.

Resource utilization (CPU and Memory) are normal on the underlying node.