Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-36295

Denoise: fsnotify: can't remove non-existent watch: /etc/kubernetes/kubelet-ca.crt

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.13, 4.14, 4.17
    • Node / Kubelet
    • Moderate
    • None
    • OCPNODE Sprint 257 (Green), OCP Node Sprint 258 (Green), OCP Node Sprint 259 (Green), OCP Node Sprint 260 (Green)
    • 4
    • False
    • Hide

      None

      Show
      None

      Description of problem

      Kubelet logs contain entries like:

      Jun 13 10:05:14.141073 ip-10-0-133-167 kubenswrapper[1385]: I0613 10:05:14.141043    1385 dynamic_cafile_content.go:211] "Failed to remove file watch, it may have been deleted" file="/etc/kubernetes/kubelet-ca.crt" err="can't remove non-existent watcher: /etc/kubernetes/kubelet-ca.crt"
      

      I'm not sure if that's a problem or not, but it is distracting noise for folks trying to understand Kubelet behavior, and we should either fix the problem, or denoise the red-herring.

      Version-Release number of selected component

      Seen in 4.13.44, 4.14.31, and 4.17.0-0.nightly-2024-06-25-162526 (details in Additional info).
      Not seen in 4.12.60, so presumably a 4.12 to 4.13 change.

      How reproducible

      Every time.

      Steps to Reproduce

      1. Run a cluster.
      2. Check node/kubelet logs for one control-plane node.

      Actual results

      Lots of can't remove non-existent watcher: /etc/kubernetes/kubelet-ca.crt messages.

      Expected results

      No can't remove non-existent watcher: /etc/kubernetes/kubelet-ca.crt messages.

      Additional info

      Checking recent builds in assorted 4.y streams.

      4.12.60

      4.12.60 > aws-sdn-serial > Artifacts > ... > gather-extra artifacts:

      $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.12-e2e-aws-sdn-serial/1803708035177123840/artifacts/e2e-aws-sdn-serial/gather-extra/artifacts/nodes.json | jq -r '.items[].metadata | select(.labels["node-role.kubernetes.io/control-plane"] == "").name'
      ip-10-0-156-214.us-west-1.compute.internal
      ip-10-0-158-171.us-west-1.compute.internal
      ip-10-0-203-59.us-west-1.compute.internal
      $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.12-e2e-aws-sdn-serial/1803708035177123840/artifacts/e2e-aws-sdn-serial/gather-extra/artifacts/nodes/ip-10-0-156-214.us-west-1.compute.internal/journal | zgrep kubelet-ca.crt | tail -n3
      Jun 20 08:47:07.734060 ip-10-0-156-214 ignition[1087]: INFO     : files: createFilesystemsFiles: createFiles: op(11): [finished] writing file "/sysroot/etc/kubernetes/kubelet-ca.crt"
      Jun 20 08:49:29.274949 ip-10-0-156-214 kubenswrapper[1384]: I0620 08:49:29.274923    1384 dynamic_cafile_content.go:119] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
      Jun 20 08:49:29.275084 ip-10-0-156-214 kubenswrapper[1384]: I0620 08:49:29.275067    1384 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
      

      is clean.

      4.13.44

      4.13.44 > aws-sdn-serial > Artifacts > ... > gather-extra artifacts:

      $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-sdn-serial/1801188570212339712/artifacts/e2e-aws-sdn-serial/gather-extra/artifacts/nodes.json |  jq -r '.items[].metadata | select(.labels["node-role.kubernetes.io/control-plane"] == "").name'
      ip-10-0-133-167.us-west-1.compute.internal
      ip-10-0-170-3.us-west-1.compute.internal
      ip-10-0-203-13.us-west-1.compute.internal
      $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.13-e2e-aws-sdn-serial/1801188570212339712/artifacts/e2e-aws-sdn-serial/gather-extra/artifacts/nodes/ip-10-0-133-167.us-west-1.compute.internal/journal | zgrep kubelet-ca.crt | tail -n3
      Jun 13 10:05:00.464260 ip-10-0-133-167 kubenswrapper[1385]: I0613 10:05:00.464190    1385 dynamic_cafile_content.go:211] "Failed to remove file watch, it may have been deleted" file="/etc/kubernetes/kubelet-ca.crt" err="can't remove non-existent watcher: /etc/kubernetes/kubelet-ca.crt"
      Jun 13 10:05:13.320867 ip-10-0-133-167 kubenswrapper[1385]: I0613 10:05:13.320824    1385 dynamic_cafile_content.go:211] "Failed to remove file watch, it may have been deleted" file="/etc/kubernetes/kubelet-ca.crt" err="can't remove non-existent watcher: /etc/kubernetes/kubelet-ca.crt"
      Jun 13 10:05:14.141073 ip-10-0-133-167 kubenswrapper[1385]: I0613 10:05:14.141043    1385 dynamic_cafile_content.go:211] "Failed to remove file watch, it may have been deleted" file="/etc/kubernetes/kubelet-ca.crt" err="can't remove non-existent watcher: /etc/kubernetes/kubelet-ca.crt"
      

      is exposed.

      4.14.31

      4.14.31 > aws-ovn-serial > Artifacts > ... > gather-extra artifacts:

      $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-serial/1803746771264868352/artifacts/e2e-aws-ovn-serial/gather-extra/artifacts/nodes.json | jq -r '.items[].metadata | select(.labels["node-role.kubernetes.io/control-plane"] == "").name'
      ip-10-0-17-181.us-west-2.compute.internal
      ip-10-0-66-68.us-west-2.compute.internal
      ip-10-0-97-83.us-west-2.compute.internal
      $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.14-e2e-aws-ovn-serial/1803746771264868352/artifacts/e2e-aws-ovn-serial/gather-extra/artifacts/nodes/ip-10-0-17-181.us-west-2.compute.internal/journal | zgrep kubelet-ca.crt | tail -n3
      Jun 20 11:42:31.931470 ip-10-0-17-181 kubenswrapper[2226]: I0620 11:42:31.931404    2226 dynamic_cafile_content.go:211] "Failed to remove file watch, it may have been deleted" file="/etc/kubernetes/kubelet-ca.crt" err="can't remove non-existent watcher: /etc/kubernetes/kubelet-ca.crt"
      Jun 20 11:42:31.980499 ip-10-0-17-181 kubenswrapper[2226]: I0620 11:42:31.980448    2226 dynamic_cafile_content.go:211] "Failed to remove file watch, it may have been deleted" file="/etc/kubernetes/kubelet-ca.crt" err="can't remove non-existent watcher: /etc/kubernetes/kubelet-ca.crt"
      Jun 20 11:42:32.757888 ip-10-0-17-181 kubenswrapper[2226]: I0620 11:42:32.757846    2226 dynamic_cafile_content.go:211] "Failed to remove file watch, it may have been deleted" file="/etc/kubernetes/kubelet-ca.crt" err="can't remove non-existent watcher: /etc/kubernetes/kubelet-ca.crt"
      

      4.17.0-0.nightly-2024-06-25-162526

      4.17.0-0.nightly-2024-06-25-162526 > aws-ovn-serial > Artifacts > ... > gather-extra artifacts:

      $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-aws-ovn-serial/1805639599624556544/artifacts/e2e-aws-ovn-serial/gather-extra/artifacts/nodes.json | jq -r '.items[].metadata | select(.labels["node-role.kubernetes.io/control-plane"] == "").name'
      ip-10-0-125-200.ec2.internal
      ip-10-0-47-81.ec2.internal
      ip-10-0-8-158.ec2.internal
      $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.17-e2e-aws-ovn-serial/1805639599624556544/artifacts/e2e-aws-ovn-serial/gather-extra/artifacts/nodes/ip-10-0-8-158.ec2.internal/journal | zgrep kubelet-ca.crt | tail -n3
      Jun 25 19:56:13.452559 ip-10-0-8-158 kubenswrapper[2243]: I0625 19:56:13.452512    2243 dynamic_cafile_content.go:211] "Failed to remove file watch, it may have been deleted" file="/etc/kubernetes/kubelet-ca.crt" err="fsnotify: can't remove non-existent watch: /etc/kubernetes/kubelet-ca.crt"
      Jun 25 19:56:13.512277 ip-10-0-8-158 kubenswrapper[2243]: I0625 19:56:13.512213    2243 dynamic_cafile_content.go:211] "Failed to remove file watch, it may have been deleted" file="/etc/kubernetes/kubelet-ca.crt" err="fsnotify: can't remove non-existent watch: /etc/kubernetes/kubelet-ca.crt"
      Jun 25 19:56:14.403001 ip-10-0-8-158 kubenswrapper[2243]: I0625 19:56:14.402953    2243 dynamic_cafile_content.go:211] "Failed to remove file watch, it may have been deleted" file="/etc/kubernetes/kubelet-ca.crt" err="fsnotify: can't remove non-existent watch: /etc/kubernetes/kubelet-ca.crt"
      

            skunkerk Sohan Kunkerkar
            trking W. Trevor King
            Min Li Min Li
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated: