Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-1341

Node churn leaks PodNetworkConnectivityChecks

    XMLWordPrintable

Details

    • Moderate
    • 3
    • OCP VE Sprint 225, OCP VE Sprint 226, OCP VE Sprint 227, OCP VE Sprint 228, OCP VE Sprint 229, OCP VE Sprint 230, OCP VE Sprint 231, OCP VE Sprint 232, OCP VE Sprint 233, OCP VE Sprint 234, OCP VE Sprint 235
    • 11
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • N/A
    • Release Note Not Required

    Description

      I haven't gone back to pin down all affected versions, but I wouldn't be surprised if we've had this exposure for a while. On a 4.12.0-ec.2 cluster, we have:

      cluster:usage:resources:sum{resource="podnetworkconnectivitychecks.controlplane.operator.openshift.io"}

      currently clocking in around 67983. I've gathered a dump with:

      $ oc --as system:admin -n openshift-network-diagnostics get podnetworkconnectivitychecks.controlplane.operator.openshift.io | gzip >checks.gz

      And many, many of these reference nodes which no longer exist (the cluster is aggressively autoscaled, with nodes coming and going all the time). We should fix garbage collection on this resource, to avoid consuming excessive amounts of memory in the Kube API server and etcd as they attempt to list the large resource set.

      Attachments

        Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. SDN-3636 Kube 1.26 rebase for CNO

          • Normal - To be worked after urgent and high priorities are resolved. This term is chosen when the severity of an issue is relatively close to the level of effort to fix it. The existence of an easily implemented workaround can also lead to this priority level instead of a higher priority.
          • Closed
          is cloned by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-17721 [release-4.13] Node churn leaks PodNetworkConnectivityChecks

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          is depended on by

          Bug - A problem that impairs or prevents the functions of the product. OCPBUGS-17721 [release-4.13] Node churn leaks PodNetworkConnectivityChecks

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Closed
          links to

          GitHub openshift/cluster-network-operator#1566: OCPBUGS-1341: Set owner reference for pod network connectivity check

          GitHub openshift/cluster-network-operator#1649: OCPBUGS-1341: Enhance check controller to remove old check objects

          GitHub openshift/library-go#1430: OCPBUGS-1341: Remove stale pod network connectivity checks

          Web Link RHEA-2023:5006 rpm

          Activity

            People

              pepalani@redhat.com Periyasamy Palanichamy
              trking W. Trevor King
              Michael Fiedler Michael Fiedler
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: