Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-34273

[4.12z] slow ovnkube-node initialization on large number of services with externalIps

XMLWordPrintable

    • +
    • No
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, if you configured an {product-title} cluster with a high number of internal services or user-managed load balancer IP addresses, you experienced a delayed startup time for the OVN-Kubernetes service. This delay occurred when the OVN-Kubernetes service attempted to install `iptables` rules on a node. With this release, the OVN-Kubernetes service can process a large number of services in a few seconds. Additionally, you can access a new log to view the status of installing `iptables` rules on a node. (link:https://issues.redhat.com/browse/OCPBUGS-34273[*OCPBUGS-34273*])
      _________
      *Cause*: What actions or circumstances cause this bug to present.
      When ovnkube-node pod is started and there are many external IPs or load balancer IP services exist.
      *Consequence*: What happens when the bug presents.
      ovnkube-node will take a very long time to start up due to it programming iptables rules one at a time for all services. Iterating over external IPs or load balancer iPs is especially costly. The length of start up time depends on the number of services and external IPs/load balancer IPs. We have seen this time take from several minutes to almost an hour.
      *Fix*: What was done to fix the bug.
      Optimizations in the service parsing logic reduces duplicate calls to create iptables rules. Now iptables rules are atomically created using iptable-restore rather than a separate call to iptables for each external IP.
      *Result*: Bug doesn’t present anymore.
      The start up time is much faster, even at high scale it should be on the order of seconds rather than minutes.
      Show
      * Previously, if you configured an {product-title} cluster with a high number of internal services or user-managed load balancer IP addresses, you experienced a delayed startup time for the OVN-Kubernetes service. This delay occurred when the OVN-Kubernetes service attempted to install `iptables` rules on a node. With this release, the OVN-Kubernetes service can process a large number of services in a few seconds. Additionally, you can access a new log to view the status of installing `iptables` rules on a node. (link: https://issues.redhat.com/browse/OCPBUGS-34273 [* OCPBUGS-34273 *]) _________ *Cause*: What actions or circumstances cause this bug to present. When ovnkube-node pod is started and there are many external IPs or load balancer IP services exist. *Consequence*: What happens when the bug presents. ovnkube-node will take a very long time to start up due to it programming iptables rules one at a time for all services. Iterating over external IPs or load balancer iPs is especially costly. The length of start up time depends on the number of services and external IPs/load balancer IPs. We have seen this time take from several minutes to almost an hour. *Fix*: What was done to fix the bug. Optimizations in the service parsing logic reduces duplicate calls to create iptables rules. Now iptables rules are atomically created using iptable-restore rather than a separate call to iptables for each external IP. *Result*: Bug doesn’t present anymore. The start up time is much faster, even at high scale it should be on the order of seconds rather than minutes.
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-32426. The following is the description of the original issue:

      on clusters with a large number of services with externalIPs or services from type loadBalancer the ovnkube-node initialization can take up to 50 min

      The problem is after a node reboot done by MCO the unschedule taint is removed from the node so the api allocates pods to that node that get stuck on ContrainerCreating and other nodes continue to go down for reboot making the workloads unavailable. (if no PDB exists for the workload to protect it)

              trozet@redhat.com Tim Rozet
              openshift-crt-jira-prow OpenShift Prow Bot
              Jean Chen Jean Chen
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: