Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-64826

iptables_alerter pod is being throttled by cfs

XMLWordPrintable

    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Moderate
    • None
    • x86_64
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      Reviewing CFS metrics from multiple clusters shows iptables-alerter DaemonSet pods being throttled, eg:
      
      $ oc -n openshift-monitoring port-forward prometheus-k8s-0 9090:9090 --address='0.0.0.0' &
      
      $ curl -ks 'http://localhost:9090/api/v1/query' --data-urlencode 'query=sort_desc(rate(container_cpu_cfs_throttled_periods_total[30m]))' |jq -r '.data.result[] | [.value[1], .metric.namespace, .metric.pod] | @tsv' | sed 's/\t/ /g'  | grep ipta
      1.8054096033892513 openshift-network-operator iptables-alerter-8l749
      1.5280656859472481 openshift-network-operator iptables-alerter-8lk9j
      1.5202738858741267 openshift-network-operator iptables-alerter-pfjsz
      0.9603678925351364 openshift-network-operator iptables-alerter-vmh25
      0.8210337602333986 openshift-network-operator iptables-alerter-lfbm5
      0.8030910999316458 openshift-network-operator iptables-alerter-69rlp
      0.7933652001688735 openshift-network-operator iptables-alerter-km7wr
      0.7764942528735631 openshift-network-operator iptables-alerter-mcccz
      0.7553201867306674 openshift-network-operator iptables-alerter-dtsqg
      0.7482219855274379 openshift-network-operator iptables-alerter-svvb7
      0.7015708424719596 openshift-network-operator iptables-alerter-d9bkg
      0.37218384643277824 openshift-network-operator iptables-alerter-sxj6j
      0 openshift-network-operator iptables-alerter-24f9v
      0 openshift-network-operator iptables-alerter-2xqsq
      0 openshift-network-operator iptables-alerter-w2259
      0 openshift-network-operator iptables-alerter-mncbc
      0 openshift-network-operator iptables-alerter-m9v6v
      0 openshift-network-operator iptables-alerter-4rkwl
      

      Version-Release number of selected component (if applicable):

      Seen on 4.18 cluster and 4.20.0-ec.5

      How reproducible:

          100%

      Steps to Reproduce:

      On the 4.20.0-ec.5 cluster
          1. Cluster is configured with workload partitioning enabled
          2. Review prometheus metrics as shown in description
          3. iptables-alerter pods consistently show cfs throttling
      
      
          

      Actual results:

          

      Expected results:

      No throttling of pods    

      Additional info:

          

       

              dwinship@redhat.com Dan Winship
              rhn-support-imiller Ian Miller
              None
              None
              Anurag Saxena Anurag Saxena
              None
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: