Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-49711

openshift-etcd container count increased from 4 -> 5

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • 4.18.0
    • Etcd
    • None
    • Important
    • None
    • Rejected
    • False
    • Hide

      None

      Show
      None

      Description of problem:

          Telco KPI testing of rc.4+ shows that oopenshift-etcd _ etcd-<host> container count increased from 4 -> 5

      Version-Release number of selected component (if applicable):

          4.18.0-rc.4+

      How reproducible:

          100%

      Steps to Reproduce:

          1. Measure baseline at 4.17.0
          2. Deploy 4.18.0-rc.4
          3. Check container count     

      Actual results:

          Container count is 5

      Expected results:

          Container count remains at 4

      Additional info:

          I'm trying to understand the reason for a container count increase and whether we can avoid it in a RDS deployment as we're trying to keep any changes to load or container counts to a minimum.

            [OCPBUGS-49711] openshift-etcd container count increased from 4 -> 5

            rhn-support-imiller I don't have a per-container breakdown, but the whole pod is within 2-3% of CPU increase in worst case scenario (2 out of 7 runs), other 5 runs are within 1% difference. Do we continue as "noted" and just move on in this case? This is stats from the rc.9 builds btw.

            Alexander Gurenko added a comment - rhn-support-imiller I don't have a per-container breakdown, but the whole pod is within 2-3% of CPU increase in worst case scenario (2 out of 7 runs), other 5 runs are within 1% difference. Do we continue as "noted" and just move on in this case? This is stats from the rc.9 builds btw.

            rhn-support-imiller there is no concerning (as in violating our 10mc/20%) cpu utilization increase related to this change, I can try to get raw numbers now. API utilization metric is something I'm still trying to make sense and there is an unaccounted increase.

            tjungblu@redhat.com it was suggested in a conversation that kubeapi requests to etcd are tracked in prometheus together, I'm personally not 100% confident in this, so I'm still exploring

            Alexander Gurenko added a comment - rhn-support-imiller there is no concerning (as in violating our 10mc/20%) cpu utilization increase related to this change, I can try to get raw numbers now. API utilization metric is something I'm still trying to make sense and there is an unaccounted increase. tjungblu@redhat.com it was suggested in a conversation that kubeapi requests to etcd are tracked in prometheus together, I'm personally not 100% confident in this, so I'm still exploring

            Ian Miller added a comment -

            Is there any data on how active (cpu use) this container is? Can the container's cpu use be pulled from a test cluster?

            Ian Miller added a comment - Is there any data on how active (cpu use) this container is? Can the container's cpu use be pulled from a test cluster?

            I don't think so, this container goes straight to etcd. Check your audit logs.

            Thomas Jungblut added a comment - I don't think so, this container goes straight to etcd. Check your audit logs.

            tjungblu@redhat.com quick question, I've also noticed increased api/verb utilization for kubeapi server, can this be caused by said extra container in etcd?

            Alexander Gurenko added a comment - tjungblu@redhat.com quick question, I've also noticed increased api/verb utilization for kubeapi server, can this be caused by said extra container in etcd?

              dwest@redhat.com Dean West
              agurenko@redhat.com Alexander Gurenko
              Ge Liu Ge Liu
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated: