Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-53267

Samples Operator capable of spamming updates to etcd

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • 4.19.0
    • Samples Operator
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • No
    • None
    • None
    • Approved
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      While investigating OCPBUGS-52848 we uncovered a number of deficiencies in the deprecated / maintenance only samples-operator. Despite it's maint status, we are requesting some of these be fixed because they are related to etcd load and the health of the apiserver, and should not be left in the wild.

      The issues have been broken out into three separate bugs.

      OCPBUGS-54589
      OCPBUGS-54590
      OCPBUGS-54591

      To view the updates your operator is making to it's ClusterOperator CR Progressing condition, open the artifacts for a prow job run and drill down to observers-resource-watch/artifacts/resource-watch-store.tar, example: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-ci-4.19-e2e-gcp-ovn-techpreview/1898814955482779648/artifacts/e2e-gcp-ovn-techpreview/observers-resource-watch/artifacts/

      Untar, cd repo, and run a git history viewer such as 'tig' on cluster-scoped-resources/config.openshift.io/clusteroperators/openshift-samples.yaml. Each commit is an observed change to the object in etcd during the job run. Excessive commits is the concern with etcd load. Some of the runs we looked at in the original bug had over 2000, but without the underlying issue back in 4.18 I still see over 500 updates, which is dramatically more writes to etcd than should be happening and the key number we want to reduce in this bug.

              rhn-support-spoole Shannon Poole
              rhn-engineering-dgoodwin Devan Goodwin
              None
              None
              Jitendar Singh Jitendar Singh
              None
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: