Market Problem

Recently in a scale test for saturating SC with 300 and 240 HCPs, and while running a workload across all available HCPs, it was observed that a policy could flap on the SC between compliant and non-compliant which produces a large amount of excess etcd db usage as well as many API requests against the kube-apiserver and driving high cpu util for KAS.  We had ev en observed a few KAS container restarts in the KAS pods. We need to monitor for any rapidly flapping policy updates in order to see if and when the problem occurs.

Why it Matters

If etcd usage grows beyond quota, the entire SC cluster will be frozen until manual defrag and compaction is ran. If KAS usage is higher than cpu capacity of the instance KAS is running on it will result in poor performance or service outages when the KAS pod restarts.

Illustrative User Stories or Scenarios

...

Expected Outcomes

This Section: Articulates and defines the value proposition from a users
point of view

...

Effect

...

Challenges and Open Questions:

...

Partner

This Section: Partner is any specific delivery vehicle requirement.
These are feature components that MUST be delivered in concert with or
through a particular BU external product.

This represents part of the “how” statement for a feature. This would be
the optional only category.

• ...

Additional/tangential areas for future development (TBK)

…