Uploaded image for project: 'Managed Service - Streams'
  1. Managed Service - Streams
  2. MGDSTRM-8076

Implement a KubePersistentVolumeFillingUp alert and a SOP to resolve it

XMLWordPrintable

    • Icon: Story Story
    • Resolution: Done
    • Icon: Critical Critical
    • None
    • None
    • None
    • 5
    • False
    • None
    • False
    • No
    • ---
    • ---
    • MK - Sprint 219

      During the issue https://issues.redhat.com/browse/OHSS-11275, the broker volumes were completely filled up and there were no prior alerts that warned SRE indicating that a volume is about to run out of space.

      Story
      As an SRE I want to be alerted with a volume that will be inevitably filled completely based on the disk available (critical) so that I can proactively run a SOP and minimize the chance of it from happening.

      Notes:
      1. This disk available must be the amount of disk space the brokers can tolerate before it will become unstable.
      2. There is a storage quota plugin however this alert is a failsafe mechanism for when this plugin will fail. The cost of recovering the brokers to a stable state is very high and can be catastrophic, therefore we need to ensure the brokers have enough space all the time even in the event of errors, bugs, etc. If this plugin can be assured to never fail then please reject this ticket.

              stobin1@redhat.com Steven Tobin
              jcueto@redhat.com Jose Gavine Cueto
              MK - Running the Service
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: