Uploaded image for project: 'OpenShift API Server'
  1. OpenShift API Server
  2. API-1273

Alert on poorly configured admission webhooks

XMLWordPrintable

    • Icon: Epic Epic
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • None
    • Warn against badly configured admission webhooks risking stability
    • None
    • 0% To Do, 0% In Progress, 100% Done
    • False
    • None
    • False
    • None
    • None
    • None
    • 3

      Research into ways to protect ourselves against possibly harmful admission webhooks. 3rd party webhooks that affect k8s.io or openshift.io resources without label selector or namespace selector can block resource creations and updates, putting the clusters at risk. E.g., https://github.com/hashicorp/vault-k8s/blob/master/deploy/injector-mutating-webhook.yaml doesn't have any namespace selector.

      Slack thread: https://coreos.slack.com/archives/CC3CZCQHM/p1621585026315100?thread_ts=1621268975.202600&cid=CC3CZCQHM

      Ideas:

      • alert about latency per webhook
      • alert about total latency (we hava a metric afaik)
      • some info-level alert (or other way, warnings, conditions) about webhooks of core resources, i.e. not about CRDs provided by an operator

      Example customer cases:

          1.
          		 add alert specific to webhooks Sub-task Closed Undefined Stefan Schimanski (Inactive)
          2.
          		 finish enhancement "Improving Supportability of API webhooks" Sub-task Closed Undefined Stefan Schimanski (Inactive)
          3.
          		 set kube-apiserver degraded=true if an admission webhook is installed for a virtual resource Sub-task Closed Undefined Luis Sanchez
          4.
          		 set kube-apiserver degraded=true if an admission webhook service or url is invalid Sub-task Closed Undefined Luis Sanchez
          5.
          		 set kube-apiserver degraded=true if a crd conversion webhook service or url is invalid Sub-task Closed Undefined Luis Sanchez
          6.
          		 informational alert on admission webhook installed with rule matching a core resource Sub-task Closed Undefined Stefan Schimanski (Inactive)
          7.
          		 informational alert on admission webhook installed with rule matching security a sensitive resource Sub-task Closed Undefined Stefan Schimanski (Inactive)
          8.
          		 informational alert on crd conversion webhook installed with rule maching a core resource Sub-task Closed Undefined Stefan Schimanski (Inactive)
          9.
          		 critical alert if a crd conversion webhook fails > 1% of the time Sub-task Closed Undefined Stefan Schimanski (Inactive)
          10.
          		 critical alert if admission webhook fails > 1% of the time Sub-task Closed Undefined Stefan Schimanski (Inactive)
          11.
          		 critical alert if the latency of an admission webhook is > 1s Sub-task Closed Undefined Luis Sanchez
          12.
          		 critical alert if the latency of an admission webhook that affects critical resources is > 0.5s Sub-task Closed Undefined Stefan Schimanski (Inactive)
          13.
          		 critical alert if the total webhook latency for a resource is > 1s Sub-task Closed Undefined Stefan Schimanski (Inactive)
          14.
          		 critical alert if the total webhook latency for a critical resource is > 0.5s Sub-task Closed Undefined Stefan Schimanski (Inactive)
          15.
          		 QE Tracker Sub-task Closed Undefined Ke Wang
          16.
          		 Docs Tracker Sub-task Closed Undefined Unassigned
          17.
          		 TE Tracker Sub-task Closed Undefined Unassigned

              Unassigned Unassigned
              sttts@redhat.com Stefan Schimanski (Inactive)
              None
              None
              Ke Wang Ke Wang
              None
              Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated: