Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-20354

On an SNO with Telco DU profile must-gather perf-node-gather-daemonset fails: Error creating: pods "perf-node-gather-daemonset-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the pod namespace does not allow the workload type management

XMLWordPrintable

    • Moderate
    • No
    • False
    • Hide

      None

      Show
      None

      This is a clone of issue OCPBUGS-19761. The following is the description of the original issue:

      Description of problem:

      When running must-gather against an SNO with Telco DU profile the perf-node-gather-daemonset seems to not be able to start with the error below:
      
       Warning  FailedCreate  2m37s (x16 over 5m21s)  daemonset-controller  Error creating: pods "perf-node-gather-daemonset-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the pod namespace "openshift-must-gather-sbhml" does not allow the workload type management
      
      must-gather shows it's retrying for 300s and reports that performance data collection was complete even though the daemonset pod didn't come up.
      
      [must-gather-nhbgr] POD 2023-09-26T10:15:39.591582116Z Waiting for performance profile collector pods to become ready: 1
      [..]
      [must-gather-nhbgr] POD 2023-09-26T10:21:07.108893075Z Waiting for performance profile collector pods to become ready: 300
      [must-gather-nhbgr] POD 2023-09-26T10:21:08.473217146Z daemonset.apps "perf-node-gather-daemonset" deleted
      [must-gather-nhbgr] POD 2023-09-26T10:21:08.480906220Z INFO: Node performance data collection complete.
      

      Version-Release number of selected component (if applicable):

      4.14.0-rc.2

      How reproducible:

      100%

      Steps to Reproduce:

      1. Deploy SNO with Telco DU profile
      2. Run oc adm must-gather
      

      Actual results:

      performance data collection doesn't run because daemonset cannot be scheduled.

      Expected results:

      performance data collection runs.

      Additional info:

      DaemonSet describe:
      
      oc -n openshift-must-gather-sbhml describe ds
      Name:           perf-node-gather-daemonset
      Selector:       name=perf-node-gather-daemonset
      Node-Selector:  <none>
      Labels:         <none>
      Annotations:    deprecated.daemonset.template.generation: 1
      Desired Number of Nodes Scheduled: 1
      Current Number of Nodes Scheduled: 0
      Number of Nodes Scheduled with Up-to-date Pods: 0
      Number of Nodes Scheduled with Available Pods: 0
      Number of Nodes Misscheduled: 0
      Pods Status:  0 Running / 0 Waiting / 0 Succeeded / 0 Failed
      Pod Template:
        Labels:       name=perf-node-gather-daemonset
        Annotations:  target.workload.openshift.io/management: {"effect": "PreferredDuringScheduling"}
        Containers:
         node-probe:
          Image:      registry.kni-qe-0.lab.eng.rdu2.redhat.com:5000/openshift-release-dev@sha256:2af2c135f69f162ed8e0cede609ddbd207d71a3c7bd49e9af3fcbb16737aa25a
          Port:       <none>
          Host Port:  <none>
          Command:
            /bin/bash
            -c
            echo ok > /tmp/healthy && sleep INF
          Limits:
            cpu:     100m
            memory:  256Mi
          Requests:
            cpu:        100m
            memory:     256Mi
          Readiness:    exec [cat /tmp/healthy] delay=5s timeout=1s period=5s #success=1 #failure=3
          Environment:  <none>
          Mounts:
            /host/podresources from podres (rw)
            /host/proc from proc (ro)
            /host/sys from sys (ro)
            /lib/modules from lib-modules (ro)
        Volumes:
         sys:
          Type:          HostPath (bare host directory volume)
          Path:          /sys
          HostPathType:  Directory
         proc:
          Type:          HostPath (bare host directory volume)
          Path:          /proc
          HostPathType:  Directory
         lib-modules:
          Type:          HostPath (bare host directory volume)
          Path:          /lib/modules
          HostPathType:  Directory
         podres:
          Type:          HostPath (bare host directory volume)
          Path:          /var/lib/kubelet/pod-resources
          HostPathType:  Directory
      Events:
        Type     Reason        Age                     From                  Message
        ----     ------        ----                    ----                  -------
        Warning  FailedCreate  2m37s (x16 over 5m21s)  daemonset-controller  Error creating: pods "perf-node-gather-daemonset-" is forbidden: autoscaling.openshift.io/ManagementCPUsOverride the pod namespace "openshift-must-gather-sbhml" does not allow the workload type management
      

            rh-ee-rbaturov Ronny Baturov
            openshift-crt-jira-prow OpenShift Prow Bot
            Mallapadi Niranjan Mallapadi Niranjan
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: