Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-29922

[4.15] Infinite PODs loop creation with "NodeAffinity" status

XMLWordPrintable

    • Moderate
    • No
    • False
    • Hide

      None

      Show
      None
    • Hide
      Previously, projects that specified a non-intersecting openshift.io/node-selector project selector with pods .spec.nodeName could cause runaway Pod creation in Deployments. With this release, pods with non-intersecting .spec.nodeName are not admitted by the API server which resolves the issue.
      Show
      Previously, projects that specified a non-intersecting openshift.io/node-selector project selector with pods .spec.nodeName could cause runaway Pod creation in Deployments. With this release, pods with non-intersecting .spec.nodeName are not admitted by the API server which resolves the issue.
    • Bug Fix
    • Done

      This is a clone of issue OCPBUGS-17249. The following is the description of the original issue:

      Description of problem:

      When on projects is used the openshift.io/node-selector option in the annotation pointing to the default "node-role.kubernetes.io/worker=" key and a deployment with a running replicaset/PODS is set to use a different role then the default one the scheduler enter in an infinite loop of POD creation.

      Version-Release number of selected component (if applicable):

      Tested on 4.11

      Steps to Reproduce:

      1. Create a project via [oc create/apply] with annotation: openshift.io/node-selector: node-role.kubernetes.io/worker= 
      2. Create a deployment that creates a running POD 
      3. Edit the deployment and add the nodeName: option pointing to a different role than the worker one

      Actual results:

      Infinite POD creation: 
      ❯ oc get po
      NAME                                     READY   STATUS         RESTARTS   AGE
      infinite-pod-creation-7458cbbd88-98zpn   1/1     Running        0          8m40s
      infinite-pod-creation-7688f685c7-2grmh   0/1     NodeAffinity   0          1s
      infinite-pod-creation-7688f685c7-4g7dd   0/1     NodeAffinity   0          2s
      infinite-pod-creation-7688f685c7-59zr6   0/1     NodeAffinity   0          1s
      infinite-pod-creation-7688f685c7-5l5xl   0/1     NodeAffinity   0          2s
      infinite-pod-creation-7688f685c7-5nw22   0/1     NodeAffinity   0          2s
      infinite-pod-creation-7688f685c7-5qr7z   0/1     NodeAffinity   0          1s
      infinite-pod-creation-7688f685c7-5wp2q   0/1     NodeAffinity   0          2s
      infinite-pod-creation-7688f685c7-6kxjg   0/1     NodeAffinity   0          1s
      infinite-pod-creation-7688f685c7-74d7m   0/1     NodeAffinity   0          2s
      infinite-pod-creation-7688f685c7-78hzm   0/1     NodeAffinity   0          1s 
      ....
      ...
      ..
      .

      Expected results:

      The scheduler should be able to find a conflict and throw an error 

      Additional info:

      apiVersion: project.openshift.io/v1
      kind: Project
      metadata:
        annotations:
          openshift.io/description: ""
          openshift.io/display-name: ""
          openshift.io/node-selector: node-role.kubernetes.io/worker=
          openshift.io/requester: kube:admin
          openshift.io/sa.scc.mcs: s0:c29,c19
          openshift.io/sa.scc.supplemental-groups: 1000850000/10000
          openshift.io/sa.scc.uid-range: 1000850000/10000
        labels:
          kubernetes.io/metadata.name: infinite-pod-creation
          pod-security.kubernetes.io/audit: restricted
          pod-security.kubernetes.io/audit-version: v1.24
          pod-security.kubernetes.io/warn: restricted
          pod-security.kubernetes.io/warn-version: v1.24
        name: infinite-pod-creation
      spec:
        finalizers:
        - kubernetes
      
      ======================================================
      
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        annotations:
        labels:
          app: infinite-pod-creation
          app.kubernetes.io/component: infinite-pod-creation
          app.kubernetes.io/instance: infinite-pod-creation
        name: infinite-pod-creation
      spec:
        progressDeadlineSeconds: 600
        replicas: 1
        revisionHistoryLimit: 10
        selector:
          matchLabels:
            deployment: infinite-pod-creation
        strategy:
          rollingUpdate:
            maxSurge: 25%
            maxUnavailable: 25%
          type: RollingUpdate
        template:
          metadata:
            annotations:
            creationTimestamp: null
            labels:
              deployment: infinite-pod-creation
              app: infinite-pod-creation
          spec:
            topologySpreadConstraints:
            - maxSkew: 1
              topologyKey: kubernetes.io/hostname
              whenUnsatisfiable: DoNotSchedule
              labelSelector:
                matchLabels:
                  app: infinite-pod-creation
            containers:
            - image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a9db83f67aa4389811bad29af878d038e18bc39f63673fe77fe30f9bf1bd97de
              imagePullPolicy: IfNotPresent
              name: infinite-pod-creation
              ports:
              - containerPort: 8080
                protocol: TCP
              - containerPort: 8888
                protocol: TCP
              resources: {}
              terminationMessagePath: /dev/termination-log
              terminationMessagePolicy: File
            dnsPolicy: ClusterFirst
            restartPolicy: Always
            schedulerName: default-scheduler
            securityContext: {}
            terminationGracePeriodSeconds: 30
       
      ============================================================
      
      oc patch deployment infinite-pod-creation -p '{"spec":{"template":{"spec":{"nodeName": "$NODE-NAME-DIFFERENT-FROM-WORKER-ROLE"}}}}'

              fkrepins@redhat.com Filip Krepinsky
              openshift-crt-jira-prow OpenShift Prow Bot
              ying zhou ying zhou
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: