Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-1274

machine-api-termination-handler Pods don't launch on tainted spot instances

XMLWordPrintable

    • None
    • CLOUD Sprint 226
    • 1
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, Machine API termination handler pods did not launch on spot instances. As a result, pods that were running on tainted spot instances did not receive a termination signal if the instance was terminated. This could result in loss of data in workload applications. With this release, the Machine API termination handler deployment is modified to tolerate the taints andpods running on spot instances with taints now receive termination signals.
      (link:https://issues.redhat.com/browse/OCPBUGS-1274[*OCPBUGS-1274*)
      Show
      * Previously, Machine API termination handler pods did not launch on spot instances. As a result, pods that were running on tainted spot instances did not receive a termination signal if the instance was terminated. This could result in loss of data in workload applications. With this release, the Machine API termination handler deployment is modified to tolerate the taints andpods running on spot instances with taints now receive termination signals. (link: https://issues.redhat.com/browse/OCPBUGS-1274 [* OCPBUGS-1274 *)
    • Bug Fix
    • Done

      Description of problem:

      When spot instances with taints are added to the cluster on AWS, machine-api-termination-handler daemonset pods do not launch on these instances because of the taints. 

machine-api-termination-handler is used for checking the notification of  intance termination, so if it doesn't launch properly, application pods on spot instances could stop without normal shut down procedures. 

It is common to use taint-toleration to specify workloads on spot instances, because it does not require changing application manifests of other workloads. 

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      100%

      Steps to Reproduce:

      1. Creating ROSA cluster
2. Adding spot instances with taints on OCM
3. oc get daemonset machine-api-termination-handler -n openshift-machine-api

      Actual results:

      machine-api-termination-handler pods do not launch on spot instances

      Expected results:

      machine-api-termination-handler pods launch on spot instances

      Additional info:

      Adding followings to machine-api-termination-handler daemonset could resolve the problem.
---  
tolerations:        
- operator: Exists

        1. taints-spot.png
          101 kB
          Junpei Ishikawa

              mimccune@redhat.com Michael McCune
              jishikaw2@redhat.com Junpei Ishikawa
              Zhaohua Sun Zhaohua Sun
              Jeana Routh Jeana Routh
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: