Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-5031

Request to support AWS spot instance for lokistack configuration.

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Normal Normal
    • None
    • None
    • Log Storage
    • False
    • None
    • False
    • NEW
    • NEW
    • If Release Note Needed, Set a Value

      Request to support AWS spot instance for lokistack configuration.

      After configuring node-selector to schedule the loki pods to spot instance infra nodes.

      LokiStack works fine initially, after some time the ingester started to fail, being in running state but with 0/1 replicas and showing the following message on the events: 
      Readiness probe error: HTTP probe failed with statuscode: 503 body: Some services are not Running: Starting: 1 Running: 6
      The below events are present that node was not ready:

      Unknown Warning FailedScheduling pod/logging-loki-index-gateway-0 0/8 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 2 node(s) didn't match Pod's node affinity/selector, 2 node(s) had volume node affinity conflict, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/8 nodes are available: 8 Preemption is not helpful for scheduling.
      Unknown Warning FailedScheduling pod/logging-loki-index-gateway-0 0/9 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 2 node(s) had volume node affinity conflict, 3 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/9 nodes are available: 9 Preemption is not helpful for scheduling.

       

      After LokiStack re-installation from the scratch, the behavior was exactly the same, at first everything was working fine, but somehow the ingester begin to fail again, being on exactly the same state as we've described before.

       

      Currently lokistack is not able to run on AWS spot instances properly.

       

      Also, the next time, I'd recommend to go through the Collaboration workflow asking collab from a senior support engineering before opening a bug.

       

            ptsiraki@redhat.com Periklis Tsirakidis
            rhn-support-amanverm Aman Dev Verma
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: