Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14678

When stopping microshift.service, microshift-etcd shuts down independently of microshift and before kube-apiserver

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done-Errata
    • Icon: Major Major
    • None
    • 4.14
    • MicroShift
    • None
    • No
    • uShift Sprint 241
    • 1
    • False
    • Hide

      None

      Show
      None
    • Fixed a problem that prevented the etcd database used by MicroShift from shutting down cleanly in some circumstances.
    • Bug Fix

      Description of problem:

      Running command systemctl stop microshift.service also signals microshift-etcd.scope to stop.
      This means that it's not microshift controlling the etcd shutdown.
      
      This results in etcd stopping to soon - before kube-apiserver had chance to write final items.
      
      Depending on what KAS wanted to persist, it can result in microshift taking around 40-50 second to stop (because it waits for go context to timeout)

      Version-Release number of selected component (if applicable):

      main

      How reproducible:

      100%

      Steps to Reproduce:

      1. Watch `microshift.service` and microshift-etcd.scope` logs side by side
      2. Run `sudo systemctl stop microshift`
      3. Compare time when both processes received interrupt signal
      

      Actual results:

      Jun 07 08:35:26 localhost.localdomain microshift[10939]: I0607 08:35:26.720566   10939 run.go:135] microshift-etcd received signal terminated - stopping
      
      Jun 07 08:35:26 localhost.localdomain microshift[10899]: ??? I0607 08:35:26.721436   10899 run.go:212] Interrupt received
      
      microshift-etcd got signal at 08:35:26.720566
      microshift got signal at 08:35:26.721436 - a little bit later

      Expected results:

      microshift manages microshift-etcd and shuts it down AFTER kube-apiserver (in reverse order to the start up sequence)

      Additional info:

      Added extra debug log to see when microshift wants to shutdown etcd just to be sure:
      
      Jun 07 08:35:26 localhost.localdomain microshift[10899]: etcd I0607 08:35:26.723597   10899 etcd.go:120] "+++PMTK Signalling etcd"
      Jun 07 08:35:26 localhost.localdomain microshift[10899]: etcd I0607 08:35:26.723620   10899 manager.go:123] etcd completed

            eslutsky Evgeny Slutsky
            pmatusza@redhat.com Patryk Matuszak
            John George John George
            Allen Ray
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: