Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-11592

Microshift-etcd doesn't start up with memoryLimitMB set to 50.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done-Errata
    • Major
    • None
    • 4.14
    • MicroShift
    • None
    • Moderate
    • No
    • uShift Sprint 235
    • 1
    • False
    • Hide

      None

      Show
      None
    • Hide
      Before this update, MicroShift silently adjusted the maximum memory limit for etcd to match the minimum supported level, if the value in the configuration file was too small. With this update, MicroShift enforces valid configuration by exiting with an error if the memory limit for etcd is set below the minimum allowed value of 128 MB.
      Show
      Before this update, MicroShift silently adjusted the maximum memory limit for etcd to match the minimum supported level, if the value in the configuration file was too small. With this update, MicroShift enforces valid configuration by exiting with an error if the memory limit for etcd is set below the minimum allowed value of 128 MB.
    • Bug Fix

    Description

      Description of problem:

      Microshift-etcd doesn't start up with memoryLimitMB set to 50. Even with 80, it throws lot of errors. 
      Need to figure out a minimum permissible value for memoryLimitMB in order for Microshift to work properly. 

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      Always

      Steps to Reproduce:

      1. edit config.yaml file in path /etc/microshift/. 
      etcd:
        memoryLimitMB: 80
      
      2.sudo systemctl restart microshift
      3.systemctl status microshift
      4.journalctl -u microshift-etcd.scope  

      Actual results:

      systemctl status microshift
      ● microshift.service - MicroShift
           Loaded: loaded (/usr/lib/systemd/system/microshift.service; enabled; preset: disabled)
           Active: activating (start) since Thu 2023-04-06 12:11:15 EDT; 1min 28s ago
         Main PID: 98609 (microshift)
            Tasks: 8 (limit: 22857)
           Memory: 27.1M
              CPU: 761ms
           CGroup: /system.slice/microshift.service
                   └─98609 microshift runApr 06 12:11:58 localhost.localdomain microshift[98609]:   "Metadata": null
      Apr 06 12:11:58 localhost.localdomain microshift[98609]: }. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
      Apr 06 12:12:20 localhost.localdomain microshift[98609]: etcd W0406 12:12:20.716382   98609 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "Addr": "localhost:2379",
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "ServerName": "localhost",
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "Attributes": null,
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "BalancerAttributes": null,
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "Type": 0,
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "Metadata": null
      Apr 06 12:12:20 localhost.localdomain microshift[98609]: }. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
      ----------------------------------------------------------------------------
      
      [redhat@localhost microshift]$ journalctl -u microshift-etcd.scope
      
      
      I0406 04:47:14.013658    8798 run.go:199] backend store fragmented: 28.68 %, dbSize: 2813952
      Apr 06 04:47:15 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:15.444-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"135.803258ms","expected->
      Apr 06 04:47:15 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:15.459-0400","caller":"traceutil/trace.go:171","msg":"trace[42915613] transaction","detail":"{read_only:false; resp>
      Apr 06 04:47:16 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:16.019-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"184.833464ms","expected->
      Apr 06 04:47:16 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:16.039-0400","caller":"traceutil/trace.go:171","msg":"trace[108528780] compact","detail":"{revision:17115; response>
      Apr 06 04:47:16 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:16.020-0400","caller":"mvcc/index.go:214","msg":"compact tree index","revision":17115}
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:18.684-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"1.204397124s","expected->
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:18.697-0400","caller":"traceutil/trace.go:171","msg":"trace[766890132] linearizableReadLoop","detail":"{readStateIn>
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:18.697-0400","caller":"traceutil/trace.go:171","msg":"trace[1515916930] transaction","detail":"{read_only:false; re>
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:18.723-0400","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-04-06T04:47:17.479-0400",>
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:18.723-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"164.799766ms","expected->
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:18.723-0400","caller":"traceutil/trace.go:171","msg":"trace[1607496797] range","detail":"{range_begin:/kubernetes.i>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.050-0400","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":1711>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.050-0400","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":765658342,"revision":17115,"compact-revisi>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.368-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"361.603455ms","expected->
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.369-0400","caller":"traceutil/trace.go:171","msg":"trace[116310655] linearizableReadLoop","detail":"{readStateIn>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.369-0400","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-04-06T04:47:19.006-0400",>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.369-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"361.954023ms","expected->
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.369-0400","caller":"traceutil/trace.go:171","msg":"trace[1030155816] range","detail":"{range_begin:/kubernetes.i>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.369-0400","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-04-06T04:47:19.007-0400",>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.383-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"125.33249ms","expected-d>
      
      
      
      
      

      Please note : Once the memoryLimitMB is set to 128 or more, It works fine. 

      [redhat@localhost microshift]$ systemctl status microshift-etcd.scope
      ● microshift-etcd.scope - /usr/bin/microshift-etcd run
           Loaded: loaded (/run/systemd/transient/microshift-etcd.scope; transient)
        Transient: yes
           Active: active (running) since Thu 2023-04-06 12:22:04 EDT; 28s ago
            Tasks: 9 (limit: 22857)
           Memory: 109.6M (high: 128.0M available: 18.3M)
              CPU: 910ms
           CGroup: /system.slice/microshift-etcd.scope
                   └─99808 /usr/bin/microshift-etcd runApr 06 12:22:06 localhost.localdomain microshift[99808]: {"level":"info","ts":"2023-04-06T12:22:06.565-0400","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a70d4343126
      ..............................................
      
      
      [redhat@localhost microshift]$ systemctl status microshift
      ● microshift.service - MicroShift
           Loaded: loaded (/usr/lib/systemd/system/microshift.service; enabled; preset: disabled)
           Active: active (running) since Thu 2023-04-06 12:22:42 EDT; 4s ago
         Main PID: 99798 (microshift)
            Tasks: 11 (limit: 22857)
           Memory: 357.5M
              CPU: 7.370s
           CGroup: /system.slice/microshift.service
                   └─99798 microshift runApr 06 12:22:42 localhost.localdomain microshift[99798]: kubelet I0406 12:22:42.470407   99798 prober_manager.go:287] "Failed to trigger a manual run" probe="Readiness"
      Apr 06 12:22:42 localhost.localdomain microshift[99798]: kubelet I0406 12:22:42.470477   99798 prober_manager.go:287] "Failed to trigger a manual run" probe="Readiness"
      
       

      Expected results:

      Microshift-etcd should run successfully even after configuring memoryLimitMB to a minimum permissible value.

      Additional info:

       

      Attachments

        Issue Links

          Activity

            People

              alray@redhat.com Allen Ray
              rhn-support-skundu Sandeep Kundu
              Sandeep Kundu Sandeep Kundu
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: