Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-11497

Microshift-etcd doesn't start up with memoryLimitMB set to 50.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Major
    • 4.13.0
    • 4.13
    • MicroShift
    • None
    • Moderate
    • No
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      Microshift-etcd doesn't start up with memoryLimitMB set to 50. Even with 80, it throws lot of errors. 
      Need to figure out a minimum permissible value for memoryLimitMB in order for Microshift to work properly. 

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      Always

      Steps to Reproduce:

      1. edit config.yaml file in path /etc/microshift/. 
      etcd:
        memoryLimitMB: 80
      
      2.sudo systemctl restart microshift
      3.systemctl status microshift
      4.journalctl -u microshift-etcd.scope  

      Actual results:

      systemctl status microshift
      ● microshift.service - MicroShift
           Loaded: loaded (/usr/lib/systemd/system/microshift.service; enabled; preset: disabled)
           Active: activating (start) since Thu 2023-04-06 12:11:15 EDT; 1min 28s ago
         Main PID: 98609 (microshift)
            Tasks: 8 (limit: 22857)
           Memory: 27.1M
              CPU: 761ms
           CGroup: /system.slice/microshift.service
                   └─98609 microshift runApr 06 12:11:58 localhost.localdomain microshift[98609]:   "Metadata": null
      Apr 06 12:11:58 localhost.localdomain microshift[98609]: }. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
      Apr 06 12:12:20 localhost.localdomain microshift[98609]: etcd W0406 12:12:20.716382   98609 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "Addr": "localhost:2379",
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "ServerName": "localhost",
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "Attributes": null,
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "BalancerAttributes": null,
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "Type": 0,
      Apr 06 12:12:20 localhost.localdomain microshift[98609]:   "Metadata": null
      Apr 06 12:12:20 localhost.localdomain microshift[98609]: }. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded"
      ----------------------------------------------------------------------------
      
      [redhat@localhost microshift]$ journalctl -u microshift-etcd.scope
      
      
      I0406 04:47:14.013658    8798 run.go:199] backend store fragmented: 28.68 %, dbSize: 2813952
      Apr 06 04:47:15 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:15.444-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"135.803258ms","expected->
      Apr 06 04:47:15 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:15.459-0400","caller":"traceutil/trace.go:171","msg":"trace[42915613] transaction","detail":"{read_only:false; resp>
      Apr 06 04:47:16 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:16.019-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"184.833464ms","expected->
      Apr 06 04:47:16 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:16.039-0400","caller":"traceutil/trace.go:171","msg":"trace[108528780] compact","detail":"{revision:17115; response>
      Apr 06 04:47:16 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:16.020-0400","caller":"mvcc/index.go:214","msg":"compact tree index","revision":17115}
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:18.684-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"1.204397124s","expected->
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:18.697-0400","caller":"traceutil/trace.go:171","msg":"trace[766890132] linearizableReadLoop","detail":"{readStateIn>
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:18.697-0400","caller":"traceutil/trace.go:171","msg":"trace[1515916930] transaction","detail":"{read_only:false; re>
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:18.723-0400","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-04-06T04:47:17.479-0400",>
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:18.723-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"164.799766ms","expected->
      Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:18.723-0400","caller":"traceutil/trace.go:171","msg":"trace[1607496797] range","detail":"{range_begin:/kubernetes.i>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.050-0400","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":1711>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.050-0400","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":765658342,"revision":17115,"compact-revisi>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.368-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"361.603455ms","expected->
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.369-0400","caller":"traceutil/trace.go:171","msg":"trace[116310655] linearizableReadLoop","detail":"{readStateIn>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.369-0400","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-04-06T04:47:19.006-0400",>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.369-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"361.954023ms","expected->
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.369-0400","caller":"traceutil/trace.go:171","msg":"trace[1030155816] range","detail":"{range_begin:/kubernetes.i>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.369-0400","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-04-06T04:47:19.007-0400",>
      Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.383-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"125.33249ms","expected-d>
      
      
      
      
      

      Please note : Once the memoryLimitMB is set to 128 or more, It works fine. 

      [redhat@localhost microshift]$ systemctl status microshift-etcd.scope
      ● microshift-etcd.scope - /usr/bin/microshift-etcd run
           Loaded: loaded (/run/systemd/transient/microshift-etcd.scope; transient)
        Transient: yes
           Active: active (running) since Thu 2023-04-06 12:22:04 EDT; 28s ago
            Tasks: 9 (limit: 22857)
           Memory: 109.6M (high: 128.0M available: 18.3M)
              CPU: 910ms
           CGroup: /system.slice/microshift-etcd.scope
                   └─99808 /usr/bin/microshift-etcd runApr 06 12:22:06 localhost.localdomain microshift[99808]: {"level":"info","ts":"2023-04-06T12:22:06.565-0400","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a70d4343126
      ..............................................
      
      
      [redhat@localhost microshift]$ systemctl status microshift
      ● microshift.service - MicroShift
           Loaded: loaded (/usr/lib/systemd/system/microshift.service; enabled; preset: disabled)
           Active: active (running) since Thu 2023-04-06 12:22:42 EDT; 4s ago
         Main PID: 99798 (microshift)
            Tasks: 11 (limit: 22857)
           Memory: 357.5M
              CPU: 7.370s
           CGroup: /system.slice/microshift.service
                   └─99798 microshift runApr 06 12:22:42 localhost.localdomain microshift[99798]: kubelet I0406 12:22:42.470407   99798 prober_manager.go:287] "Failed to trigger a manual run" probe="Readiness"
      Apr 06 12:22:42 localhost.localdomain microshift[99798]: kubelet I0406 12:22:42.470477   99798 prober_manager.go:287] "Failed to trigger a manual run" probe="Readiness"
      
       

      Expected results:

      Microshift-etcd should run successfully even after configuring memoryLimitMB to a minimum permissible value.

      Additional info:

       

      Attachments

        Activity

          People

            alray@redhat.com Allen Ray
            rhn-support-skundu Sandeep Kundu
            Sandeep Kundu Sandeep Kundu
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: