-
Bug
-
Resolution: Done-Errata
-
Major
-
None
-
4.14
-
None
-
Moderate
-
No
-
uShift Sprint 235
-
1
-
False
-
-
-
Bug Fix
Description of problem:
Microshift-etcd doesn't start up with memoryLimitMB set to 50. Even with 80, it throws lot of errors. Need to figure out a minimum permissible value for memoryLimitMB in order for Microshift to work properly.
Version-Release number of selected component (if applicable):
How reproducible:
Always
Steps to Reproduce:
1. edit config.yaml file in path /etc/microshift/. etcd: memoryLimitMB: 80 2.sudo systemctl restart microshift 3.systemctl status microshift 4.journalctl -u microshift-etcd.scope
Actual results:
systemctl status microshift ● microshift.service - MicroShift Loaded: loaded (/usr/lib/systemd/system/microshift.service; enabled; preset: disabled) Active: activating (start) since Thu 2023-04-06 12:11:15 EDT; 1min 28s ago Main PID: 98609 (microshift) Tasks: 8 (limit: 22857) Memory: 27.1M CPU: 761ms CGroup: /system.slice/microshift.service └─98609 microshift runApr 06 12:11:58 localhost.localdomain microshift[98609]: "Metadata": null Apr 06 12:11:58 localhost.localdomain microshift[98609]: }. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded" Apr 06 12:12:20 localhost.localdomain microshift[98609]: etcd W0406 12:12:20.716382 98609 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to { Apr 06 12:12:20 localhost.localdomain microshift[98609]: "Addr": "localhost:2379", Apr 06 12:12:20 localhost.localdomain microshift[98609]: "ServerName": "localhost", Apr 06 12:12:20 localhost.localdomain microshift[98609]: "Attributes": null, Apr 06 12:12:20 localhost.localdomain microshift[98609]: "BalancerAttributes": null, Apr 06 12:12:20 localhost.localdomain microshift[98609]: "Type": 0, Apr 06 12:12:20 localhost.localdomain microshift[98609]: "Metadata": null Apr 06 12:12:20 localhost.localdomain microshift[98609]: }. Err: connection error: desc = "transport: authentication handshake failed: context deadline exceeded" ---------------------------------------------------------------------------- [redhat@localhost microshift]$ journalctl -u microshift-etcd.scope I0406 04:47:14.013658 8798 run.go:199] backend store fragmented: 28.68 %, dbSize: 2813952 Apr 06 04:47:15 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:15.444-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"135.803258ms","expected-> Apr 06 04:47:15 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:15.459-0400","caller":"traceutil/trace.go:171","msg":"trace[42915613] transaction","detail":"{read_only:false; resp> Apr 06 04:47:16 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:16.019-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"184.833464ms","expected-> Apr 06 04:47:16 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:16.039-0400","caller":"traceutil/trace.go:171","msg":"trace[108528780] compact","detail":"{revision:17115; response> Apr 06 04:47:16 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:16.020-0400","caller":"mvcc/index.go:214","msg":"compact tree index","revision":17115} Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:18.684-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"1.204397124s","expected-> Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:18.697-0400","caller":"traceutil/trace.go:171","msg":"trace[766890132] linearizableReadLoop","detail":"{readStateIn> Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:18.697-0400","caller":"traceutil/trace.go:171","msg":"trace[1515916930] transaction","detail":"{read_only:false; re> Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:18.723-0400","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-04-06T04:47:17.479-0400",> Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:18.723-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"164.799766ms","expected-> Apr 06 04:47:18 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:18.723-0400","caller":"traceutil/trace.go:171","msg":"trace[1607496797] range","detail":"{range_begin:/kubernetes.i> Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.050-0400","caller":"mvcc/kvstore_compaction.go:66","msg":"finished scheduled compaction","compact-revision":1711> Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.050-0400","caller":"mvcc/hash.go:137","msg":"storing new hash","hash":765658342,"revision":17115,"compact-revisi> Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.368-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"361.603455ms","expected-> Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.369-0400","caller":"traceutil/trace.go:171","msg":"trace[116310655] linearizableReadLoop","detail":"{readStateIn> Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.369-0400","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-04-06T04:47:19.006-0400",> Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.369-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"361.954023ms","expected-> Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"info","ts":"2023-04-06T04:47:19.369-0400","caller":"traceutil/trace.go:171","msg":"trace[1030155816] range","detail":"{range_begin:/kubernetes.i> Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.369-0400","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-04-06T04:47:19.007-0400",> Apr 06 04:47:19 localhost.localdomain microshift[8798]: {"level":"warn","ts":"2023-04-06T04:47:19.383-0400","caller":"etcdserver/util.go:166","msg":"apply request took too long","took":"125.33249ms","expected-d>
Please note : Once the memoryLimitMB is set to 128 or more, It works fine.
[redhat@localhost microshift]$ systemctl status microshift-etcd.scope ● microshift-etcd.scope - /usr/bin/microshift-etcd run Loaded: loaded (/run/systemd/transient/microshift-etcd.scope; transient) Transient: yes Active: active (running) since Thu 2023-04-06 12:22:04 EDT; 28s ago Tasks: 9 (limit: 22857) Memory: 109.6M (high: 128.0M available: 18.3M) CPU: 910ms CGroup: /system.slice/microshift-etcd.scope └─99808 /usr/bin/microshift-etcd runApr 06 12:22:06 localhost.localdomain microshift[99808]: {"level":"info","ts":"2023-04-06T12:22:06.565-0400","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"a70d4343126 .............................................. [redhat@localhost microshift]$ systemctl status microshift ● microshift.service - MicroShift Loaded: loaded (/usr/lib/systemd/system/microshift.service; enabled; preset: disabled) Active: active (running) since Thu 2023-04-06 12:22:42 EDT; 4s ago Main PID: 99798 (microshift) Tasks: 11 (limit: 22857) Memory: 357.5M CPU: 7.370s CGroup: /system.slice/microshift.service └─99798 microshift runApr 06 12:22:42 localhost.localdomain microshift[99798]: kubelet I0406 12:22:42.470407 99798 prober_manager.go:287] "Failed to trigger a manual run" probe="Readiness" Apr 06 12:22:42 localhost.localdomain microshift[99798]: kubelet I0406 12:22:42.470477 99798 prober_manager.go:287] "Failed to trigger a manual run" probe="Readiness"
Expected results:
Microshift-etcd should run successfully even after configuring memoryLimitMB to a minimum permissible value.
Additional info:
- is depended on by
-
OCPBUGS-11497 Microshift-etcd doesn't start up with memoryLimitMB set to 50.
- Closed
- links to
-
RHSA-2023:5008 OpenShift Container Platform 4.14.z security update