Normal Description of problem:
ETCD showing multiple warnings like this:
2024-10-25T05:44:29.451067060Z {"level":"warn","ts":"2024-10-25T05:44:29.44813Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"[::1]:49220","server-name":"localhost","error":"EOF"}
2024-10-25T05:55:29.456816446Z {"level":"warn","ts":"2024-10-25T05:55:29.456708Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"[::1]:40788","server-name":"localhost","error":"EOF"}
2024-10-25T06:15:49.292624944Z {"level":"warn","ts":"2024-10-25T06:15:49.290607Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.102.12.7:55766","server-name":"","error":"EOF"}
2024-10-25T06:19:28.789618300Z {"level":"warn","ts":"2024-10-25T06:19:28.778419Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"10.102.12.7:50446","server-name":"","error":"EOF"}
The port number mentioned in the error above changes between warnings time to time and is not related to a firewall blocking the connections as verified.
The port numbers, as indicated above, are also not documented[1] in our official documentation for allowing access in the firewall or infra
There are no pending CSRs so this does not appear to be a certificate issue.
The etcd pods appear to be healthy:
$ omc get po -n openshift-etcd NAME READY STATUS RESTARTS AGE etcd-guard-svitmadocpro01 1/1 Running 0 98d etcd-guard-svitmadocpro02 1/1 Running 1 98d etcd-guard-svitmadocpro03 1/1 Running 2 98d etcd-svitmadocpro01 4/4 Running 4 98d etcd-svitmadocpro02 4/4 Running 9 98d etcd-svitmadocpro03 4/4 Running 12 98d revision-pruner-12-svitmadocpro01 0/1 Completed 0 98d revision-pruner-12-svitmadocpro02 0/1 Completed 0 98d revision-pruner-12-svitmadocpro03 0/1 Completed 0 98d
Based on the below test, ETCD health looks to be fine:
sh-5.1# etcdctl member list -w table +------------------+---------+----------------+-----------------------------+-----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+----------------+-----------------------------+-----------------------------+------------+ | 329b4a7d0a3109a0 | started | svitmadocpro01 | https://10.102.12.6:2380 | https://10.102.12.6:2379 | false | | d75585bb6c33f85c | started | svitmadocpro03 | https://10.223.254.235:2380 | https://10.223.254.235:2379 | false | | e170ffaf8e78163e | started | svitmadocpro02 | https://10.102.12.7:2380 | https://10.102.12.7:2379 | false | +------------------+---------+----------------+-----------------------------+-----------------------------+------------+ sh-5.1# etcdctl endpoint health --cluster https://10.102.12.6:2379 is healthy: successfully committed proposal: took = 10.140173ms https://10.102.12.7:2379 is healthy: successfully committed proposal: took = 15.744742ms https://10.223.254.235:2379 is healthy: successfully committed proposal: took = 22.150999ms sh-5.1# etcdctl endpoint status -w table +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://10.102.12.6:2379 | 329b4a7d0a3109a0 | 3.5.14 | 233 MB | true | false | 83 | 200457558 | 200457558 | | | https://10.102.12.7:2379 | e170ffaf8e78163e | 3.5.14 | 242 MB | false | false | 83 | 200457558 | 200457558 | | | https://10.223.254.235:2379 | d75585bb6c33f85c | 3.5.14 | 233 MB | false | false | 83 | 200457558 | 200457558 | | +-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ sh-5.1#
Version-Release number of selected component (if applicable):
OpenShift v4.16.2
How reproducible:
Steps to Reproduce:
1.
2.
3.
Actual results:
ETCD showing multiple warnings and port numbers mentioned in the error messages are also not being used inside the cluster anywhere.
Expected results:
ETCD should not show multiple warnings like this.
Additional info:
