Currently it's not possible to scale NB/SB db replicas>1 with TLS Enabled(Default).

Fails like:-
Starting ovsdb-nb 2024-05-08T14:04:10Z|00001|vlog|INFO|opened log file /dev/null
2024-05-08T14:04:10Z|00002|raft|INFO|local server ID is 1682
2024-05-08T14:04:10Z|00003|ovsdb_server|INFO|ovsdb-server (Open vSwitch) 3.2.3
2024-05-08T14:04:10Z|00004|raft|WARN|pssl:6643:ovsdbserver-nb-1.ovsdbserver-nb.openstack.svc.cluster.local: listen failed (Resource temporarily unavailable)
2024-05-08T14:04:10Z|00005|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connecting...
2024-05-08T14:04:10Z|00006|stream_ssl|ERR|SSL_connect: unexpected SSL_ERROR_ZERO_RETURN
2024-05-08T14:04:10Z|00007|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connection attempt failed (Protocol error)
2024-05-08T14:04:11Z|00008|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connecting...
2024-05-08T14:04:11Z|00009|stream_ssl|ERR|SSL_connect: unexpected SSL_ERROR_ZERO_RETURN
2024-05-08T14:04:11Z|00010|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connection attempt failed (Protocol error)
2024-05-08T14:04:11Z|00011|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: waiting 2 seconds before reconnect
2024-05-08T14:04:13Z|00012|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connecting...
2024-05-08T14:04:13Z|00013|stream_ssl|ERR|SSL_connect: unexpected SSL_ERROR_ZERO_RETURN
2024-05-08T14:04:13Z|00014|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connection attempt failed (Protocol error)
2024-05-08T14:04:13Z|00015|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: waiting 4 seconds before reconnect
2024-05-08T14:04:17Z|00016|stream_ssl|ERR|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connect: Address family not supported by protocol
2024-05-08T14:04:17Z|00017|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connecting...
2024-05-08T14:04:17Z|00018|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connection attempt failed (Address family not supported by protocol)
2024-05-08T14:04:17Z|00019|reconnect|INFO|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: continuing to reconnect in the background but suppressing further logging
2024-05-08T14:04:20Z|00020|memory|INFO|13620 kB peak resident set size after 10.0 seconds
2024-05-08T14:04:20Z|00021|memory|INFO|atoms:15 cells:20 monitors:0 n-weak-refs:0
2024-05-08T14:04:25Z|00022|stream_ssl|ERR|SSL_connect: unexpected SSL_ERROR_ZERO_RETURN
2024-05-08T14:04:33Z|00023|stream_ssl|ERR|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connect: Address family not supported by protocol
2024-05-08T14:04:41Z|00024|stream_ssl|ERR|SSL_connect: unexpected SSL_ERROR_ZERO_RETURN
2024-05-08T14:04:49Z|00025|stream_ssl|ERR|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connect: Address family not supported by protocol
2024-05-08T14:04:57Z|00026|stream_ssl|ERR|SSL_connect: unexpected SSL_ERROR_ZERO_RETURN
2024-05-08T14:05:05Z|00027|stream_ssl|ERR|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connect: Address family not supported by protocol
2024-05-08T14:05:13Z|00028|stream_ssl|ERR|SSL_connect: unexpected SSL_ERROR_ZERO_RETURN
2024-05-08T14:05:21Z|00029|stream_ssl|ERR|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connect: Address family not supported by protocol
2024-05-08T14:05:29Z|00030|stream_ssl|ERR|SSL_connect: unexpected SSL_ERROR_ZERO_RETURN
2024-05-08T14:05:37Z|00031|stream_ssl|ERR|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connect: Address family not supported by protocol
2024-05-08T14:05:45Z|00032|stream_ssl|ERR|SSL_connect: unexpected SSL_ERROR_ZERO_RETURN
2024-05-08T14:05:53Z|00033|stream_ssl|ERR|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connect: Address family not supported by protocol
2024-05-08T14:06:01Z|00034|stream_ssl|ERR|SSL_connect: unexpected SSL_ERROR_ZERO_RETURN
2024-05-08T14:06:09Z|00035|stream_ssl|ERR|ssl:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643: connect: Address family not supported by protocol
 

On master pod:-
2024-05-08T14:12:25Z|00222|jsonrpc|WARN|Dropped 1 log messages in last 15 seconds (most recently, 15 seconds ago) due to excessive rate
2024-05-08T14:12:25Z|00223|jsonrpc|WARN|tcp:192.168.21.26:36598: error parsing stream: line 0, column 0, byte 0: invalid character U+0016
2024-05-08T14:12:25Z|00224|jsonrpc|WARN|Dropped 1 log messages in last 15 seconds (most recently, 15 seconds ago) due to excessive rate
2024-05-08T14:12:25Z|00225|jsonrpc|WARN|tcp:192.168.21.26:36598: received SSL data on JSON-RPC channel
2024-05-08T14:12:25Z|00226|reconnect|WARN|tcp:192.168.21.26:36598: connection dropped (Protocol error)
2024-05-08T14:12:26Z|00227|jsonrpc|WARN|tcp:192.168.17.66:36532: error parsing stream: line 0, column 0, byte 0: invalid character U+0016
2024-05-08T14:12:26Z|00228|jsonrpc|WARN|tcp:192.168.17.66:36532: received SSL data on JSON-RPC channel
2024-05-08T14:12:26Z|00229|reconnect|WARN|tcp:192.168.17.66:36532: connection dropped (Protocol error)
 

Issue happens as with tls enabled local address still set to tcp:
sh-5.1$ ovs-appctl -t /tmp/ovnnb_db.ctl cluster/status OVN_Northbound
f241
Name: OVN_Northbound
Cluster ID: 78b6 (78b61b3f-02c4-419f-b50a-e88e3de80482)
Server ID: f241 (f241c988-2fa0-42af-ae12-ea6731ead702)
Address: tcp:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643
Status: cluster member
Role: leader
Term: 3
Leader: self
Vote: self

Last Election started 20859047 ms ago, reason: timeout
Last Election won: 20859047 ms ago
Election timer: 10000
Log: [16537, 17756]
Entries not yet committed: 0
Entries not yet applied: 0
Connections:
Disconnections: 180
Servers:
f241 (f241 at tcp:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643) (self) next_index=6 match_index=17755

This happening as somehow even with TLS enabled OVNDB initial started with TLS=false[2] first and db cluster get's initialized with local-addr as tcp:ovsdbserver-nb-0.ovsdbserver-nb.openstack.svc.cluster.local:6643[1] and then it again created with TLS=true[2] but as dbs already bootstrapped local address do not change/switch to ssl.

So need to check why even with TLS enabled why it starts without it, something wrong in openstack-operator side.

On OVN side need to check how to handle tls to non-tls switch or reverse or any other missing case, if that kind of use case supported. Considering local-address from OVN side that's immutable[3].

 [1] https://github.com/ovn-org/ovn/blob/4f614b4b9231f40319e594ef56b727bf76e58926/utilities/ovn-ctl#L261

 [2] https://github.com/openstack-k8s-operators/ovn-operator/blob/main/controllers/ovndbcluster_controller.go#L791

 [3] https://github.com/openvswitch/ovs/blob/main/ovsdb/raft.c#L199