Loading...

Type: Task
Resolution: Unresolved
Priority: Undefined
Fix Version/s: None
Affects Version/s: None
Component/s: openvswitch3.3
Labels:
- OVS-QE-Test-Coverage

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Epic Link:
ovn db upgrade to the new schema not happening with the larger DBs
Acceptance Criteria:

Hide

( ) The test coverage is aligned with the epic's acceptance criteria

Show
( ) The test coverage is aligned with the epic's acceptance criteria
OS:
rhel-9
Planning:
None
Intelligence Requested:
Market:

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

This task is tracking the test case writing activities to cover the bug described below.

Problem Description: Clearly explain the issue.

This was observed when Customer performing FFU for their OpenStack cloud as part of https://issues.redhat.com/browse/RHOSPPRIO-839. As part of the FFU when SB DB was upgraded to newer version OVN24.03 from OVN-2021, SB DB was not upgraded to new schema[3], NB went fine

NB/SB db size[2]

The root cause is hard coded time to 30 seconds[A] and non handling of the failure(when SB DB not ready) during the db upgrade. With local check having 60+ timeout it went fine. So having some higher timeout by default and may be having an option to customize for some larger db would be good addition.

[A] https://github.com/openvswitch/ovs/blob/c8ec54a598dd0e78479304b518b82df82b8f3df0/utilities/ovs-lib.in#L496-L505

Impact Assessment:

All components OVN Controller, OVN NB db, OVN Northd upgraded and complaining about missing tables/columns[1].

Software Versions: Specify the exact versions in use

ovn24.03-24.03.6-54.el9fdp

openvswitch3.3-3.3.6-141.el9fdp.x86_64

Issue Type: New and observed when large DB are involved as in this case

Reproducibility: Yes with such db size can be reproduced consistantly

Reproduction Steps: Provide detailed steps or scripts to replicate the issue.

Expected Behavior: DB upgrade to the new schema should succeed

Observed Behavior: DB not got updated to the new schema and required manual conversion

Troubleshooting Actions:

Manual recovery done using below command:-

ovsdb-client convert unix:/var/run/ovn/ovnsb_db.sock /usr/share/ovn/ovn-sb.ovsschema -v

Locally importing same db issue can be reproduced consistently.

[root@controller-0 /]# time ovsdb-tool compact ovnsb_db.db.backup20.21.0-2362446865
real    1m15.567s
user    1m10.549s
sys     0m3.377s

[root@controller-0 /]# time ovsdb-tool convert ovnsb_db.db.backup20.21.0-2362446865 /usr/share/ovn/ovn-sb.ovsschema -v
2026-02-08T08:32:23Z|00001|hmap|DBG|../lib/shash.c:114: 1 bucket with 6+ nodes, including 1 bucket with 7 nodes (32 nodes total across 32 buckets)
2026-02-08T08:32:23Z|00002|hmap|DBG|../lib/shash.c:114: 1 bucket with 6+ nodes, including 1 bucket with 7 nodes (8 nodes total across 8 buckets)
2026-02-08T08:32:23Z|00003|hmap|DBG|../lib/shash.c:114: 1 bucket with 6+ nodes, including 1 bucket with 7 nodes (32 nodes total across 32 buckets)
2026-02-08T08:32:23Z|00004|hmap|DBG|../lib/shash.c:114: 1 bucket with 6+ nodes, including 1 bucket with 7 nodes (8 nodes total across 8 buckets)
2026-02-08T08:32:23Z|00005|hmap|DBG|../lib/shash.c:114: 1 bucket with 6+ nodes, including 1 bucket with 7 nodes (32 nodes total across 32 buckets)
2026-02-08T08:32:23Z|00006|hmap|DBG|../lib/shash.c:114: 1 bucket with 6+ nodes, including 1 bucket with 7 nodes (8 nodes total across 8 buckets)
2026-02-08T08:32:23Z|00007|hmap|DBG|../lib/shash.c:114: 1 bucket with 6+ nodes, including 1 bucket with 7 nodes (32 nodes total across 32 buckets)
2026-02-08T08:32:23Z|00008|hmap|DBG|../lib/shash.c:114: 1 bucket with 6+ nodes, including 1 bucket with 7 nodes (8 nodes total across 8 buckets)
2026-02-08T08:32:23Z|00009|hmap|DBG|../lib/shash.c:114: 1 bucket with 6+ nodes, including 1 bucket with 7 nodes (32 nodes total across 32 buckets)
2026-02-08T08:32:23Z|00010|hmap|DBG|../lib/shash.c:114: 2 buckets with 6+ nodes, including 1 bucket with 8 nodes (128 nodes total across 128 buckets)
2026-02-08T08:32:30Z|00011|hmap|DBG|Dropped 58 log messages in last 7 seconds (most recently, 3 seconds ago) due to excessive rate
2026-02-08T08:32:30Z|00012|hmap|DBG|../lib/shash.c:114: 7847 buckets with 6+ nodes, including 1 bucket with 13 nodes (524288 nodes total across 524288 buckets)
2026-02-08T08:32:35Z|00013|hmap|DBG|Dropped 145 log messages in last 2 seconds (most recently, 0 seconds ago) due to excessive rate
2026-02-08T08:32:35Z|00014|hmap|DBG|../ovsdb/transaction.c:1496: 2160 buckets with 6+ nodes, including 5 buckets with 10 nodes (262144 nodes total across 262144 buckets)
2026-02-08T08:32:46Z|00015|hmap|DBG|Dropped 57 log messages in last 10 seconds (most recently, 6 seconds ago) due to excessive rate
2026-02-08T08:32:46Z|00016|hmap|DBG|../ovsdb/transaction.c:1017: 1 bucket with 6+ nodes, including 1 bucket with 6 nodes (32 nodes total across 32 buckets)
2026-02-08T08:32:47Z|00017|hmap|DBG|Dropped 1322 log messages in last 2 seconds (most recently, 1 seconds ago) due to excessive rate
2026-02-08T08:32:47Z|00018|hmap|DBG|../ovsdb/transaction.c:549: 2 buckets with 6+ nodes, including 1 bucket with 7 nodes (128 nodes total across 128 buckets)
2026-02-08T08:32:54Z|00019|hmap|DBG|Dropped 69 log messages in last 6 seconds (most recently, 2 seconds ago) due to excessive rate
2026-02-08T08:32:54Z|00020|hmap|DBG|../lib/shash.c:114: 4347 buckets with 6+ nodes, including 1 bucket with 12 nodes (524288 nodes total across 524288 buckets)
real    1m2.068s
user    0m57.414s
sys     0m3.266s 


[root@controller-0 /]# ovsdb-client -t 20 wait unix:/var/run/ovn/ovnsb_db.sock OVN_Southbound connected
2026-02-09T05:02:16Z|00001|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting...
2026-02-09T05:02:16Z|00002|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (Connection refused)
2026-02-09T05:02:17Z|00003|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting...
2026-02-09T05:02:17Z|00004|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (Connection refused)
2026-02-09T05:02:17Z|00005|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: waiting 2 seconds before reconnect
2026-02-09T05:02:19Z|00006|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting...
2026-02-09T05:02:19Z|00007|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (Connection refused)
2026-02-09T05:02:19Z|00008|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: waiting 4 seconds before reconnect
2026-02-09T05:02:23Z|00009|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connecting...
2026-02-09T05:02:23Z|00010|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: connection attempt failed (Connection refused)
2026-02-09T05:02:23Z|00011|reconnect|INFO|unix:/var/run/ovn/ovnsb_db.sock: continuing to reconnect in the background but suppressing further logging
2026-02-09T05:02:36Z|00012|fatal_signal|WARN|terminating with signal 14 (Alarm clock)
Alarm clock
[root@controller-0 /]# echo $?
142

[root@controller-0 /]# time ovsdb-client -t 30 convert unix:/var/run/ovn/ovnsb_db.sock /usr/share/ovn/ovn-sb.ovsschema
2026-02-09T05:24:09Z|00001|ovsdb|WARN|/usr/share/ovn/ovn-sb.ovsschema: changed 2 columns in 'OVN_Southbound' database from ephemeral to persistent, including 'status' column in 'Connection' table, because clusters do not support ephemeral columns
real    0m13.100s
user    0m0.007s
sys     0m0.005s

Logs: Are included in the case

[1]

2026-02-06T02:23:53.109Z|02811|ovsdb_idl|WARN|BFD table in OVN_Southbound database lacks chassis_name column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02812|ovsdb_idl|WARN|OVN_Southbound database lacks Chassis_Template_Var table (database needs upgrade?)
2026-02-06T02:23:53.109Z|02813|ovsdb_idl|WARN|DNS table in OVN_Southbound database lacks options column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02814|ovsdb_idl|WARN|FDB table in OVN_Southbound database lacks timestamp column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02815|ovsdb_idl|WARN|IGMP_Group table in OVN_Southbound database lacks chassis_name column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02816|ovsdb_idl|WARN|Load_Balancer table in OVN_Southbound database lacks datapath_group column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02817|ovsdb_idl|WARN|Load_Balancer table in OVN_Southbound database lacks lr_datapath_group column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02818|ovsdb_idl|WARN|Load_Balancer table in OVN_Southbound database lacks ls_datapath_group column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02819|ovsdb_idl|WARN|MAC_Binding table in OVN_Southbound database lacks timestamp column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02820|ovsdb_idl|WARN|OVN_Southbound database lacks Mirror table (database needs upgrade?)
2026-02-06T02:23:53.109Z|02821|ovsdb_idl|WARN|Port_Binding table in OVN_Southbound database lacks additional_chassis column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02822|ovsdb_idl|WARN|Port_Binding table in OVN_Southbound database lacks additional_encap column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02823|ovsdb_idl|WARN|Port_Binding table in OVN_Southbound database lacks mirror_rules column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02824|ovsdb_idl|WARN|Port_Binding table in OVN_Southbound database lacks port_security column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02825|ovsdb_idl|WARN|Port_Binding table in OVN_Southbound database lacks requested_additional_chassis column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02826|ovsdb_idl|WARN|Service_Monitor table in OVN_Southbound database lacks chassis_name column (database needs upgrade?)
2026-02-06T02:23:53.109Z|02827|ovsdb_idl|WARN|OVN_Southbound database lacks Static_MAC_Binding table (database needs upgrade?)

[2]

$ ls -ltrh 0570-openvswitch.tar.gz/var/lib/openvswitch/ovn/*backup*
 -rw-rw-rw-+ 1 yank yank  60M Feb  5 23:48 0570-openvswitch.tar.gz/var/lib/openvswitch/ovn/ovnnb_db.db.backup5.35.2-1159513746
 -rw-rw-rw-+ 1 yank yank 407M Feb  5 23:49 0570-openvswitch.tar.gz/var/lib/openvswitch/ovn/ovnsb_db.db.backup20.21.0-2362446865

[3]

[root@overcloud-controller-0 /]# ovsdb-client get-schema-version unix:/var/run/openvswitch/ovnsb\_db.sock OVN\_Southbound
20.21.0     

root@overcloud-controller-0 /]# ovsdb-client needs-conversion unix:/var/run/openvswitch/ovnsb_db.sock /usr/share/ovn/ovn-sb.ovsschema 
2026-02-06T13:49:35Z|00001|ovsdb|WARN|/usr/share/ovn/ovn-sb.ovsschema: changed 2 columns in 'OVN_Southbound' database from ephemeral to persistent, including 'status' column in 'Connection' table, because clusters do not support ephemeral columns                                                    
yes

Details

Description

Problem Description: Clearly explain the issue.

Impact Assessment:

Software Versions: Specify the exact versions in use

Issue Type: New and observed when large DB are involved as in this case

Reproducibility: Yes with such db size can be reproduced consistantly

Reproduction Steps: Provide detailed steps or scripts to replicate the issue.

Expected Behavior: DB upgrade to the new schema should succeed

Observed Behavior: DB not got updated to the new schema and required manual conversion

Troubleshooting Actions:

Logs: Are included in the case

Attachments

Easy Agile Planning Poker

Activity

People

Dates