-
Bug
-
Resolution: Done-Errata
-
Critical
-
rhel-9.4
-
NetworkManager-1.53.4-1.el10
-
None
-
Moderate
-
ZStream
-
Customer Escalated
-
rhel-net-mgmt
-
ssg_networking
-
2
-
False
-
False
-
-
None
-
None
-
Approved Blocker
-
Definition of Done:
Please mark each item below with ( / ) if completed or ( x ) if incomplete:
The acceptance criteria defined below are met.Given that an OCP system administrator has policies targeting OVS bridges that include non-NetworkManager-managed ports,
When they update the bridge.options.mcast-snooping-enable option to true in the NNCP,
Then NetworkManager must correctly remove the existing OVS bridge from OVSDB upon deactivation and fully recreate it during reactivation, so that the updated multicast snooping setting is applied successfully without triggering verification error.
( ) Code changes are included in a downstream build attached to an errata.
( ) All required testing (manual and/or automated) passes successfully.
( ) Related documentation updates (if applicable) have been completed.
( ) All necessary backports to the related RHEL streams (linked as 'relates' in this issue) have been completed and verified.
Definition of Done: Please mark each item below with ( / ) if completed or ( x ) if incomplete: The acceptance criteria defined below are met. Given that an OCP system administrator has policies targeting OVS bridges that include non-NetworkManager-managed ports, When they update the bridge.options.mcast-snooping-enable option to true in the NNCP, Then NetworkManager must correctly remove the existing OVS bridge from OVSDB upon deactivation and fully recreate it during reactivation, so that the updated multicast snooping setting is applied successfully without triggering verification error. ( ) Code changes are included in a downstream build attached to an errata. ( ) All required testing (manual and/or automated) passes successfully. ( ) Related documentation updates (if applicable) have been completed. ( ) All necessary backports to the related RHEL streams (linked as 'relates' in this issue) have been completed and verified. -
Pass
-
Automated
-
Unspecified
-
Unspecified
-
Unspecified
-
None
Description of problem:
Customer is attempting to correct OVN's default multicast propagation behavior by setting `mcast-snooping-enable` to `true` in the NNCPs.
These bridges are being used for OpenShift CNV VM network attachments with localnet topology.
Some NNCPs complete, while others have failed or aborted NNCEs. Failure rate seen here is 3/11:
$ oc get nncp | grep -E 'NAME|vlan'
NAME STATUS REASON
vlan1004-virtualnetwork Degraded FailedToConfigure
vlan1011-virtualnetwork Available SuccessfullyConfigured
vlan1012-virtualnetwork Degraded FailedToConfigure
vlan1013-virtualnetwork Available SuccessfullyConfigured
vlan1014-virtualnetwork Available SuccessfullyConfigured
vlan1016-virtualnetwork Available SuccessfullyConfigured
vlan1017-virtualnetwork Degraded FailedToConfigure
vlan1018-virtualnetwork Available SuccessfullyConfigured
vlan1019-virtualnetwork Available SuccessfullyConfigured
vlan1048-virtualnetwork Available SuccessfullyConfigured
vlan1049-virtualnetwork Available SuccessfullyConfigured
$ oc get nnce | grep vlan1004
worker-0.sb16.caas.edc.ford.com.vlan1004-virtualnetwork Available 2025-04-09T11:13:56Z SuccessfullyConfigured
worker-1.sb16.caas.edc.ford.com.vlan1004-virtualnetwork Failing 2025-04-09T11:14:24Z FailedToConfigure
worker-2.sb16.caas.edc.ford.com.vlan1004-virtualnetwork Available 2025-04-09T11:13:57Z SuccessfullyConfigured
worker-3.sb16.caas.edc.ford.com.vlan1004-virtualnetwork Available 2025-04-09T11:13:57Z SuccessfullyConfigured
$ oc get nnce worker-1.sb16.caas.edc.ford.com.vlan1004-virtualnetwork -o yaml -o jsonpath='{.status.conditions[1].message}' error reconciling NodeNetworkConfigurationPolicy on node worker-1.sb16.caas.edc.ford.com at desired state apply: "", failed to execute nmstatectl apply --no-commit --timeout 480: 'exit status 1' '' '[2025-04-09T11:13:55Z INFO nmstatectl] Nmstate version: 2.2.40 [2025-04-09T11:13:55Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:13:55Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 6c9131a39b10258 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface patch-br-int-to-vlan.1004_ovn_localnet_port type ovs-interface [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface bc330bcd98350ee type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 0a7f0888c4c58_3 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 4ba279ef7f6379e type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface a365db5b34e6480 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 690851e8e74c918 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface c703721702621f2 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 0a7f0888c4c582f type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 8798f2ff27f31de type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface genev_sys_6081 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 824d1f8b1fb347f type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 8932935abc56168 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 7410ce1e33dc2_3 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 0b0ef6e3fb93e82 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface babbbe1147889ec type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 0dc8107c61dcf99 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface bac93bec53c75e7 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 12b27f45e4eb570 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface patch-br-int-to-br-ex_worker-1.sb16.caas.edc.ford.com type ovs-interface [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 3281e6c15c0c828 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface fac3a1fef162d0d type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 58412ba70c6a325 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface ee7a28d219e6070 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface fd66d2f6831a642 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface c7014b56bc2bdd3 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 7410ce1e33dc23f type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface ovn-k8s-mp0 type ovs-interface [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 074196e7b6d6309 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface dc038ea64a3bd15 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 0bcb63e7cd810bb type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface c703721702621_3 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 489a33f563f4683 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface cd426802f582759 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 6f05afe80a2d761 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 7ed8c8062b3c833 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface dbb90faa2456b76 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface c6aea02acc356cc type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface patch-br-int-to-vlan.1017_ovn_localnet_port type ovs-interface [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface fac3a1fef162d_3 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface c7014b56bc2bd_3 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface patch-br-ex_worker-1.sb16.caas.edc.ford.com-to-br-int type ovs-interface [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface a3889d100938f80 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 191de757e94a9fd type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface b9290e0d954fc8b type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface a365db5b34e64_3 type ethernet [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface br-int type ovs-interface [2025-04-09T11:13:55Z INFO nmstate::ifaces::inter_ifaces] Ignoring interface 6c9131a39b102_3 type ethernet [2025-04-09T11:13:55Z INFO nmstate::query_apply::net_state] Created checkpoint /org/freedesktop/NetworkManager/Checkpoint/54 [2025-04-09T11:13:56Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("4860bde5-e305-46f7-93ff-e57916bf4e2e"), ID Some("bond0.1004"), type Some(Vlan) name Some("bond0.1004") [2025-04-09T11:13:56Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("f6fe07a4-c76a-4181-a292-5a53e250c4ec"), ID Some("ovs-br-vlan-1004-br"), type Some(OvsBridge) name Some("ovs-br-vlan-1004") [2025-04-09T11:13:56Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("09766bea-b85e-4071-a37f-5503b934ea75"), ID Some("bond0.1004-port"), type Some(OvsPort) name Some("bond0.1004") [2025-04-09T11:13:56Z INFO nmstate::nm::query_apply::profile] Reapplying connection f6fe07a4-c76a-4181-a292-5a53e250c4ec: ovs-br-vlan-1004/ovs-bridge [2025-04-09T11:13:56Z INFO nmstate::nm::query_apply::profile] Reapply operation failed on ovs-bridge ovs-br-vlan-1004 f6fe07a4-c76a-4181-a292-5a53e250c4ec, reason: IncompatibleReapply:Can't reapply any changes to 'ovs-bridge' setting, retry on normal activation [2025-04-09T11:13:56Z INFO nmstate::nm::query_apply::profile] Reapplying connection 09766bea-b85e-4071-a37f-5503b934ea75: bond0.1004/ovs-port [2025-04-09T11:13:56Z INFO nmstate::nm::query_apply::profile] Reapplying connection 4860bde5-e305-46f7-93ff-e57916bf4e2e: bond0.1004/vlan [2025-04-09T11:13:56Z INFO nmstate::nm::query_apply::profile] Reapplying connection 4860bde5-e305-46f7-93ff-e57916bf4e2e: bond0.1004/vlan [2025-04-09T11:13:57Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:13:57Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:13:58Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:13:58Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:13:59Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:13:59Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:01Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:01Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:02Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:02Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:03Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:03Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:05Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:05Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:06Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:06Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:07Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:07Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:09Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:09Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:11Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("4860bde5-e305-46f7-93ff-e57916bf4e2e"), ID Some("bond0.1004"), type Some(Vlan) name Some("bond0.1004") [2025-04-09T11:14:11Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("f6fe07a4-c76a-4181-a292-5a53e250c4ec"), ID Some("ovs-br-vlan-1004-br"), type Some(OvsBridge) name Some("ovs-br-vlan-1004") [2025-04-09T11:14:11Z INFO nmstate::nm::query_apply::profile] Modifying connection UUID Some("09766bea-b85e-4071-a37f-5503b934ea75"), ID Some("bond0.1004-port"), type Some(OvsPort) name Some("bond0.1004") [2025-04-09T11:14:11Z INFO nmstate::nm::query_apply::profile] Reapplying connection f6fe07a4-c76a-4181-a292-5a53e250c4ec: ovs-br-vlan-1004/ovs-bridge [2025-04-09T11:14:11Z INFO nmstate::nm::query_apply::profile] Reapplying connection 09766bea-b85e-4071-a37f-5503b934ea75: bond0.1004/ovs-port [2025-04-09T11:14:11Z INFO nmstate::nm::query_apply::profile] Reapplying connection 4860bde5-e305-46f7-93ff-e57916bf4e2e: bond0.1004/vlan [2025-04-09T11:14:11Z INFO nmstate::nm::query_apply::profile] Reapplying connection 4860bde5-e305-46f7-93ff-e57916bf4e2e: bond0.1004/vlan [2025-04-09T11:14:12Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:12Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:13Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:13Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:14Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:14Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:16Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:16Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:17Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:17Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:18Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:18Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:20Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:20Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:21Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:21Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:22Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:22Z INFO nmstate::query_apply::net_state] Retrying on: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false' [2025-04-09T11:14:24Z INFO nmstate::nm::show] Got unsupported interface type generic: genev_sys_6081, ignoring [2025-04-09T11:14:24Z INFO nmstate::query_apply::net_state] Rollbacked to checkpoint /org/freedesktop/NetworkManager/Checkpoint/54 NmstateError: VerificationError: Verification failure: ovs-br-vlan-1004.interface.bridge.options.mcast-snooping-enable desire 'true', current 'false'
Version-Release number of selected component (if applicable):
- OpenShift 4.18.5
- Rhel 9.4
- NetworkManager 1.46.0-26.el9_4
- Nmstate 2.2.40
How reproducible:
Customer has encountered this consistently across several clusters
Steps to Reproduce:
1) Configure networks without setting `mcast-snooping-enable`
spec: desiredState: interfaces: - description: vlan using bond0 name: bond0.1004 state: up type: vlan vlan: base-iface: bond0 id: 1004 - bridge: allow-extra-patch-ports: true options: stp: false port: - name: bond0.1004 description: OVS bridge for VLAN 1004 name: ovs-br-vlan-1004 state: up type: ovs-bridge ovn: bridge-mappings: - bridge: ovs-br-vlan-1004 localnet: vlan-1004 state: present
2) Update bridge.options.mcast-snopping-enable = true
3) Watch the NNCP rollout
Actual results:
NNCEs fail without discernible pattern, causing a few NNCPs to fail.
Expected results:
NNCPs reconcile successfully
Additional info:
Customer and RH support have tried rolling back and reapplying, as well as manually reconciling on disk, without success.
Attached case 04102310 is long running.
Earliest comment dealing with this issue is the call summary in comment 108 Created By: Tim Dawson (4/8/2025 8:50 PM) Last Modified By: Tim Dawson (4/9/2025 1:31 AM)
Relevant files:
Sos: sosreport-worker-1-2025-04-09-jhwropl.tar.xz
MG: must-gather.local.3214263822206124373.zip
- links to
-
RHBA-2025:146815
NetworkManager bug fix and enhancement update