+++ This bug was initially created as a clone of Bug #2131248 +++

1. 1. 1. 1. 1. Summary:

• Issue:
• - os-net-config is interrupting mellanox connectx6 nics during runs when no change should be applied.
• Conclusion:
• - There is a bug likely introduced in the following change log entry:
• - - 2021-12-06 OSP Prod Chain <dev-null@redhat.com> 11.5.1-2.20211207004922.d9e94d8
• - - - Update patches
• - - - - Add ifcfg-* scripts on boot for Mellanox NIC interface

1. 1. 1. 1. 1. DETAILS

// os-net-config version in use vs available and changelog between

Installed: os-net-config-11.5.1-2.20211207004924
Available: os-net-config-11.5.1-2.20220404114957
Changelog:

• 2022-04-04 OSP Prod Chain <dev-null@redhat.com> 11.5.1-2.20220404114957.173ef73
• - Fix failure in dpdk driver binding with VF during reboot (rhbz#1986423)
• 2022-03-03 OSP Prod Chain <dev-null@redhat.com> 11.5.1-2.20220303155030.173ef73
• - Revert "Updating sriov_config.service to run before driverctl.slice and network-pre"
• 2022-02-04 OSP Prod Chain <dev-null@redhat.com> 11.5.1-2.20220204125254.173ef73
• - Update patches
• - For linux bond over VFs, disable spoofcheck by default
• 2022-01-22 OSP Prod Chain <dev-null@redhat.com> 11.5.1-2.20220123005237.f560c56
• - Update patches
• - Adding steering_mode config to sriov_pf object
• 2022-01-22 OSP Prod Chain <dev-null@redhat.com> 11.5.1-2.20220123014844.f560c56
• - Update patches
• 2022-01-20 OSP Prod Chain <dev-null@redhat.com> 11.5.1-2.20220120114842.d9e94d8
• - Adding steering_mode config to sriov_pf object
• 2022-01-20 OSP Prod Chain <dev-null@redhat.com> 11.5.1-2.20220120225222.0b31308
• - Update patches
• - The default max/min tx rates shall not halt deployment

// it should be noted the change log entry just before the currently in use version mentions Mellanox and adding ifcfg files

2021-12-06 OSP Prod Chain <dev-null@redhat.com> 11.5.1-2.20211207004922.d9e94d8

• Update patches
• - Add ifcfg-* scripts on boot for Mellanox NIC interface

// md5sum of each config.json confirms no changes:

89832bc3d9ae51cb6d076b8000b8d2dc 0140-sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb.tar.xz/sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb/etc/os-net-config/config.json
89832bc3d9ae51cb6d076b8000b8d2dc 0140-sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb.tar.xz/sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb/etc/os-net-config/config.json.2022-09-15T01:18:48
89832bc3d9ae51cb6d076b8000b8d2dc 0140-sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb.tar.xz/sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb/etc/os-net-config/config.json.2022-09-15T01:43:30
89832bc3d9ae51cb6d076b8000b8d2dc 0140-sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb.tar.xz/sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb/etc/os-net-config/config.json.2022-09-15T02:05:35
89832bc3d9ae51cb6d076b8000b8d2dc 0140-sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb.tar.xz/sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb/etc/os-net-config/config.json.2022-09-15T02:26:27
89832bc3d9ae51cb6d076b8000b8d2dc 0140-sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb.tar.xz/sosreport-cmp34f2hpa1r02u16-com34-2022-09-15-lafpsyb/etc/os-net-config/config.json.2022-09-15T02:47:10

// Yet confirmation each bond is impacted when timestamp adjusted for IST

2022-09-14T19:48:50.558Z|05331|bond|INFO|bond dpdkbond0: all members disabled
2022-09-14T19:48:50.811Z|05334|bond|INFO|bond dpdkbond1: all members disabled
2022-09-14T19:48:51.360Z|05340|bond|INFO|bond dpdkbond2: all members disabled
2022-09-14T20:13:32.402Z|05462|bond|INFO|bond dpdkbond0: all members disabled
2022-09-14T20:13:32.668Z|05465|bond|INFO|bond dpdkbond1: all members disabled
2022-09-14T20:13:33.230Z|05470|bond|INFO|bond dpdkbond2: all members disabled
2022-09-14T20:35:36.758Z|05521|bond|INFO|bond dpdkbond0: all members disabled
2022-09-14T20:35:37.362Z|05527|bond|INFO|bond dpdkbond1: all members disabled
2022-09-14T20:35:37.921Z|05533|bond|INFO|bond dpdkbond2: all members disabled
2022-09-14T20:56:28.759Z|05567|bond|INFO|bond dpdkbond0: all members disabled
2022-09-14T20:56:29.323Z|05594|bond|INFO|bond dpdkbond1: all members disabled
2022-09-14T20:56:29.893Z|05600|bond|INFO|bond dpdkbond2: all members disabled
2022-09-14T21:17:12.394Z|05660|bond|INFO|bond dpdkbond0: all members disabled
2022-09-14T21:17:12.902Z|05666|bond|INFO|bond dpdkbond1: all members disabled
2022-09-14T21:17:13.473Z|05671|bond|INFO|bond dpdkbond2: all members disabled

// Example os-net-config tearing down mellanox nics

2022-09-15 02:26:51,813 p=928519 u=mistral n=ansible | 2022-09-15 02:26:51.812919 | 52540090-9f6e-102b-80c7-000000000078 | OK | NetworkConfig stdout | cmp34f2hpa1r02u16 | result={
"NetworkConfig_result.stderr_lines": [
"+ '[' -n '{\"network_config\": [{\"addresses\": [

{\"ip_netmask\": \"172.18.65.117/24\"}

], \"bonding_options\": \"mode=802.3ad miimon=100 lacp_rate=fast\", \"members\": [

{\"mtu\": 9000, \"name\": \"eno5\", \"primary\": true, \"type\": \"interface\"}

,

{\"mtu\": 9000, \"name\": \"eno6\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"bond0\", \"routes\": [

{\"default\": true, \"next_hop\": \"172.18.65.1\"}

], \"type\": \"linux_bond\", \"use_dhcp\": false}, {\"addresses\": [

{\"ip_netmask\": \"172.16.130.219/24\"}

], \"device\": \"bond0\", \"mtu\": 9000, \"routes\": [], \"type\": \"vlan\", \"vlan_id\": 504}, {\"addresses\": [

{\"ip_netmask\": \"172.16.18.52/24\"}

], \"device\": \"bond0\", \"mtu\": 9000, \"routes\": [], \"type\": \"vlan\", \"vlan_id\": 503}, {\"addresses\": [

{\"ip_netmask\": \"192.168.204.106/24\"}

], \"device\": \"bond0\", \"mtu\": 9000, \"routes\": [], \"type\": \"vlan\", \"vlan_id\": 508}, {\"addresses\": [

{\"ip_netmask\": \"192.168.203.122/24\"}

], \"device\": \"bond0\", \"mtu\": 9000, \"routes\": [], \"type\": \"vlan\", \"vlan_id\": 507}, {\"members\": [{\"members\": [{\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens1f0\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk0\", \"type\": \"ovs_dpdk_port\"}, {\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens1f1\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk1\", \"type\": \"ovs_dpdk_port\"}], \"mtu\": 9000, \"name\": \"dpdkbond0\", \"ovs_options\": \"bond_mode=balance-slb lacp=active\", \"rx_queue\": 8, \"type\": \"ovs_dpdk_bond\"}], \"mtu\": 9000, \"name\": \"br-link0\", \"type\": \"ovs_user_bridge\", \"use_dhcp\": false}, {\"members\": [{\"members\": [{\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens3f0\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk2\", \"type\": \"ovs_dpdk_port\"}, {\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens3f1\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk3\", \"type\": \"ovs_dpdk_port\"}], \"mtu\": 9000, \"name\": \"dpdkbond1\", \"ovs_options\": \"bond_mode=balance-slb lacp=active\", \"rx_queue\": 8, \"type\": \"ovs_dpdk_bond\"}], \"mtu\": 9000, \"name\": \"br-link1\", \"type\": \"ovs_user_bridge\", \"use_dhcp\": false}, {\"members\": [{\"members\": [{\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens2f0\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk4\", \"type\": \"ovs_dpdk_port\"}, {\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens2f1\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk5\", \"type\": \"ovs_dpdk_port\"}], \"mtu\": 9000, \"name\": \"dpdkbond2\", \"ovs_options\": \"bond_mode=balance-slb lacp=active\", \"rx_queue\": 8, \"type\": \"ovs_dpdk_bond\"}], \"mtu\": 9000, \"name\": \"br-link2\", \"type\": \"ovs_user_bridge\", \"use_dhcp\": false}]}' ']'",
"+ '[' -z '' ']'",
"+ trap configure_safe_defaults EXIT",
"++ date +%Y-%m-%dT%H:%M:%S",
"+ DATETIME=2022-09-15T02:26:27",
"+ '[' -f /etc/os-net-config/config.json ']'",
"+ mv /etc/os-net-config/config.json /etc/os-net-config/config.json.2022-09-15T02:26:27",
"+ mkdir -p /etc/os-net-config",
"+ echo '{\"network_config\": [{\"addresses\": [

{\"ip_netmask\": \"172.18.65.117/24\"}

], \"bonding_options\": \"mode=802.3ad miimon=100 lacp_rate=fast\", \"members\": [

{\"mtu\": 9000, \"name\": \"eno5\", \"primary\": true, \"type\": \"interface\"}

,

{\"mtu\": 9000, \"name\": \"eno6\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"bond0\", \"routes\": [

{\"default\": true, \"next_hop\": \"172.18.65.1\"}

], \"type\": \"linux_bond\", \"use_dhcp\": false}, {\"addresses\": [

{\"ip_netmask\": \"172.16.130.219/24\"}

], \"device\": \"bond0\", \"mtu\": 9000, \"routes\": [], \"type\": \"vlan\", \"vlan_id\": 504}, {\"addresses\": [

{\"ip_netmask\": \"172.16.18.52/24\"}

], \"device\": \"bond0\", \"mtu\": 9000, \"routes\": [], \"type\": \"vlan\", \"vlan_id\": 503}, {\"addresses\": [

{\"ip_netmask\": \"192.168.204.106/24\"}

], \"device\": \"bond0\", \"mtu\": 9000, \"routes\": [], \"type\": \"vlan\", \"vlan_id\": 508}, {\"addresses\": [

{\"ip_netmask\": \"192.168.203.122/24\"}

], \"device\": \"bond0\", \"mtu\": 9000, \"routes\": [], \"type\": \"vlan\", \"vlan_id\": 507}, {\"members\": [{\"members\": [{\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens1f0\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk0\", \"type\": \"ovs_dpdk_port\"}, {\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens1f1\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk1\", \"type\": \"ovs_dpdk_port\"}], \"mtu\": 9000, \"name\": \"dpdkbond0\", \"ovs_options\": \"bond_mode=balance-slb lacp=active\", \"rx_queue\": 8, \"type\": \"ovs_dpdk_bond\"}], \"mtu\": 9000, \"name\": \"br-link0\", \"type\": \"ovs_user_bridge\", \"use_dhcp\": false}, {\"members\": [{\"members\": [{\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens3f0\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk2\", \"type\": \"ovs_dpdk_port\"}, {\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens3f1\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk3\", \"type\": \"ovs_dpdk_port\"}], \"mtu\": 9000, \"name\": \"dpdkbond1\", \"ovs_options\": \"bond_mode=balance-slb lacp=active\", \"rx_queue\": 8, \"type\": \"ovs_dpdk_bond\"}], \"mtu\": 9000, \"name\": \"br-link1\", \"type\": \"ovs_user_bridge\", \"use_dhcp\": false}, {\"members\": [{\"members\": [{\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens2f0\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk4\", \"type\": \"ovs_dpdk_port\"}, {\"driver\": \"mlx5_core\", \"members\": [

{\"name\": \"ens2f1\", \"type\": \"interface\"}

], \"mtu\": 9000, \"name\": \"dpdk5\", \"type\": \"ovs_dpdk_port\"}], \"mtu\": 9000, \"name\": \"dpdkbond2\", \"ovs_options\": \"bond_mode=balance-slb lacp=active\", \"rx_queue\": 8, \"type\": \"ovs_dpdk_bond\"}], \"mtu\": 9000, \"name\": \"br-link2\", \"type\": \"ovs_user_bridge\", \"use_dhcp\": false}]}'",
"++ type -t network_config_hook",
"+ '[' '' = function ']'",
"+ sed -i 's/: \"bridge_name/: \"br-ex/g' /etc/os-net-config/config.json",
"+ sed -i s/interface_name/nic1/g /etc/os-net-config/config.json",
"+ set +e",
"+ os-net-config -c /etc/os-net-config/config.json -v --detailed-exit-codes",
"[2022/09/15 02:26:27 AM] [INFO] Using config file at: /etc/os-net-config/config.json",
"[2022/09/15 02:26:27 AM] [INFO] Ifcfg net config provider created.",
"[2022/09/15 02:26:27 AM] [INFO] Not using any mapping file.",
"[2022/09/15 02:26:28 AM] [INFO] Finding active nics",
"[2022/09/15 02:26:28 AM] [INFO] ens1f0 is an active nic",
"[2022/09/15 02:26:28 AM] [INFO] vlan504 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] bonding_masters is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] lo is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] eno3 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] br-ovs is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] ens2f1 is an active nic",
"[2022/09/15 02:26:28 AM] [INFO] eno1 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] br-int is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] br-link1 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] ovs-netdev is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] vlan507 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] ens3f0 is an active nic",
"[2022/09/15 02:26:28 AM] [INFO] eno6 is an embedded active nic",
"[2022/09/15 02:26:28 AM] [INFO] ens1f1 is an active nic",
"[2022/09/15 02:26:28 AM] [INFO] eno4 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] vlan503 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] eno2 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] ens2f0 is an active nic",
"[2022/09/15 02:26:28 AM] [INFO] br-link2 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] br-tun is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] br-link0 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] vlan508 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] ens3f1 is an active nic",
"[2022/09/15 02:26:28 AM] [INFO] bond0 is not an active nic",
"[2022/09/15 02:26:28 AM] [INFO] eno5 is an embedded active nic",
"[2022/09/15 02:26:28 AM] [INFO] No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml)",
"[2022/09/15 02:26:28 AM] [INFO] Active nics are ['eno5', 'eno6', 'ens1f0', 'ens1f1', 'ens2f0', 'ens2f1', 'ens3f0', 'ens3f1']",
"[2022/09/15 02:26:28 AM] [INFO] nic3 mapped to: ens1f0",
"[2022/09/15 02:26:28 AM] [INFO] nic5 mapped to: ens2f0",
"[2022/09/15 02:26:28 AM] [INFO] nic8 mapped to: ens3f1",
"[2022/09/15 02:26:28 AM] [INFO] nic2 mapped to: eno6",
"[2022/09/15 02:26:28 AM] [INFO] nic7 mapped to: ens3f0",
"[2022/09/15 02:26:28 AM] [INFO] nic4 mapped to: ens1f1",
"[2022/09/15 02:26:28 AM] [INFO] nic1 mapped to: eno5",
"[2022/09/15 02:26:28 AM] [INFO] nic6 mapped to: ens2f1",
"[2022/09/15 02:26:28 AM] [INFO] adding linux bond: bond0",
"[2022/09/15 02:26:28 AM] [INFO] adding custom route for interface: bond0",
"[2022/09/15 02:26:28 AM] [INFO] adding interface: eno5",
"[2022/09/15 02:26:28 AM] [INFO] adding interface: eno6",
"[2022/09/15 02:26:28 AM] [INFO] adding vlan: vlan504",
"[2022/09/15 02:26:28 AM] [INFO] adding vlan: vlan503",
"[2022/09/15 02:26:28 AM] [INFO] adding vlan: vlan508",
"[2022/09/15 02:26:28 AM] [INFO] adding vlan: vlan507",
"[2022/09/15 02:26:28 AM] [INFO] adding ovs user bridge: br-link0",
"[2022/09/15 02:26:28 AM] [INFO] adding ovs dpdk bond: dpdkbond0",
"[2022/09/15 02:26:28 AM] [INFO] Driver (mlx5_core) is already bound to the device (ens1f0)",
"[2022/09/15 02:26:28 AM] [INFO] Driver (mlx5_core) is already bound to the device (ens1f1)",
"[2022/09/15 02:26:28 AM] [INFO] Getting devargs for Mellanox cards",
"[2022/09/15 02:26:28 AM] [INFO] Getting devargs for Mellanox cards",
"[2022/09/15 02:26:28 AM] [INFO] adding interface: ens1f0",
"[2022/09/15 02:26:28 AM] [INFO] adding interface: ens1f1",
"[2022/09/15 02:26:28 AM] [INFO] adding ovs user bridge: br-link1",
"[2022/09/15 02:26:28 AM] [INFO] adding ovs dpdk bond: dpdkbond1",
"[2022/09/15 02:26:28 AM] [INFO] Driver (mlx5_core) is already bound to the device (ens3f0)",
"[2022/09/15 02:26:28 AM] [INFO] Driver (mlx5_core) is already bound to the device (ens3f1)",
"[2022/09/15 02:26:28 AM] [INFO] Getting devargs for Mellanox cards",
"[2022/09/15 02:26:28 AM] [INFO] Getting devargs for Mellanox cards",
"[2022/09/15 02:26:28 AM] [INFO] adding interface: ens3f0",
"[2022/09/15 02:26:28 AM] [INFO] adding interface: ens3f1",
"[2022/09/15 02:26:28 AM] [INFO] adding ovs user bridge: br-link2",
"[2022/09/15 02:26:28 AM] [INFO] adding ovs dpdk bond: dpdkbond2",
"[2022/09/15 02:26:28 AM] [INFO] Driver (mlx5_core) is already bound to the device (ens2f0)",
"[2022/09/15 02:26:28 AM] [INFO] Driver (mlx5_core) is already bound to the device (ens2f1)",
"[2022/09/15 02:26:28 AM] [INFO] Getting devargs for Mellanox cards",
"[2022/09/15 02:26:28 AM] [INFO] Getting devargs for Mellanox cards",
"[2022/09/15 02:26:28 AM] [INFO] Getting devargs for Mellanox cards",
"[2022/09/15 02:26:28 AM] [INFO] adding interface: ens2f0",
"[2022/09/15 02:26:28 AM] [INFO] adding interface: ens2f1",
"[2022/09/15 02:26:28 AM] [INFO] applying network configs...",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for interface: eno5",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for interface: eno6",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for interface: dpdkbond0",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for interface: dpdkbond1",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for interface: dpdkbond2",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for bridge: br-link0",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for bridge: br-link1",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for bridge: br-link2",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for linux bond: bond0",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for vlan interface: vlan504",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for vlan interface: vlan503",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for vlan interface: vlan508",
"[2022/09/15 02:26:28 AM] [INFO] No changes required for vlan interface: vlan507",
"[2022/09/15 02:26:28 AM] [INFO] running ifdown on interface: ens1f0",
"[2022/09/15 02:26:28 AM] [INFO] interface ens1f0 is up, trying with ip command",
"[2022/09/15 02:26:28 AM] [INFO] running ifdown on interface: ens1f1",
"[2022/09/15 02:26:28 AM] [INFO] interface ens1f1 is up, trying with ip command",
"[2022/09/15 02:26:28 AM] [INFO] running ifdown on interface: ens3f0",
"[2022/09/15 02:26:28 AM] [INFO] interface ens3f0 is up, trying with ip command",
"[2022/09/15 02:26:29 AM] [INFO] running ifdown on interface: ens3f1",
"[2022/09/15 02:26:29 AM] [INFO] interface ens3f1 is up, trying with ip command",
"[2022/09/15 02:26:29 AM] [INFO] running ifdown on interface: ens2f0",
"[2022/09/15 02:26:29 AM] [INFO] interface ens2f0 is up, trying with ip command",
"[2022/09/15 02:26:29 AM] [INFO] running ifdown on interface: ens2f1",
"[2022/09/15 02:26:29 AM] [INFO] interface ens2f1 is up, trying with ip command",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens1f0",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens1f1",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens3f0",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens3f1",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f0",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f1",
"[2022/09/15 02:26:29 AM] [INFO] running ifup on interface: ens1f0",
"[2022/09/15 02:26:32 AM] [INFO] running ifup on interface: ens1f1",
"[2022/09/15 02:26:35 AM] [INFO] running ifup on interface: ens3f0",
"[2022/09/15 02:26:38 AM] [INFO] running ifup on interface: ens3f1",
"[2022/09/15 02:26:41 AM] [INFO] running ifup on interface: ens2f0",
"[2022/09/15 02:26:44 AM] [INFO] running ifup on interface: ens2f1",
"+ RETVAL=2",
"+ set -e",
"+ [[ 2 == 2 ]]",
"+ '[' -f /etc/udev/rules.d/99-dhcp-all-interfaces.rules ']'",
"+ '[' -f /usr/libexec/os-apply-config/templates/etc/os-net-config/config.json ']'",
"+ '[' -f /usr/libexec/os-apply-config/templates/etc/os-net-config/element_config.json ']'",
"+ configure_safe_defaults",
"+ [[ 0 == 0 ]]",
"+ return 0"
],
"changed": false,
"failed_when_result": false

— Additional comment from coldford@redhat.com on 2022-09-30 12:54:52 UTC —

I'll attach a sosreport from one of the impacted computes.

— Additional comment from coldford@redhat.com on 2022-09-30 13:00:36 UTC —

— Additional comment from coldford@redhat.com on 2022-09-30 13:08:55 UTC —

The sosreport was too large to attach. They can be found in the link sfdc case.

— Additional comment from coldford@redhat.com on 2022-09-30 15:00:52 UTC —

// Driver and firmware info
// ens1f1
driver: mlx5_core
version: 5.0-0
firmware-version: 16.23.8054 (HPE0000000039)
expansion-rom-version:
bus-info: 0000:12:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

// ens1f1
driver: mlx5_core
version: 5.0-0
firmware-version: 16.23.8054 (HPE0000000039)
expansion-rom-version:
bus-info: 0000:12:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

// ens2f0
driver: mlx5_core
version: 5.0-0
firmware-version: 16.23.8054 (HPE0000000039)
expansion-rom-version:
bus-info: 0000:37:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

// ens2f1
driver: mlx5_core
version: 5.0-0
firmware-version: 16.23.8054 (HPE0000000039)
expansion-rom-version:
bus-info: 0000:37:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

// ens3f0
driver: mlx5_core
version: 5.0-0
firmware-version: 16.23.8054 (HPE0000000039)
expansion-rom-version:
bus-info: 0000:d8:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

// ens3f1
driver: mlx5_core
version: 5.0-0
firmware-version: 16.23.8054 (HPE0000000039)
expansion-rom-version:
bus-info: 0000:d8:00.1
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

— Additional comment from coldford@redhat.com on 2022-09-30 15:55:33 UTC —

Note the interfaces carry the dpdk bonds

Bridge br-link2
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: netdev
Port phy-br-link2
Interface phy-br-link2
type: patch
options:

{peer=int-br-link2}

Port dpdkbond2
Interface dpdk4
type: dpdk
options:

{dpdk-devargs="0000:37:00.0", n_rxq="8"}

Interface dpdk5
type: dpdk
options:

{dpdk-devargs="0000:37:00.1", n_rxq="8"}

Port br-link2
Interface br-link2
type: internal
Bridge br-link1
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: netdev
Port dpdkbond1
Interface dpdk3
type: dpdk
options:

{dpdk-devargs="0000:d8:00.1", n_rxq="8"}

Interface dpdk2
type: dpdk
options:

{dpdk-devargs="0000:d8:00.0", n_rxq="8"}

Port phy-br-link1
Interface phy-br-link1
type: patch
options:

{peer=int-br-link1}

Port br-link1
Interface br-link1
type: internal
Bridge br-link0
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: netdev
Port br-link0
Interface br-link0
type: internal
Port dpdkbond0
Interface dpdk1
type: dpdk
options:

{dpdk-devargs="0000:12:00.1", n_rxq="8"}

Interface dpdk0
type: dpdk
options:

{dpdk-devargs="0000:12:00.0", n_rxq="8"}

Port phy-br-link0
Interface phy-br-link0
type: patch
options:

{peer=int-br-link0}

— Additional comment from coldford@redhat.com on 2022-09-30 16:59:27 UTC —

12:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
12:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
37:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
37:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
d8:00.0 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]
d8:00.1 Ethernet controller [0200]: Mellanox Technologies MT27800 Family [ConnectX-5] [15b3:1017]

— Additional comment from Rohini Diwakar on 2022-10-03 09:48:14 UTC —

Hi Team,

Everytime cu re-runs the deploy command for making some modifications in stack, the cu faces huge downtime on the VNFs running on ovs-dpdk nodes even though the modifications in templates are unrelated to ovs-dpdk nodes.
We observed that this is due to os-net-config interrupting nics during runs when no change should be applied.

This has been observed in 2 different sites on RHOSP16.2.2

We have advised them to use role specific NetworkDeploymentActions: ['CREATE','UPDATE'] and always keep it NetworkDeploymentActions: ['CREATE'] during the deploy as per the KCS https://access.redhat.com/solutions/2213711

The cu is using same configuration in RHOSP13 with NetworkDeploymentActions: ['CREATE','UPDATE'] in templates, but the same behaviour/downtime never happens during any stack related activity. There as well they are using Mellanox connectx-5 nic cards.

It's urgent for us to figure out the reason for this, as the cu have multiple sites already in production where they have to modify templates and re-run the delpoy.

Thanks,
Rohini Diwakar

— Additional comment from Haresh Khandelwal on 2022-10-03 12:26:24 UTC —

Hi,

Can you please share the templates used here in the stack update?

Also, If customer is using role specific Networkdeployment for non-dpdk nodes, it should not impact dpdk nodes. There may be something else happening to dpdk nodes which is impacting the traffic.
os-net-config looks to fine me on the attached ansible.log file provided in comment#2.
Though adding @dsneddon@redhat.com @ksundara@redhat.com to confirm if os-net-config is culprit here or not.

Thanks

— Additional comment from Rohini Diwakar on 2022-10-03 13:15:09 UTC —

Hi Harsh,

The templates and the sosreports of the affected node are on supportshell.

drwxrwxrwx. 3 yank yank 26 Sep 15 05:02 0060-template.tar.gz
drwxrwxrwx. 3 yank yank 25 Sep 28 00:20 0240-mistral_overcloud_20220927_1348.tar.gz
drwxrwxrwx. 3 yank yank 68 Sep 28 02:18 0250-sosreport-cmp04f2hpa1r04u08-2022-09-27-phzlfeh.tar.xz
drwxrwxrwx. 3 yank yank 68 Sep 28 02:21 0260-sosreport-cmp33f2hpa1r01u16-2022-09-27-uvwonnh.tar.xz. <---- this is where vnf related vm was on
drwxrwxrwx. 3 yank yank 68 Sep 28 03:05 0270-sosreport-cnt01f2hpa1r01u25-2022-09-27-kvwmgok.tar.xz
drwxrwxrwx. 3 yank yank 68 Sep 28 03:07 0280-sosreport-cnt02f2hpa1r02u25-2022-09-27-ipqqljo.tar.xz
drwxrwxrwx. 3 yank yank 68 Sep 28 03:35 0320-sosreport-cmp30f2hpa1r02u15-2022-09-27-ybbzmaq.tar.xz
drwxrwxrwx. 3 yank yank 59 Sep 28 04:08 0330-sosreport-director-2022-09-27-rewssvd.tar.xz
drwxrwxrwx. 3 yank yank 68 Sep 28 04:47 0340-sosreport-cnt03f2hpa1r03u25-2022-09-27-cpgeqru.tar.xz

So the latest impact was seen on 23rd Sep.

This is what I found in var/log/messages-20220925 of compute node [0260-sosreport-cmp33f2hpa1r01u16-2022-09-27-uvwonnh.tar.xz]

Sep 23 02:49:52 cmp33f2hpa1r01u16 ansible-file[744602]: Invoked with mode=493 follow=False dest=/var/lib/tripleo-config/scripts/run_os_net_config.sh _original_basename=NetworkConfig recurse=False state=file path=/var/lib/tripleo-config/scripts/run_os_net_config.sh force=False modification_time_format=%Y%m%d%H%M.%S access_time_format=%Y%m%d%H%M.%S unsafe_writes=False _diff_peek=None src=None modification_time=None access_time
=None owner=None group=None seuser=None serole=None selevel=None setype=None attributes=None content=NOT_LOGGING_PARAMETER backup=None remote_src=None regexp=None delimiter=None directory_mode=None
.
.
Sep 23 02:49:54 cmp33f2hpa1r01u16 ifdown[744795]: You are using 'ifdown' script provided by 'network-scripts', which are now deprecated.
Sep 23 02:49:54 cmp33f2hpa1r01u16 ifdown[744796]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Sep 23 02:49:54 cmp33f2hpa1r01u16 ifdown[744797]: It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744818]: You are using 'ifdown' script provided by 'network-scripts', which are now deprecated.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744819]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744820]: It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744868]: You are using 'ifdown' script provided by 'network-scripts', which are now deprecated.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744871]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744873]: It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744898]: You are using 'ifdown' script provided by 'network-scripts', which are now deprecated.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744899]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744900]: It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744920]: You are using 'ifdown' script provided by 'network-scripts', which are now deprecated.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744921]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Sep 23 02:49:55 cmp33f2hpa1r01u16 ifdown[744922]: It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
Sep 23 02:49:56 cmp33f2hpa1r01u16 ifdown[744942]: You are using 'ifdown' script provided by 'network-scripts', which are now deprecated.
Sep 23 02:49:56 cmp33f2hpa1r01u16 ifdown[744943]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Sep 23 02:49:56 cmp33f2hpa1r01u16 ifdown[744944]: It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
Sep 23 02:49:56 cmp33f2hpa1r01u16 ifup[744995]: You are using 'ifup' script provided by 'network-scripts', which are now deprecated.
Sep 23 02:49:56 cmp33f2hpa1r01u16 ifup[744999]: 'network-scripts' will be removed in one of the next major releases of RHEL.
Sep 23 02:49:56 cmp33f2hpa1r01u16 ifup[745002]: It is advised to switch to 'NetworkManager' instead - it provides 'ifup/ifdown' scripts as well.
Sep 23 02:49:56 cmp33f2hpa1r01u16 NetworkManager[2146]: <info> [1663881596.4023] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifcfg-ens1f0" pid=745007 uid=0 result="success"
Sep 23 02:49:56 cmp33f2hpa1r01u16 NetworkManager[2146]: <info> [1663881596.4285] audit: op="connections-load" args="/etc/sysconfig/network-scripts/ifcfg-ens1f0" pid=745018 uid=0 result="success"
Sep 23 02:49:57 cmp33f2hpa1r01u16 kernel: mlx5_core 0000:12:00.0 ens1f0: Link up
Sep 23 02:49:57 cmp33f2hpa1r01u16 kernel: IPv6: ADDRCONF(NETDEV_UP): ens1f0: link is not ready
Sep 23 02:49:57 cmp33f2hpa1r01u16 NetworkManager[2146]: <info> [1663881597.1635] device (ens1f0): carrier: link connected
Sep 23 02:49:57 cmp33f2hpa1r01u16 kernel: 8021q: adding VLAN 0 to HW filter on device ens1f0
Sep 23 02:49:57 cmp33f2hpa1r01u16 kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ens1f0: link becomes ready

All the nics on dpdkbonds flapped during the time of stack re-run, their ifcfg files were also regenerated.

Regards,
Rohini Diwakar

— Additional comment from coldford@redhat.com on 2022-10-03 13:17:18 UTC —

— Additional comment from Rohini Diwakar on 2022-10-05 04:23:06 UTC —

Hello,

Can we please have an update on this?

— Additional comment from Shailesh Chhabdiya on 2022-10-06 11:46:18 UTC —

Hello Team,

This BZ is for ESS customer and they are requesting for an update

Do we have anything to share or is there anything needed from Customer ?

The behavior is not same in RHOSP13 because over there the metadata is checked first until there is no change in existing and newly provided metadata UPDATE doesn't reflects whereas in RHOSP16 though Metadata remains same `UPDATE` seems to be regenerating the network files on Overcloud Nodes

Thank You

— Additional comment from Vijayalakshmi Candappa on 2022-10-06 13:26:52 UTC —

From the original BZ description, noted the following:

"--
2021-12-06 OSP Prod Chain <dev-null@redhat.com> 11.5.1-2.20211207004922.d9e94d8

• Update patches
  ' - Add ifcfg-* scripts on boot for Mellanox NIC interface
  --"

This patch is expected to re-write the ifcfg* scripts for Mellonix NICs with DPDK enabled
(It was provided as fix for https://bugzilla.redhat.com/show_bug.cgi?id=1998662)

Can you please provide the o/p of "cat /etc/sysconfig/network-scripts/ifcfg-*" eg. ens1f0/ens1f1/ens2f0/ens2f0 from any of the affected compute nodes before and after stack update ?

You can see that there will be two additional configs after this patch in the ifcfg* files:

NM_CONTROLLED=no
BOOTPROTO=none

This patch should be the reason for the dpdkbonds being flapped

— Additional comment from coldford@redhat.com on 2022-10-06 17:23:28 UTC —

@vcandapp@redhat.com

The files are generated every time without an actual change being applied.

Every stack run we see the same behaviour.

"[2022/09/15 02:26:28 AM] [INFO] running ifdown on interface: ens1f0",
"[2022/09/15 02:26:28 AM] [INFO] interface ens1f0 is up, trying with ip command",
"[2022/09/15 02:26:28 AM] [INFO] running ifdown on interface: ens1f1",
"[2022/09/15 02:26:28 AM] [INFO] interface ens1f1 is up, trying with ip command",
"[2022/09/15 02:26:28 AM] [INFO] running ifdown on interface: ens3f0",
"[2022/09/15 02:26:28 AM] [INFO] interface ens3f0 is up, trying with ip command",
"[2022/09/15 02:26:29 AM] [INFO] running ifdown on interface: ens3f1",
"[2022/09/15 02:26:29 AM] [INFO] interface ens3f1 is up, trying with ip command",
"[2022/09/15 02:26:29 AM] [INFO] running ifdown on interface: ens2f0",
"[2022/09/15 02:26:29 AM] [INFO] interface ens2f0 is up, trying with ip command",
"[2022/09/15 02:26:29 AM] [INFO] running ifdown on interface: ens2f1",
"[2022/09/15 02:26:29 AM] [INFO] interface ens2f1 is up, trying with ip command",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens1f0",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens1f1",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens3f0",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens3f1",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f0",
"[2022/09/15 02:26:29 AM] [INFO] Writing config /etc/sysconfig/network-scripts/ifcfg-ens2f1",
"[2022/09/15 02:26:29 AM] [INFO] running ifup on interface: ens1f0",
"[2022/09/15 02:26:32 AM] [INFO] running ifup on interface: ens1f1",
"[2022/09/15 02:26:35 AM] [INFO] running ifup on interface: ens3f0",
"[2022/09/15 02:26:38 AM] [INFO] running ifup on interface: ens3f1",
"[2022/09/15 02:26:41 AM] [INFO] running ifup on interface: ens2f0",
"[2022/09/15 02:26:44 AM] [INFO] running ifup on interface: ens2f1",

— Additional comment from coldford@redhat.com on 2022-10-07 12:33:35 UTC —

@vcandapp@redhat.com

Looking again at:

2022-09-14T19:48:50.558Z|05331|bond|INFO|bond dpdkbond0: all members disabled
2022-09-14T19:48:50.811Z|05334|bond|INFO|bond dpdkbond1: all members disabled
2022-09-14T19:48:51.360Z|05340|bond|INFO|bond dpdkbond2: all members disabled

2022-09-14T20:13:32.402Z|05462|bond|INFO|bond dpdkbond0: all members disabled
2022-09-14T20:13:32.668Z|05465|bond|INFO|bond dpdkbond1: all members disabled
2022-09-14T20:13:33.230Z|05470|bond|INFO|bond dpdkbond2: all members disabled

2022-09-14T20:35:36.758Z|05521|bond|INFO|bond dpdkbond0: all members disabled
2022-09-14T20:35:37.362Z|05527|bond|INFO|bond dpdkbond1: all members disabled
2022-09-14T20:35:37.921Z|05533|bond|INFO|bond dpdkbond2: all members disabled

2022-09-14T20:56:28.759Z|05567|bond|INFO|bond dpdkbond0: all members disabled
2022-09-14T20:56:29.323Z|05594|bond|INFO|bond dpdkbond1: all members disabled
2022-09-14T20:56:29.893Z|05600|bond|INFO|bond dpdkbond2: all members disabled

2022-09-14T21:17:12.394Z|05660|bond|INFO|bond dpdkbond0: all members disabled
2022-09-14T21:17:12.902Z|05666|bond|INFO|bond dpdkbond1: all members disabled
2022-09-14T21:17:13.473Z|05671|bond|INFO|bond dpdkbond2: all members disabled

It's clear every stack update attempted results in a ifdown and ifup of the nics under the bonds.

— Additional comment from Rohini Diwakar on 2022-10-10 14:59:25 UTC —

Hi,

Can we please have an update on this?

This is critical for Airtel and it's for an ESS account, there are other sites pending for stack update and we don't know why this happens.

Post stack update on 15th sep:
==============================
[sosreport-cmp33f2hpa1r01u16-com33-2022-09-15-nqxevku ]$ cat etc/sysconfig/network-scripts/ifcfg-ens1f0

1. This file is autogenerated by os-net-config
   DEVICE=ens1f0
   ONBOOT=yes
   HOTPLUG=no
   NM_CONTROLLED=no
   PEERDNS=no
   BOOTPROTO=none
   ==============================

Post stack update on 23rd sep:
==============================
[sosreport-cmp33f2hpa1r01u16-2022-09-27-uvwonnh ]$ cat etc/sysconfig/network-scripts/ifcfg-ens1f0

1. This file is autogenerated by os-net-config
   DEVICE=ens1f0
   ONBOOT=yes
   HOTPLUG=no
   NM_CONTROLLED=no
   PEERDNS=no
   BOOTPROTO=none
   ==============================

There is no difference in the ifcfg files.

— Additional comment from Vijayalakshmi Candappa on 2022-10-11 08:46:41 UTC —

I was able to reproduce this issue and am working on a patch for the same. Will be posting the patch in master and relevant up-stream branches

— Additional comment from RHEL Program Management on 2022-10-11 11:22:16 UTC —

This bugzilla has been removed from the release since it does not have an acked release flag. For details, see https://mojo.redhat.com/docs/DOC-1144661#jive_content_id_OSP_Release_Planning.'

— Additional comment from RHEL Program Management on 2022-10-11 11:22:16 UTC —

This item has been properly Triaged and planned for the release, and Target Release is now set to match the release flag. For details, see https://mojo.redhat.com/docs/DOC-1195410

— Additional comment from Jon Schlueter on 2022-11-01 14:01:45 UTC —

upstream patch is still NEW and marked as WIP moving back to assigned till that is merged upstream

— Additional comment from Vijayalakshmi Candappa on 2022-11-03 06:23:38 UTC —

@jschluet@redhat.com , this patch 863479 was blocked due to a dependency on another two patches (https://review.opendev.org/c/openstack/os-net-config/+/862135 and https://review.opendev.org/c/openstack/os-net-config/+/863194)

I am expecting the above issue to be fixed and merged in master in couple of days - then the patch for this BZ will also be merged

— Additional comment from Vijayalakshmi Candappa on 2022-11-14 04:39:47 UTC —

@jschluet@redhat.com , the patches are merged upstream. Can you please confirm in which downstream repo you will be requiring this ? 16.1.- or 16.2.- ?

— Additional comment from Vijayalakshmi Candappa on 2022-11-17 05:12:00 UTC —

@coldford@redhat.com , this patches is already merged in rhos/rhos-16.2-trunk-patches (will wait for the next compose of 16.2.4)
Ported patch to 16.1.9 downstream repo

— Additional comment from Jon Schlueter on 2022-11-18 12:17:58 UTC —

At this point unless it is a blocker it would have to slip to 16.2.5 .