Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Critical
Fix Version/s: rhel-9.6
Affects Version/s: rhel-9.2.0.z
Component/s: nmstate
Labels:
- Triaged
- upstream_merged

Fixed in Build:
nmstate-2.2.39-1.el9
Regression:
None
Severity:
Important
Keywords:

ZStream

Pool Team:

rhel-sst-network-management
Sub-System Group:

ssg_networking

Story Points:
None

Status Summary:

Hide

Customer/Partner	Jira ID	Customer Case	Status	Details
Nokia	RHEL-64707	03834014		The customer application pod interface is being unexpectedly removed upon certain changes to NNCP configurations (bond, VLAN, routes). This is under active investigation and the root cause appears to be related to Nmstate reactivating connections rather than reapplying configurations. With the reproducer available, the team is currently working on understanding the issue more and providing a fix within the current sprint timeline (ending 2024-10-07). [2024-11-04] The upstream work to avoid some unecessary work deactivations due to routes mismatches is still ongoing. The goal is to have a fix ready to be included in the 9.2 batch update 13 (Planned Dec. 09, 2024). [2024-11-14] The upstream work is merged upstream and will be available on next week Nmstate build. The goal is still to have the fix shipped in the 9.2 batch update 13 (Planned Dec. 09, 2024). [2024-11-25] The RHEL-9.6 and RHEL-9.5 builds are done and the goal is to still to have the RHEL-9.2 fix shipped in the batch update 13 (Planned Dec. 09, 2024).

Show

Customer/Partner Jira ID Customer Case Status Details Nokia RHEL-64707 03834014 The customer application pod interface is being unexpectedly removed upon certain changes to NNCP configurations (bond, VLAN, routes). This is under active investigation and the root cause appears to be related to Nmstate reactivating connections rather than reapplying configurations. With the reproducer available, the team is currently working on understanding the issue more and providing a fix within the current sprint timeline (ending 2024-10-07). [2024-11-04] The upstream work to avoid some unecessary work deactivations due to routes mismatches is still ongoing. The goal is to have a fix ready to be included in the 9.2 batch update 13 (Planned Dec. 09, 2024). [2024-11-14] The upstream work is merged upstream and will be available on next week Nmstate build. The goal is still to have the fix shipped in the 9.2 batch update 13 (Planned Dec. 09, 2024). [2024-11-25] The RHEL-9.6 and RHEL-9.5 builds are done and the goal is to still to have the RHEL-9.2 fix shipped in the batch update 13 (Planned Dec. 09, 2024).

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Sprint:
None
Release Blocker:
Approved Blocker
Target Backport Versions:

rhel-9.2.0.z

Acceptance Criteria:
Hide

Given a system administrator has configured multiple NNCPs with VLAN and route settings on an application pod in OCP and the pod includes multiple interfaces managed by Nmstate,

When they make a change to an NNCP configuration,

Then, Nmstate should correctly reapply the configuration without removing or disrupting any existing interfaces.

Definition of Done:

The implementation meets the acceptance criteria

Integration tests are written and pass

The code is part of a downstream build attached to an errata

The fix is backported to rhel-9.2.z
Show
Given a system administrator has configured multiple NNCPs with VLAN and route settings on an application pod in OCP and the pod includes multiple interfaces managed by Nmstate, When they make a change to an NNCP configuration, Then, Nmstate should correctly reapply the configuration without removing or disrupting any existing interfaces. Definition of Done: The implementation meets the acceptance criteria Integration tests are written and pass The code is part of a downstream build attached to an errata The fix is backported to rhel-9.2.z
Git Pull Request:
https://github.com/nmstate/nmstate/pull/2809
Preliminary Testing:
Pass
Errata Link:
https://errata.engineering.redhat.com/advisory/139365
Test Coverage:

Automated

Experience:
RH Private Keywords:

PX Impact Score:
SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

This comes from this bug https://issues.redhat.com/browse/OCPBUGS-41906

Customer application pod have multiple interfaces NNCPs (bond + vlan + routes) and 2 NADs based on vlans and one of the interfaces gets removed; Once the pod gets restarted the interface is restored

k-nmstate version

$ omc get csv -n openshift-nmstate
NAME                                              DISPLAY                       VERSION               REPLACES                                          PHASE
kubernetes-nmstate-operator.4.14.0-202410162010   Kubernetes NMState Operator   4.14.0-202410162010   kubernetes-nmstate-operator.4.14.0-202409250809   Succeeded

Steps to reproduce the issue:

reproducer the same issue (https://gitlab.cee.redhat.com/jclaretm/case03834014) (ocp 4.14.7), I can see that one of the interfaces gets deleted when I make changes to one of the nncp's (bond-tenant-vlan111-ipv4-route)

# oc get nnce
NAME                                        STATUS      STATUS AGE   REASON
master-0.bond-tenant-policy                 Available   94m          SuccessfullyConfigured
master-0.bond-tenant-vlan110-ip-worker-16   Available   3h19m        SuccessfullyConfigured
master-0.bond-tenant-vlan111-ip-worker-16   Available   3h19m        SuccessfullyConfigured
master-0.bond-tenant-vlan111-ipv4-route     Available   49m          SuccessfullyConfigured
master-0.bond-tenant-vlan112-ip-worker-16   Available   3h19m        SuccessfullyConfigured
master-0.bond-tenant-vlan112-ipv4-route     Available   59m          SuccessfullyConfigured
master-1.bond-tenant-policy                 Available   94m          SuccessfullyConfigured
master-1.bond-tenant-vlan110-ip-worker-16   Available   3h20m        SuccessfullyConfigured
master-1.bond-tenant-vlan111-ip-worker-16   Available   3h20m        SuccessfullyConfigured
master-1.bond-tenant-vlan111-ipv4-route     Available   49m          SuccessfullyConfigured
master-1.bond-tenant-vlan112-ip-worker-16   Available   3h20m        SuccessfullyConfigured
master-1.bond-tenant-vlan112-ipv4-route     Available   59m          SuccessfullyConfigured
master-2.bond-tenant-policy                 Available   94m          SuccessfullyConfigured
master-2.bond-tenant-vlan110-ip-worker-16   Available   3h20m        SuccessfullyConfigured
master-2.bond-tenant-vlan111-ip-worker-16   Available   3h20m        SuccessfullyConfigured
master-2.bond-tenant-vlan111-ipv4-route     Available   49m          SuccessfullyConfigured
master-2.bond-tenant-vlan112-ip-worker-16   Available   3h20m        SuccessfullyConfigured
master-2.bond-tenant-vlan112-ipv4-route     Available   59m          SuccessfullyConfigured

# oc get net-attach-def -n osns0288
NAME                   AGE
ext-oam-lan            3h16m
osns0288-ext-app-lan   3h16m
osns0288-ext-db-lan    3h16m

# oc get pod -n osns0288
NAME                                 READY   STATUS    RESTARTS   AGE
sample-deployment-6878d8d688-4htn8   1/1     Running   0          50m

# oc rsh sample-deployment-6878d8d688-4htn8
sh-4.4$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if3881: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether 0a:58:0a:80:01:fa brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.128.1.250/23 brd 10.128.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::858:aff:fe80:1fa/64 scope link 
       valid_lft forever preferred_lft forever
3: eth3@if3874: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default 
    link/ether b4:83:51:0a:84:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 100.73.157.1/22 brd 100.73.159.255 scope global eth3
       valid_lft forever preferred_lft forever
    inet6 fe80::b483:5100:10a:8486/64 scope link 
       valid_lft forever preferred_lft forever
4: eth5@if3549: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default 
    link/ether b4:83:51:0a:84:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 100.73.153.1/22 brd 100.73.155.255 scope global eth5
       valid_lft forever preferred_lft forever
    inet6 fe80::b483:5100:10a:8486/64 scope link 
       valid_lft forever preferred_lft forever

Once the application is deployed, edit nncp bond-tenant-vlan111-ipv4-route and make a change in the description to force re-apply

# oc edit nncp bond-tenant-vlan111-ipv4-route

# oc get nnce
NAME                                        STATUS        STATUS AGE   REASON
...
master-0.bond-tenant-vlan111-ipv4-route     Progressing   3s           ConfigurationProgressing
master-1.bond-tenant-vlan111-ipv4-route     Progressing   2s           ConfigurationProgressing
master-2.bond-tenant-vlan111-ipv4-route     Progressing   2s           ConfigurationProgressing
...

# oc get pod
NAME                                 READY   STATUS    RESTARTS   AGE
sample-deployment-6878d8d688-tmsbg   1/1     Running   0          119s

# oc rsh sample-deployment-6878d8d688-tmsbg
sh-4.4$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if3881: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether 0a:58:0a:80:01:fa brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.128.1.250/23 brd 10.128.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::858:aff:fe80:1fa/64 scope link 
       valid_lft forever preferred_lft forever
4: eth5@if3549: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default 
    link/ether b4:83:51:0a:84:86 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 100.73.153.1/22 brd 100.73.155.255 scope global eth5
       valid_lft forever preferred_lft forever
    inet6 fe80::b483:5100:10a:8486/64 scope link 
       valid_lft forever preferred_lft forever

after restarting the pod, the interface is there again

# oc delete pod sample-deployment-6878d8d688-tmsbg
pod "sample-deployment-6878d8d688-tmsbg" deleted

# oc get pod
NAME                                 READY   STATUS    RESTARTS   AGE
sample-deployment-6878d8d688-mnxj5   1/1     Running   0          4s

# oc rsh sample-deployment-6878d8d688-mnxj5
sh-4.4$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if3909: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default 
    link/ether 0a:58:0a:83:01:c9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.131.1.201/23 brd 10.131.1.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::858:aff:fe83:1c9/64 scope link 
       valid_lft forever preferred_lft forever
3: eth3@if3907: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default 
    link/ether b4:83:51:0a:88:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 100.73.157.1/22 brd 100.73.159.255 scope global eth3
       valid_lft forever preferred_lft forever
    inet6 fe80::b483:5100:10a:8830/64 scope link 
       valid_lft forever preferred_lft forever
4: eth5@if3564: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default 
    link/ether b4:83:51:0a:88:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 100.73.153.1/22 brd 100.73.155.255 scope global eth5
       valid_lft forever preferred_lft forever
    inet6 fe80::b483:5100:10a:8830/64 scope link 
       valid_lft forever preferred_lft forever

The pod interface is being removed also restarting the handler pod or the node

causes

OCPBUGS-41906 Missing Pod interface

Closed

relates to

OCPBUGS-41906 Missing Pod interface

Closed

links to

RHEA-2024:139365 nmstate bug fix and enhancement update

Assignee:: Inigo Huguet

Reporter:: Jorge Claret Membrado

Developer:: Network Management Team

QA Contact:: Mingyu Shi

Votes:: 0 Vote for this issue

Watchers:: 14 Start watching this issue

Created:: 2024/10/24 10:59 AM

Updated:: 2025/01/20 8:44 AM

Next Planned Release Date:: 2025/05/13

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates