Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Critical
Fix Version/s: rhos-18.0.4
Affects Version/s: None
Component/s: data-plane-adoption, edpm-ansible, os-net-config
Labels:
None

Story Points:
3
Epic Link:
OSPRH-8494
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Dev Approval:
?
Docs Approval:
No Docs Impact
Fixed in Build:
openstack-ansible-ee-container-1.0.4-1
PM Approval:
?
QE Approval:
?
Regression:
None
Release Note Text:

Hide
.Update to latest RHOSP 17.1 version before adopting

When performing adoption of a source environment which is older than RHOSP 17.1.4, the workloads experience a prolonged network connectivity disruption. Make sure to update the source environment at least to RHOSP 17.1.4 before adopting.

Show
.Update to latest RHOSP 17.1 version before adopting When performing adoption of a source environment which is older than RHOSP 17.1.4, the workloads experience a prolonged network connectivity disruption. Make sure to update the source environment at least to RHOSP 17.1.4 before adopting.
Release Note Type:
Known Issue
Release Note Status:
Done
Intelligence Requested:
Market:
Release Blocker:
Rejected

Sprint:
Neutron Sprint 1, Neutron Sprint 2, Neutron Sprint 3, Neutron Sprint 4
sprint_count:
4
Severity:
Critical

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

After adding some VMs into adoption+networker job that covers:
1. VM with FIP
2. VM with external ip
3. VM ping to external using router (SNAT)
4. VM's pinging each other from different computes.

There are ping loss for all VMs for 5,55 min - more less during task:wait for dataplane node set to be read : "start": "2024-09-19 09:57:03.983260~end": "2024-09-19 10:19:46.594802
https://sf.hosted.upshift.rdu2.redhat.com/logs/55/355/204edcee371235481f6fe64981791cf69fdd50f[...]sts/logs/test_minimal_out_2024-09-19T09:05:31EDT.log

sh-5.1$ openstack server list --all
+--------------------------------------+-----------------------+--------+-------------------------------------------------------+-------------------------------------+-------------------+
| ID                                   | Name                  | Status | Networks                                              | Image                               | Flavor            |
+--------------------------------------+-----------------------+--------+-------------------------------------------------------+-------------------------------------+-------------------+
| accba1c3-b20b-459c-b3f8-94ca3e2d74f0 | adoption-server5      | ACTIVE | adoption-net0=192.168.122.174, 192.168.99.173         | custom_neutron_guest_rhel_8.4.qcow2 | customized_flavor |
| b6ab012e-d94f-43d9-ae2c-41f8916e0020 | adoption-server4      | ACTIVE | adoption-net0=192.168.122.236, 192.168.99.246         | custom_neutron_guest_rhel_8.4.qcow2 | customized_flavor |
| b41a581a-bc1b-41e9-8222-68ce9fe85453 | adoption-trunk-server | ACTIVE | public=192.168.122.218                                | custom_neutron_guest_rhel_8.4.qcow2 | customized_flavor |
| 822a1f61-1aa6-442c-88c1-06cb6f71f0f9 | adoption-server3      | ACTIVE | adoption-net1=192.168.101.231; public=192.168.122.194 | custom_neutron_guest_rhel_8.4.qcow2 | customized_flavor |
| 64d107ea-b719-4071-a92e-a44f411fb250 | adoption-server2      | ACTIVE | adoption-net2=192.168.102.49; public=192.168.122.225  | custom_neutron_guest_rhel_8.4.qcow2 | customized_flavor |
| ad9e6b97-e353-4870-8013-5a904f028526 | adoption-server1      | ACTIVE | adoption-net1=192.168.101.112; public=192.168.122.196 | custom_neutron_guest_rhel_8.4.qcow2 | customized_flavor |
| ee408d3d-e6b5-4294-b4fc-c0c90be43b89 | bfv-server            | ACTIVE | private=192.168.0.143                                 | N/A (booted from volume)            | m1.small          |
| 33d1f8bb-9739-4192-801f-f070cc7a8ce5 | test                  | ACTIVE | private=192.168.0.130, 192.168.122.20                 | cirros                              | m1.small          |
+--------------------------------------+-----------------------+--------+-------------------------------------------------------+-------------------------------------+-------------------+

Ping loss:
from

[1726754584.781724] 64 bytes from 192.168.122.196: icmp_seq=2515 ttl=64 time=0.931 ms
[1726754585.783045] 64 bytes from 192.168.122.196: icmp_seq=2516 ttl=64 time=1.10 ms
[1726754600.129080] From 192.168.122.100 icmp_seq=2527 Destination Host Unreachable
[1726754600.129156] From 192.168.122.100 icmp_seq=2528 Destination Host Unreachable

to:

[1726754925.761175] From 192.168.122.100 icmp_seq=2846 Destination Host Unreachable [1726754925.761181] From 192.168.122.100 icmp_seq=2847 Destination Host Unreachable [1726754926.569386] 64 bytes from 192.168.122.196: icmp_seq=2848 ttl=64 time=808 ms [1726754926.763663] 64 bytes from 192.168.122.196: icmp_seq=2849 ttl=64 time=1.82 ms

--- 192.168.122.196 ping statistics ---
3784 packets transmitted, 3453 received, +313 errors, 8.74736% packet loss, time 3801793ms
rtt min/avg/max/mdev = 0.620/1.451/808.133/13.787 ms, pipe 4

Ping statistics:

--- 192.168.122.174 ping statistics ---
3783 packets transmitted, 3451 received, +313 errors, 8.7761% packet loss, time 3801040ms
rtt min/avg/max/mdev = 0.625/1.180/50.794/1.132 ms, pipe 4

 --- 192.168.122.194 ping statistics ---
3779 packets transmitted, 3448 received, +302 errors, 8.75893% packet loss, time 3801140ms
rtt min/avg/max/mdev = 0.522/1.218/296.176/5.136 ms, pipe 4

--- 192.168.122.196 ping statistics ---
3784 packets transmitted, 3453 received, +313 errors, 8.74736% packet loss, time 3801793ms
rtt min/avg/max/mdev = 0.620/1.451/808.133/13.787 ms, pipe 4

--- 192.168.122.218 ping statistics --- 3780 packets transmitted, 3448 received, +269 errors, 8.78307% packet loss, time 3801722ms rtt min/avg/max/mdev = 0.472/1.125/28.366/0.763 ms, pipe 4

--- 192.168.122.225 ping statistics ---
3783 packets transmitted, 3453 received, +306 errors, 8.72324% packet loss, time 3801222ms
rtt min/avg/max/mdev = 0.559/1.873/1704.121/31.209 ms, pipe 4

--- 192.168.122.236 ping statistics --- 3786 packets transmitted, 3455 received, +301 errors, 8.74274% packet loss, time 3801922ms rtt min/avg/max/mdev = 0.587/2.218/2227.092/43.135 ms, pipe 4

Test-Project: https://gitlab.cee.redhat.com/ci-framework/ci-framework-testproject/-/merge_requests/355

Result: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/c601ae42e5f34ec794055f9fddae230d

Slack thread: https://redhat-internal.slack.com/archives/C046JULBVJ7/p1726759238127619

Note:

[cloud-user@adoption-server1 ~]$ cat /proc/sys/net/ipv4/route/gc_timeout 
300
[cloud-user@adoption-server1 ~]$ cat /proc/sys/net/ipv4/neigh/default/gc_stale_time
60