-
Bug
-
Resolution: Unresolved
-
Minor
-
None
-
4.15
-
+
-
Low
-
No
-
False
-
openshift-install does not remove some ports when destroying the dualstack cluster. We realized this because our CI automation deletes the subnet and fails to find that ports still exist. It appears to be related to Amphora or OVN load balancer services on primary ipv4 and primary ipv6 dualstack cluster installations.
In IPv4-primary is the ipv4 subnet the one with leftovers:
TASK [cleanup : Remove subnet subnet-dualstack-v4] *****************************
task path: /home/rhos-ci/jenkins/workspace/DFG-osasinfra-shiftstack_ci-ocp_install-4.15-ovnkubernetes-upi/infrared/plugins/openshift/roles/cleanup/tasks/purge_projects.yml:155
Friday 09 February 2024 20:39:01 +0000 (0:00:03.337) 0:03:07.890 *******
fatal: [undercloud-0]: FAILED! => {
"changed": false,
"extra_data": {
"data": null,
"details": "Unable to complete operation on subnet a7661859-6448-4bbf-988b-8cb8b01b5d8c: One or more ports have an IP allocation from this subnet.",
"response": "{\"NeutronError\": {\"type\": \"SubnetInUse\", \"message\": \"Unable to complete operation on subnet a7661859-6448-4bbf-988b-8cb8b01b5d8c: One or more ports have an IP allocation from this subnet.\", \"detail\": \"\"}}"
}
}
MSG:
ConflictException: 409: Client Error for url: https://10.46.44.10:13696/v2.0/subnets/a7661859-6448-4bbf-988b-8cb8b01b5d8c, Unable to complete operation on subnet a7661859-6448-4bbf-988b-8cb8b01b5d8c: One or more ports have an IP allocation from this subnet.
TASK [cleanup : List all ports defined inside network network-dualstack] *******
task path: /home/rhos-ci/jenkins/workspace/DFG-osasinfra-shiftstack_ci-ocp_install-4.15-ovnkubernetes-upi/infrared/plugins/openshift/roles/cleanup/tasks/purge_projects.yml:170
Friday 09 February 2024 20:39:03 +0000 (0:00:02.363) 0:03:10.254 *******
changed: [undercloud-0] => {
"changed": true,
"cmd": "source ~/shiftstackrc && openstack port list --network network-dualstack",
"delta": "0:00:02.967053",
"end": "2024-02-09 20:39:06.942024",
"rc": 0,
"start": "2024-02-09 20:39:03.974971"
}
STDOUT:
+--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------------------------+--------+
| 11b0d26b-906b-4951-8ab7-83f5ce67e4ad | octavia-lb-85846d30-5c30-43a1-a17f-56b3ba3fc39e | fa:16:3e:98:6d:8d | ip_address='192.168.192.176', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fe98:6d8d', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| 1ad443c0-b2a1-4dc8-9478-ee12a87ddfbe | octavia-lb-2d954be4-c734-46e4-9c3e-cb5264dd7e35 | fa:16:3e:fa:f0:c3 | ip_address='192.168.192.4', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fefa:f0c3', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| 2cbb2fbf-a63b-476c-a5f0-9d65a378795a | octavia-lb-3124ac9f-eb96-44f1-a682-8ed43097b40e | fa:16:3e:9a:f7:45 | ip_address='192.168.192.105', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fe9a:f745', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| 4b10d1f2-0aea-4b93-a41c-d4fa40f84c10 | octavia-lb-43f1ac9f-ea18-4270-8426-7bba254fc0c3 | fa:16:3e:2b:c4:d5 | ip_address='192.168.192.92', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fe2b:c4d5', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| 4e6178b3-d332-4d79-909b-39926dea5891 | octavia-lb-73b3e7a1-638f-405e-9d4e-edbc0b903cbe | fa:16:3e:f0:2d:bb | ip_address='192.168.192.8', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fef0:2dbb', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| 5da3420e-143c-4b49-b761-a4fe7aefe8d1 | octavia-lb-1a139b85-eea8-42a6-b27e-0c3f743e77cb | fa:16:3e:c3:d2:0f | ip_address='192.168.192.138', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fec3:d20f', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| 7a21e92d-53c9-4f2e-b409-0d0cfdb00438 | octavia-lb-754c27d6-bbee-4d83-b72c-431de664b967 | fa:16:3e:27:d6:7b | ip_address='192.168.192.82', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fe27:d67b', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| 7c1a4dd2-4499-4df1-9b48-fd1f18b39cf9 | octavia-lb-c00ffdc2-af02-485a-bbdb-091a3137fa96 | fa:16:3e:df:d8:c4 | ip_address='192.168.192.160', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fedf:d8c4', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| a171f68a-0308-4b6c-85d7-9c8d955280c1 | octavia-lb-1d975563-bbcd-4c9f-b844-82e5be630999 | fa:16:3e:6b:6b:6f | ip_address='192.168.192.27', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fe6b:6b6f', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| a5448be0-ae08-4d97-9cf0-d145c5307fc6 | octavia-lb-567e1436-ff91-4dbc-a9ac-be65beb9fcd9 | fa:16:3e:f2:21:9a | ip_address='192.168.192.191', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fef2:219a', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| cf93694d-4088-40aa-b8e0-37713d14439c | octavia-lb-4e6ac45e-df56-4a71-9278-f3c5ded0d79f | fa:16:3e:3f:34:94 | ip_address='192.168.192.155', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fe3f:3494', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| e078cb83-4481-4f50-8a13-a1ec66a42be0 | octavia-lb-fe9f4fc6-227f-4ae7-af63-464927fccb2b | fa:16:3e:fd:b7:5e | ip_address='192.168.192.15', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fefd:b75e', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| e1ab4880-6d31-40eb-aa0e-840b14cf7149 | octavia-lb-50c298e3-5423-4d5f-8a86-785ed1e4fe13 | fa:16:3e:a6:9d:9f | ip_address='192.168.192.90', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fea6:9d9f', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| f10ec791-bd3b-44f3-bcd6-1d61e0773a5a | octavia-lb-68c999db-49ec-4bae-afb1-1a73ae49180a | fa:16:3e:8d:65:a9 | ip_address='192.168.192.62', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fe8d:65a9', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
| f97a53bc-22af-42b5-9500-5c9bf0c36e8f | | fa:16:3e:1a:c1:41 | ip_address='192.168.192.2', subnet_id='a7661859-6448-4bbf-988b-8cb8b01b5d8c' | DOWN |
| | | | ip_address='fd2e:6f44:5dd8:c956:f816:3eff:fe1a:c141', subnet_id='d5022f92-23f4-478e-9f84-a9925e85e50e' | |
+--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------------------------+--------+
In IPv6-primary looks the ipv6 subnet is the one with leftovers:
TASK [cleanup : Remove subnet subnet-dualstack-v6] *****************************
task path: /home/rhos-ci/jenkins/workspace/DFG-osasinfra-shiftstack_ci-ocp_install-4.16-ovnkubernetes-ipi/infrared/plugins/openshift/roles/cleanup/tasks/purge_projects.yml:162
Monday 12 February 2024 14:34:27 +0000 (0:00:02.365) 0:03:02.775 *******
fatal: [undercloud-0]: FAILED! => {
"changed": false,
"extra_data": {
"data": null,
"details": "Unable to complete operation on subnet 648feab4-be61-4f69-a2a7-f54a7bbe5375: One or more ports have an IP allocation from this subnet.",
"response": "{\"NeutronError\": {\"type\": \"SubnetInUse\", \"message\": \"Unable to complete operation on subnet 648feab4-be61-4f69-a2a7-f54a7bbe5375: One or more ports have an IP allocation from this subnet.\", \"detail\": \"\"}}"
}
}
MSG:
ConflictException: 409: Client Error for url: https://10.0.0.101:13696/v2.0/subnets/648feab4-be61-4f69-a2a7-f54a7bbe5375, Unable to complete operation on subnet 648feab4-be61-4f69-a2a7-f54a7bbe5375: One or more ports have an IP allocation from this subnet.
TASK [cleanup : List all ports defined inside network network-dualstack] *******
task path: /home/rhos-ci/jenkins/workspace/DFG-osasinfra-shiftstack_ci-ocp_install-4.16-ovnkubernetes-ipi/infrared/plugins/openshift/roles/cleanup/tasks/purge_projects.yml:170
Monday 12 February 2024 14:34:28 +0000 (0:00:01.772) 0:03:04.548 *******
changed: [undercloud-0] => {
"changed": true,
"cmd": "source ~/shiftstackrc && openstack port list --network network-dualstack",
"delta": "0:00:01.791027",
"end": "2024-02-12 09:34:31.251251",
"rc": 0,
"start": "2024-02-12 09:34:29.460224"
}
STDOUT:
+--------------------------------------+-------------------------------------------------+-------------------+-----------------------------------------------------------------------------------------+--------+
| ID | Name | MAC Address | Fixed IP Addresses | Status |
+--------------------------------------+-------------------------------------------------+-------------------+-----------------------------------------------------------------------------------------+--------+
| 16f34338-8253-4054-ac11-f906f0b0bdc5 | ovn-lb-vip-5c17a814-ef1b-44d7-b4a8-7173ad3309a6 | fa:16:3e:8c:ce:5b | ip_address='fd2e:6f44:5dd8:c956::16e', subnet_id='648feab4-be61-4f69-a2a7-f54a7bbe5375' | DOWN |
| 771bcacc-2e79-4a6b-ae0e-6bcef1e659d6 | ovn-lb-vip-2a7d7bd5-e3d1-4b42-a94c-8c5b1ddc68c4 | fa:16:3e:00:ae:f3 | ip_address='fd2e:6f44:5dd8:c956::2eb', subnet_id='648feab4-be61-4f69-a2a7-f54a7bbe5375' | DOWN |
| dd3b4195-0fd4-42d2-947b-d4bc69cc12e8 | | fa:16:3e:27:53:72 | | DOWN |
| f1f4d5cb-395b-4421-aa0e-de5bb66e6bd4 | ovn-lb-vip-42b3af7f-3aa0-48ec-aba3-54f0d1a98865 | fa:16:3e:cc:1b:66 | ip_address='fd2e:6f44:5dd8:c956::3dd', subnet_id='648feab4-be61-4f69-a2a7-f54a7bbe5375' | DOWN |
| ffb97068-e322-4662-931d-dba965923cca | ovn-lb-vip-3bab7bb5-ebc3-49b3-b39c-4381925e121e | fa:16:3e:fd:c9:5d | ip_address='fd2e:6f44:5dd8:c956::3b9', subnet_id='648feab4-be61-4f69-a2a7-f54a7bbe5375' | DOWN |
+--------------------------------------+-------------------------------------------------+-------------------+-----------------------------------------------------------------------------------------+--------+
Versions:
- RHOS-16.2-RHEL-8-20231215.n.1
- 4.15.0-0.nightly-2024-02-07-062935
Steps to Reproduce:
We are running openstack-tests after installation with different cloud-provider-config's (to test OVN-octavia and Amphora-provider with and without monitors). The issue is not persistent, we observed on the CI from time to time.
Actual results: Removing the BYON network after cluster destroy fails because some ports remain.
Expected results: openshift-install destroy should remove all the openstack resources related to it.
Additional info: Attaching destroy logs in private comment.