-
Bug
-
Resolution: Duplicate
-
Normal
-
rhos-18.0.0
-
None
Failure noticed first on neutron crc job [1].
On autohold for debugging, tried setting the burst_limit different configuration multiple times, sometimes network_log and sometimes DEFAULT section.
The values are set in new neutron pod under '/etc/neutron/neutron.conf.d/02-neutron-custom.conf' but not in nbdb.
Manual autohold checks can be seen here [2].
[1]
test_only_dropped_traffic_logged[id-7bab2764-2aac-49da-bd0f-161128695bfb]
...
2024-05-09 20:04:46,819 179 DEBUG [whitebox_neutron_tempest_plugin.tests.scenario.base] Set configuration command:
oc -n openstack patch $(oc -n openstack get oscp -o name) --type=merge --patch '
spec:
neutron:
template:
customServiceConfig: |
[network_log]
burst_limit = 49'
2024-05-09 20:04:46,819 179 INFO [tempest.lib.common.ssh] Creating ssh connection to '10.0.199.212:22' as 'zuul' with password None
2024-05-09 20:04:46,833 179 INFO [paramiko.transport] Connected (version 2.0, client OpenSSH_8.7)
2024-05-09 20:04:46,897 179 INFO [paramiko.transport] Authentication (publickey) successful!
2024-05-09 20:04:46,898 179 INFO [tempest.lib.common.ssh] ssh connection to zuul@10.0.199.212 successfully created
2024-05-09 20:04:47,399 179 DEBUG [whitebox_neutron_tempest_plugin.tests.scenario.base] Output:
openstackcontrolplane.core.openstack.org/openstack-galera-network-isolation patched
2024-05-09 20:04:47,399 179 DEBUG [whitebox_neutron_tempest_plugin.tests.scenario.base] Set configuration command:
oc -n openstack patch $(oc -n openstack get oscp -o name) --type=merge --patch '
spec:
neutron:
template:
customServiceConfig: |
[network_log]
rate_limit = 178'
...
2024-05-09 20:04:48,442 179 DEBUG [neutron_tempest_plugin.common.shell] Command 'oc -n openstack rsh ovsdbserver-nb-0 ovn-nbctl list meter-band' succeeded:
stderr:
None
stdout:
_uuid : e086199a-5992-486f-b521-5382f1584c56
action : drop
burst_size : 25
external_ids : {}
rate : 214
_uuid : d13f6e63-4eb2-4c03-a2c7-f2410c0829f8
action : drop
burst_size : 12
external_ids : {}
rate : 107
2024-05-09 20:04:58,447 179 DEBUG [neutron_tempest_plugin.common.shell] Executing command 'oc -n openstack rsh ovsdbserver-nb-0 ovn-nbctl list meter-band' on remote host '10.0.199.212' (timeout=60.0)...
2024-05-09 20:04:58,448 179 INFO [tempest.lib.common.ssh] Creating ssh connection to '10.0.199.212:22' as 'zuul' with password None
2024-05-09 20:04:58,461 179 INFO [paramiko.transport] Connected (version 2.0, client OpenSSH_8.7)
2024-05-09 20:04:58,524 179 INFO [paramiko.transport] Authentication (publickey) successful!
2024-05-09 20:04:58,525 179 INFO [tempest.lib.common.ssh] ssh connection to zuul@10.0.199.212 successfully created
2024-05-09 20:04:58,868 179 DEBUG [neutron_tempest_plugin.common.shell] Command 'oc -n openstack rsh ovsdbserver-nb-0 ovn-nbctl list meter-band' succeeded:
stderr:
None
stdout:
_uuid : e086199a-5992-486f-b521-5382f1584c56
action : drop
burst_size : 25
external_ids : {}
rate : 214
_uuid : d13f6e63-4eb2-4c03-a2c7-f2410c0829f8
action : drop
burst_size : 12
external_ids : {}
rate : 107
...
Traceback (most recent call last):
File "/var/lib/tempest/external-plugins/.venv/lib64/python3.9/site-packages/neutron_tempest_plugin/common/utils.py", line 85, in wait_until_true
eventlet.sleep(sleep)
File "/var/lib/tempest/external-plugins/.venv/lib64/python3.9/site-packages/eventlet/greenthread.py", line 36, in sleep
hub.switch()
File "/var/lib/tempest/external-plugins/.venv/lib64/python3.9/site-packages/eventlet/hubs/hub.py", line 313, in switch
return self.greenlet.switch()
eventlet.timeout.Timeout: 180 seconds
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/var/lib/tempest/external-plugins/.venv/lib64/python3.9/site-packages/whitebox_neutron_tempest_plugin/tests/scenario/test_security_group_logging.py", line 914, in test_only_dropped_traffic_logged
self._test_only_dropped_traffic_logged()
File "/var/lib/tempest/external-plugins/.venv/lib64/python3.9/site-packages/whitebox_neutron_tempest_plugin/tests/scenario/test_security_group_logging.py", line 479, in _test_only_dropped_traffic_logged
common_utils.wait_until_true(
File "/var/lib/tempest/external-plugins/.venv/lib64/python3.9/site-packages/neutron_tempest_plugin/common/utils.py", line 89, in wait_until_true
raise exception
RuntimeError: Timed out: burst/rate limits not configured properly on master/leader controller.
[2]
[zuul@controller ~]$ oc -n openstack patch $(oc -n openstack get oscp -o name) --type=merge --patch '
spec:
neutron:
template:
customServiceConfig: |
[network_log]
burst_limit = 51'
openstackcontrolplane.core.openstack.org/openstack-galera-network-isolation patched
[zuul@controller ~]$ sleep 120 && oc -n openstack rsh ovsdbserver-nb-0 ovn-nbctl list meter-band
_uuid : 5687ce5a-9a86-479d-a748-f511ebad64ac
action : drop
burst_size : 12
external_ids : {}
rate : 50
_uuid : be59718a-d731-4187-bdad-4f32ee6bc5ff
action : drop
burst_size : 25
external_ids : {}
rate : 100
[zuul@controller ~]$ oc -n openstack rsh neutron-6d58c77f7f-jq77q grep -IErin -B20 burst_limit /etc/neutron/neutron.conf.d/
Defaulted container "neutron-api" out of: neutron-api, neutron-httpd
/etc/neutron/neutron.conf.d/02-neutron-custom.conf-1-[network_log]
/etc/neutron/neutron.conf.d/02-neutron-custom.conf:2:burst_limit = 51
[zuul@controller ~]$ oc -n openstack rsh ovsdbserver-nb-0 ovn-nbctl list meter-band
_uuid : 5687ce5a-9a86-479d-a748-f511ebad64ac
action : drop
burst_size : 12
external_ids : {}
rate : 50
_uuid : be59718a-d731-4187-bdad-4f32ee6bc5ff
action : drop
burst_size : 25
external_ids : {}
rate : 100
- is caused by
-
-
- Closed
-
- is related to
-
-
- Closed
-
- links to
- mentioned on
(2 mentioned in, 1 mentioned on)
