-
Bug
-
Resolution: Done
-
Blocker
-
None
-
None
-
False
-
-
False
-
?
-
?
-
?
-
?
-
None
-
Release Note Not Required
-
-
-
VANS2024Q2, VANS-001
-
Important
After the creation of a resource in the DB, Octavia always fetches the new object from the DB, but it randomly fails:
octavia-api 2024-05-27 08:17:58.733 14 DEBUG octavia.db.repositories [None req-6cb948b6-87fa-4847-a9e9-84226ee68380 - fd9cf0b2fb224957a9bf87f86bafa04e - - default default] Checking quota for project: fd9cf0b2fb224957a9bf87f86bafa04e object: <class 'octavia.common.data_models.Pool'> che ck_quota_met /usr/lib/python3.9/site-packages/octavia/db/repositories.py:386\x1b[00m 2024-05-27 08:17:59.109 14 INFO octavia.api.v2.controllers.pool [None req-6cb948b6-87fa-4847-a9e9-84226ee68380 - fd9cf0b2fb224957a9bf87f86bafa04e - - default default] Sending create Pool aa9cede4-4737-4f16-8d9d-4b7ecb6c0d5e to provider octavia\x1b[00m 2024-05-27 08:17:59.126 14 DEBUG octavia.api.v2.controllers.base [None req-6cb948b6-87fa-4847-a9e9-84226ee68380 - fd9cf0b2fb224957a9bf87f86bafa04e - - default default] Pool aa9cede4-4737-4f16-8d9d-4b7ecb6c0d5e not found _get_db_obj /usr/lib/python3.9/site-packages/octavia/api/v2/controllers/base.py:71\x1b[00m 2024-05-27 08:17:59.126 14 DEBUG wsme.api [None req-6cb948b6-87fa-4847-a9e9-84226ee68380 - fd9cf0b2fb224957a9bf87f86bafa04e - - default default] Client-side error: Pool aa9cede4-4737-4f16-8d9d-4b7ecb6c0d5e not found. format_exception /usr/lib/python3.9/site-packages/wsme/api.py:223\x1b[00m 192.168.32.22 - - [27/May/2024:08:17:58 +0000] "POST /v2.0/lbaas/pools HTTP/1.1" 404 113 "-" "python-urllib3/1.26.5" octavia-worker 2024-05-27 08:17:59.123 17 INFO octavia.controller.queue.v2.endpoints [-] Creating pool 'aa9cede4-4737-4f16-8d9d-4b7ecb6c0d5e'...ESC[00m 2024-05-27 08:17:59.439 17 DEBUG octavia.controller.worker.v2.controller_worker [-] Flow 'octavia-create-pool-flow' (648d765e-5096-4d7e-9f52-28aa316d190d) transitioned into state 'RUNNING' from state 'PENDING' _flow_receiver /usr/lib/python3.9/site-packages/taskflow/listeners/logging.py:141ESC[00m 2024-05-27 08:17:59.440 17 DEBUG octavia.controller.worker.v2.controller_worker [-] Task 'octavia.controller.worker.v2.tasks.lifecycle_tasks.PoolToErrorOnRevertTask' (f823081d-c46a-4dae-93a8-a277dca80717) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python3.9/site-packages/taskflow/listeners/logging.py:190ESC[00m 2024-05-27 08:17:59.442 17 DEBUG octavia.controller.worker.v2.controller_worker [-] Task 'octavia.controller.worker.v2.tasks.lifecycle_tasks.PoolToErrorOnRevertTask' (f823081d-c46a-4dae-93a8-a277dca80717) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python3.9/site-packages/taskflow/listeners/logging.py:178ESC[00m 2024-05-27 08:17:59.443 17 DEBUG octavia.controller.worker.v2.controller_worker [-] Task 'octavia.controller.worker.v2.tasks.database_tasks.MarkPoolPendingCreateInDB' (6958ab1a-352d-497c-9665-ba49bf00af9e) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python3.9/site-packages/taskflow/listeners/logging.py:190ESC[00m 2024-05-27 08:17:59.444 17 DEBUG octavia.controller.worker.v2.tasks.database_tasks [None req-26b91571-9e1f-47f4-9504-2de93a4b536e - 8784eb28c4d1436fb0be9cfc8456d61b - - - -] Mark PENDING CREATE in DB for pool id: aa9cede4-4737-4f16-8d9d-4b7ecb6c0d5e execute /usr/lib/python3.9/site-packages/octavia/controller/worker/v2/tasks/database_tasks.py:2398ESC[00m 2024-05-27 08:17:59.459 17 DEBUG octavia.controller.worker.v2.controller_worker [-] Task 'octavia.controller.worker.v2.tasks.database_tasks.MarkPoolPendingCreateInDB' (6958ab1a-352d-497c-9665-ba49bf00af9e) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python3.9/site-packages/taskflow/listeners/logging.py:178ESC[00m [..] 2024-05-27 08:18:01.182 17 DEBUG octavia.controller.worker.v2.tasks.database_tasks [None req-7e29e1ad-6e1b-442d-95da-8a6c3d4ffc9c - 8784eb28c4d1436fb0be9cfc8456d61b - - - -] Mark ACTIVE in DB for pool id: aa9cede4-4737-4f16-8d9d-4b7ecb6c0d5e execute /usr/lib/python3.9/site-packages/octavia/controller/worker/v2/tasks/database_tasks.py:2367ESC[00m 2024-05-27 08:18:01.195 17 DEBUG octavia.controller.worker.v2.controller_worker [-] Task 'octavia.controller.worker.v2.tasks.database_tasks.MarkPoolActiveInDB' (6b894a0c-14d1-446f-aafa-e98486905bf8) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python3.9/site-packages/taskflow/listeners/logging.py:178ESC[00m 2024-05-27 08:18:01.198 17 DEBUG octavia.controller.worker.v2.controller_worker [-] Task 'octavia.controller.worker.v2.tasks.database_tasks.MarkLBAndListenersActiveInDB' (1f8b4851-0553-468a-8fce-4c9abd80b444) transitioned into state 'RUNNING' from state 'PENDING' _task_receiver /usr/lib/python3.9/site-packages/taskflow/listeners/logging.py:190ESC[00m 2024-05-27 08:18:01.200 17 DEBUG octavia.controller.worker.v2.tasks.database_tasks [None req-26b91571-9e1f-47f4-9504-2de93a4b536e - 8784eb28c4d1436fb0be9cfc8456d61b - - - -] Mark ACTIVE in DB for load balancer id: 641888f9-254e-4640-84de-c450582f9e46 and updating status for listener ids: 3be60ced-2bc6-44ae-bb62-579ce205a7d9 execute /usr/lib/python3.9/site-packages/octavia/controller/worker/v2/tasks/database_tasks.py:1256ESC[00m 2024-05-27 08:18:01.231 17 DEBUG octavia.controller.worker.v2.controller_worker [-] Task 'octavia.controller.worker.v2.tasks.database_tasks.MarkLBAndListenersActiveInDB' (1f8b4851-0553-468a-8fce-4c9abd80b444) transitioned into state 'SUCCESS' from state 'RUNNING' with result 'None' _task_receiver /usr/lib/python3.9/s
Basically the trace shows that the object was committed to the DB, but the get failed (the object cannot be found in the DB)
The impact is that some tempest tests may fail with an error:
<Resource> <UUID> not found
during a POST
it may be related to https://github.com/openstack-k8s-operators/cinder-operator/pull/392/files#diff-56d9e8f34aa2d15d13a4dd59de0140ba15cb3f1e2ddded2f45b48409a3ea34c6
