-
Bug
-
Resolution: Done
-
Major
-
None
-
None
-
None
Deletion of tempest operator is constantly timing out in In ci-framework-baremetal-static-node-rhel9-va-hci-update-trunk-patches job as in attached log. We run smoke and sanity tests only in the job and after tempest completes we run cleanup to remove all tempest related artifacts in preparation for second tempest run after update.
2024-09-09 10:26:49,163 p=26219 u=zuul n=ansible | TASK [test_operator : Get test results from all test pods (Success / Fail) kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit) }}, context={{ cifmw_openshift_context | default(omit) }}, namespace={{ cifmw_test_operator_namespace }}, kind=Pod, label_selectors=['instanceName={{ test_operator_job_name }}']] *** 29192024-09-09 10:26:49,163 p=26219 u=zuul n=ansible | Monday 09 September 2024 10:26:49 -0400 (0:00:04.885) 1:25:53.862 ****** 29202024-09-09 10:26:50,362 p=26219 u=zuul n=ansible | ok: [localhost] 29212024-09-09 10:26:50,398 p=26219 u=zuul n=ansible | TASK [test_operator : Get status from test pods pod_status={{ 2922 test_pod_results.resources | 2923 map(attribute='status.phase') | 2924 list | unique 2925}}] *** 29262024-09-09 10:26:50,398 p=26219 u=zuul n=ansible | Monday 09 September 2024 10:26:50 -0400 (0:00:01.234) 1:25:55.097 ****** 29272024-09-09 10:26:50,448 p=26219 u=zuul n=ansible | ok: [localhost] 29282024-09-09 10:26:50,473 p=26219 u=zuul n=ansible | TASK [test_operator : Check whether test pods finished successfully successful_execution={{ 2929 pod_status | length == 1 and 2930 pod_status | first == 'Succeeded' 2931}}] *** 29322024-09-09 10:26:50,473 p=26219 u=zuul n=ansible | Monday 09 September 2024 10:26:50 -0400 (0:00:00.074) 1:25:55.171 ****** 29332024-09-09 10:26:50,520 p=26219 u=zuul n=ansible | ok: [localhost] 29342024-09-09 10:26:50,544 p=26219 u=zuul n=ansible | TASK [test_operator : Fail fast if a pod did not succeed - tempest that=successful_execution] *** 29352024-09-09 10:26:50,544 p=26219 u=zuul n=ansible | Monday 09 September 2024 10:26:50 -0400 (0:00:00.071) 1:25:55.243 ****** 29362024-09-09 10:26:50,574 p=26219 u=zuul n=ansible | skipping: [localhost] 29372024-09-09 10:26:50,597 p=26219 u=zuul n=ansible | TASK [test_operator : Save result - tempest test_operator_results={{ 2938 test_operator_results | default({}) | 2939 combine({run_test_fw: successful_execution}) 2940}}] *** 29412024-09-09 10:26:50,597 p=26219 u=zuul n=ansible | Monday 09 September 2024 10:26:50 -0400 (0:00:00.052) 1:25:55.295 ****** 29422024-09-09 10:26:50,640 p=26219 u=zuul n=ansible | ok: [localhost] 29432024-09-09 10:26:50,660 p=26219 u=zuul n=ansible | TASK [test_operator : Delete tempest kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, kind={{ test_operator_kind_name }}, state=absent, api_version=test.openstack.org/v1beta1, name={{ test_operator_job_name }}, namespace={{ cifmw_test_operator_namespace }}, wait=True] *** 29442024-09-09 10:26:50,661 p=26219 u=zuul n=ansible | Monday 09 September 2024 10:26:50 -0400 (0:00:00.063) 1:25:55.359 ****** 29452024-09-09 10:28:52,552 p=26219 u=zuul n=ansible | fatal: [localhost]: FAILED! => changed=true 2946 duration: 120 2947 method: delete 2948 msg: '"Tempest" "tempest-tests": Timed out waiting on resource' 2949 result: 2950 apiVersion: test.openstack.org/v1beta1 2951 kind: Tempest 2952 metadata: 2953 creationTimestamp: '2024-09-09T14:08:08Z' 2954 deletionGracePeriodSeconds: 0 2955 deletionTimestamp: '2024-09-09T14:26:52Z' 2956 finalizers: 2957 - openstack.org/tempest 2958 generation: 2 2959 managedFields: 2960 - apiVersion: test.openstack.org/v1beta1 2961 fieldsType: FieldsV1 2962 fieldsV1: 2963 f:spec: 2964 .: {} 2965 f:SELinuxLevel: {} 2966 f:SSHKeySecretName: {} 2967 f:backoffLimit: {} 2968 f:containerImage: {} 2969 f:debug: {} 2970 f:networkAttachments: {} 2971 f:openStackConfigMap: {} 2972 f:openStackConfigSecret: {} 2973 f:parallel: {} 2974 f:storageClass: {} 2975 f:tempestRun:
PRs:
https://github.com/openstack-k8s-operators/ci-framework/pull/2305https://github.com/openstack-k8s-operators/ci-framework/pull/2346(cherry-pick for 18.0.0-proposed)https://github.com/openstack-k8s-operators/ci-framework/pull/2364https://github.com/openstack-k8s-operators/ci-framework/pull/2379(cherry-pick of #2364 for 18.0.0-proposed)https://github.com/openstack-k8s-operators/ci-framework/pull/2412https://github.com/openstack-k8s-operators/ci-framework/pull/2426
- relates to
-
OSPRH-9865 Add the `wait_timeout` parameter to all cleanup tasks for the test-operator
-
- Closed
-