Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-10069

Tempest "tempest-tests: Timed out waiting on resource" during cleanup

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • None
    • openstack-tempest
    • None
    • 7
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • ?
    • ?
    • None
    • Important

      Deletion of tempest operator is constantly timing out in In ci-framework-baremetal-static-node-rhel9-va-hci-update-trunk-patches job as in attached log. We  run smoke and sanity tests only in the job and after tempest completes we run cleanup to remove all tempest related artifacts in preparation for second tempest run after update.

      https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/build/94e963ea0cb7416580ef1888805d064d/log/logs/controller-0/ci-framework-data/logs/ansible-deploy-architecture.log 

      2024-09-09 10:26:49,163 p=26219 u=zuul n=ansible | TASK [test_operator : Get test results from all test pods (Success / Fail) kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit) }}, context={{ cifmw_openshift_context | default(omit) }}, namespace={{ cifmw_test_operator_namespace }}, kind=Pod, label_selectors=['instanceName={{ test_operator_job_name }}']] ***
      29192024-09-09 10:26:49,163 p=26219 u=zuul n=ansible | Monday 09 September 2024  10:26:49 -0400 (0:00:04.885)       1:25:53.862 ****** 
      29202024-09-09 10:26:50,362 p=26219 u=zuul n=ansible | ok: [localhost]
      29212024-09-09 10:26:50,398 p=26219 u=zuul n=ansible | TASK [test_operator : Get status from test pods pod_status={{
      2922  test_pod_results.resources |
      2923  map(attribute='status.phase') |
      2924  list | unique
      2925}}] ***
      29262024-09-09 10:26:50,398 p=26219 u=zuul n=ansible | Monday 09 September 2024  10:26:50 -0400 (0:00:01.234)       1:25:55.097 ****** 
      29272024-09-09 10:26:50,448 p=26219 u=zuul n=ansible | ok: [localhost]
      29282024-09-09 10:26:50,473 p=26219 u=zuul n=ansible | TASK [test_operator : Check whether test pods finished successfully successful_execution={{
      2929  pod_status | length == 1 and
      2930  pod_status | first == 'Succeeded'
      2931}}] ***
      29322024-09-09 10:26:50,473 p=26219 u=zuul n=ansible | Monday 09 September 2024  10:26:50 -0400 (0:00:00.074)       1:25:55.171 ****** 
      29332024-09-09 10:26:50,520 p=26219 u=zuul n=ansible | ok: [localhost]
      29342024-09-09 10:26:50,544 p=26219 u=zuul n=ansible | TASK [test_operator : Fail fast if a pod did not succeed - tempest that=successful_execution] ***
      29352024-09-09 10:26:50,544 p=26219 u=zuul n=ansible | Monday 09 September 2024  10:26:50 -0400 (0:00:00.071)       1:25:55.243 ****** 
      29362024-09-09 10:26:50,574 p=26219 u=zuul n=ansible | skipping: [localhost]
      29372024-09-09 10:26:50,597 p=26219 u=zuul n=ansible | TASK [test_operator : Save result - tempest test_operator_results={{
      2938    test_operator_results | default({}) |
      2939    combine({run_test_fw: successful_execution})
      2940}}] ***
      29412024-09-09 10:26:50,597 p=26219 u=zuul n=ansible | Monday 09 September 2024  10:26:50 -0400 (0:00:00.052)       1:25:55.295 ****** 
      29422024-09-09 10:26:50,640 p=26219 u=zuul n=ansible | ok: [localhost]
      29432024-09-09 10:26:50,660 p=26219 u=zuul n=ansible | TASK [test_operator : Delete tempest kubeconfig={{ cifmw_openshift_kubeconfig }}, api_key={{ cifmw_openshift_token | default(omit)}}, context={{ cifmw_openshift_context | default(omit)}}, kind={{ test_operator_kind_name }}, state=absent, api_version=test.openstack.org/v1beta1, name={{ test_operator_job_name }}, namespace={{ cifmw_test_operator_namespace }}, wait=True] ***
      29442024-09-09 10:26:50,661 p=26219 u=zuul n=ansible | Monday 09 September 2024  10:26:50 -0400 (0:00:00.063)       1:25:55.359 ****** 
      29452024-09-09 10:28:52,552 p=26219 u=zuul n=ansible | fatal: [localhost]: FAILED! => changed=true 
      2946  duration: 120
      2947  method: delete
      2948  msg: '"Tempest" "tempest-tests": Timed out waiting on resource'
      2949  result:
      2950    apiVersion: test.openstack.org/v1beta1
      2951    kind: Tempest
      2952    metadata:
      2953      creationTimestamp: '2024-09-09T14:08:08Z'
      2954      deletionGracePeriodSeconds: 0
      2955      deletionTimestamp: '2024-09-09T14:26:52Z'
      2956      finalizers:
      2957      - openstack.org/tempest
      2958      generation: 2
      2959      managedFields:
      2960      - apiVersion: test.openstack.org/v1beta1
      2961        fieldsType: FieldsV1
      2962        fieldsV1:
      2963          f:spec:
      2964            .: {}
      2965            f:SELinuxLevel: {}
      2966            f:SSHKeySecretName: {}
      2967            f:backoffLimit: {}
      2968            f:containerImage: {}
      2969            f:debug: {}
      2970            f:networkAttachments: {}
      2971            f:openStackConfigMap: {}
      2972            f:openStackConfigSecret: {}
      2973            f:parallel: {}
      2974            f:storageClass: {}
      2975            f:tempestRun:

      PRs:

            lpiwowar Lukáš Piwowarski
            mciecier@redhat.com Mikolaj Ciecierski
            rhos-tempest
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: