Uploaded image for project: 'Red Hat OpenStack Services on OpenShift'
  1. Red Hat OpenStack Services on OpenShift
  2. OSPRH-12704

glance image upload fails due to swift timeout

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Undefined Undefined
    • None
    • None
    • swift-operator
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • None
    • Moderate

      I am not sure whether this is a bug or misconfiguration.

      This component job failed [1] because the tempest pod was unable to upload an image to glance [2]:

       

      ++ openstack image show 11111111-1111-1111-1111-111111111111 -f value -c status Current status: importing. Waiting for image to become active... 

       

       

      According to the glance logs [3], swift failed uploading the image:

       

      2024-12-28 02:38:24.871 24 WARNING glance.async_.taskflow_executor [-] Task 'api_image_import-default_backend-ImportToStore-a8dd8ec7-d6a4-48f3-b4e5-76d94a7bbc68' (9946d8f6-70b5-49a7-b0f2-b94381351bb1) transitioned into state 'FAILURE' from state 'RUNNING'
      6 predecessors (most recent first):
        Flow 'api_image_import-default_backend'
        |__Flow 'api_image_import'
           |__Atom 'api_image_import-ConfigureStaging-a8dd8ec7-d6a4-48f3-b4e5-76d94a7bbc68' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': 'file:///var/lib/glance/os_glance_staging_store/11111111-1111-1111-1111-111111111111'}
              |__Atom 'api_image_import-ImageLock-a8dd8ec7-d6a4-48f3-b4e5-76d94a7bbc68' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': None}
                 |__Atom 'api_image_import_retry' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {}, 'provides': [(None, {})]}
                    |__Flow 'api_image_import': glance_store.exceptions.BackendException: Failed to add object to Swift.
      Got error from Swift: put_object('glance', '11111111-1111-1111-1111-111111111111-00001', ...) failure and no ability to reset contents for reupload..

       

       

      Swift proxy logs show that the request failed due to 10 seconds timeout [4]:

      proxy-server: ERROR with Object server swift-storage-0.swift-storage.openstack.svc:6200/pv re: Trying to get final status of PUT to /v1/AUTH_e2fb08c0346b4ae7b1b0d55112dfde71/glance/11111111-1111-1111-1111-111111111111-00001: Timeout (10.0s) (txn: txf799e20d762d40818e573-00676f648f) (client_ip: 10.217.0.204)
      proxy-server: Object PUT returning 503 for [503] (txn: txf799e20d762d40818e573-00676f648f) (client_ip: 10.217.0.204)

       

      But swift object-server logs show that uploading the image succeeded after 16.8 seconds [5]:

      object-server: 10.217.0.189 - - [28/Dec/2024:02:38:24 +0000] "PUT /pv/361/AUTH_e2fb08c0346b4ae7b1b0d55112dfde71/glance/11111111-1111-1111-1111-111111111111-00001" 201 - "PUT http://swift-internal.openstack.svc:8080/v1/AUTH_e2fb08c0346b4ae7b1b0d55112dfde71/glance/11111111-1111-1111-1111-111111111111-00001" "txf799e20d762d40818e573-00676f648f" "proxy-server 25" 16.8393 "-" 16 0 

       

      [1] https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/zuul/t/components-integration/build/26b8b029e8304197b66f0c477b71cfa8

      [2] https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/logs/openstack-component-network/gitlab.cee.redhat.com/ci-framework/ci-framework-jobs/main/component-network-edpm-rhel9-rhoso18.0-crc-fips/26b8b02/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/tempest-tests-tempest-multithread-workflow-step-0-6cl9h/logs/tempest-tests-tempest-tests-runner.log

      [3] https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/logs/openstack-component-network/gitlab.cee.redhat.com/ci-framework/ci-framework-jobs/main/component-network-edpm-rhel9-rhoso18.0-crc-fips/26b8b02/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/glance-default-external-api-0/logs/glance-log.log

      [4] https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/logs/openstack-component-network/gitlab.cee.redhat.com/ci-framework/ci-framework-jobs/main/component-network-edpm-rhel9-rhoso18.0-crc-fips/26b8b02/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/swift-proxy-6df76587bc-gcdw7/logs/proxy-server.log

      [5] https://sf.apps.int.gpc.ocp-hub.prod.psi.redhat.com/logs/openstack-component-network/gitlab.cee.redhat.com/ci-framework/ci-framework-jobs/main/component-network-edpm-rhel9-rhoso18.0-crc-fips/26b8b02/controller/ci-framework-data/logs/openstack-k8s-operators-openstack-must-gather/namespaces/openstack/pods/swift-storage-0/logs/object-server.log

              cschwede@redhat.com Christian Schwede
              eolivare Eduardo Olivares Toledo
              rhos-dfg-storage-squad-swift
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated: