When configure registry to use Ceph RGW storage with ODF, failed push image to internal registry.

    4.18.0-0.nightly-2024-12-05-191355

    Always

    1.Step a upi on vsphere cluster with registry storage to use Ceph RGW storage with ODF     
    2. Cannot push image to internal registry
Copying blob sha256:fa6b8e6892626db0ab9efe931e6141edb1d001d89e43c0e697fc3c7d8c9fdab8
Copying blob sha256:0a73aee7e570194d6ace6157d6693b5439c872acf23a01d32c79489f3192264f
Copying blob sha256:d1626b5b9f7a59add02f7c14fb670e63a729a8b70a53b95f56bf7f5ee8dc12d8
Copying blob sha256:88242225f03c9724a2e52d6a132db17a62e4ef419cdcbdf6fa7c520e53fb952e
Copying blob sha256:a7b88b8628085816a0ddde3ecab66775b8e5d314b719efc25c1920c9ee34ca65
Copying blob sha256:1670f2f6fa91864a695f6c642bdcbf3eaee792a9dd996e3b7ba6c3ec0bec6adc
Warning: Push failed, retrying in 5s ...
Registry server Address: 
Registry server User Name: <token>
Registry server Email: 
Registry server Password: <<non-empty>>
error: build error: Failed to push image: writing blob: initiating layer upload to /v2/wewang/ruby-ex/blobs/uploads/ in image-registry.openshift-image-registry.svc:5000: received unexpected HTTP status: 500 Internal Server Error

     3. # oc get config.image/cluster -ojsonpath='{.spec.storage.s3}' | jq .
{
  "bucket": "rgw-bucket-for-registry-5e3b253c-0ab2-4a79-b03f-6a8d4397ba2e",
  "encrypt": true,
  "region": "us-east-1",
  "regionEndpoint": "https://cephrgw-openshift-storage.apps.wewang-odf.qe.devcluster.openshift.com",
  "trustedCA": {
    "name": "odf-ca-bundle"
  },
  "virtualHostedStyle": false
}
    4. Registry pod log
time="2024-12-06T06:57:08.334687294Z" level=info msg="authorized request" go.version="go1.22.9 (Red Hat 1.22.9-2.el9_5) X:strictfipsruntime" http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=24e19d09-4c74-4ea9-a9e9-11596a86411e http.request.method=POST http.request.remoteaddr="10.129.2.32:59992" http.request.uri=/v2/wewang0/ruby-ex/blobs/uploads/ http.request.useragent="containers/5.30.1 (github.com/containers/image)" openshift.auth.user="system:serviceaccount:wewang0:builder" openshift.auth.userid=899c7ac2-8e45-4775-8311-c9e0dc0f74e0 vars.name=wewang0/ruby-ex
time="2024-12-06T06:57:08.381518217Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: InvalidArgument: \n\tstatus code: 400, request id: tx0000054ea033ecb4f8ae8-006752a044-39dc-ocs-storagecluster-cephobjectstore, host id: " err.message="unknown error" go.version="go1.22.9 (Red Hat 1.22.9-2.el9_5) X:strictfipsruntime" http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=24e19d09-4c74-4ea9-a9e9-11596a86411e http.request.method=POST http.request.remoteaddr="10.129.2.32:59992" http.request.uri=/v2/wewang0/ruby-ex/blobs/uploads/ http.request.useragent="containers/5.30.1 (github.com/containers/image)" http.response.contenttype=application/json http.response.duration=57.99477ms http.response.status=500 http.response.written=123 openshift.auth.user="system:serviceaccount:wewang0:builder" openshift.auth.userid=899c7ac2-8e45-4775-8311-c9e0dc0f74e0 vars.name=wewang0/ruby-ex
time="2024-12-06T06:57:08.381768843Z" level=info msg=response go.version="go1.22.9 (Red Hat 1.22.9-2.el9_5) X:strictfipsruntime" http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=86469d5a-5d9f-4655-b761-b27b6cb60b50 http.request.method=POST http.request.remoteaddr="10.129.2.32:59992" http.request.uri=/v2/wewang0/ruby-ex/blobs/uploads/ http.request.useragent="containers/5.30.1 (github.com/containers/image)" http.response.contenttype=application/json http.response.duration=58.285553ms http.response.status=500 http.response.written=123

    Cannot push image to internal registry

    Should push images to internal registry successfully