The following error is randomly observed while pushing certain images to the internal registry, while having the new S3 storage backend in place:

"error resolving upload: s3aws: Path not found: /docker/registry/v2/repositories/cpet-image-builder/ubi8-git/_uploads/973f6caf-46d9-4a24-93c7-9b8d18e4724d/data"

In the image-registry pod logs, the following error is observed:

time="2024-06-11T07:50:47.142434548Z" level=error msg="error resolving upload: s3aws: Path not found: /docker/registry/v2/repositories/cpet-image-builder/ubi8-git/_uploads/973f6caf-46d9-4a24-93c7-9b8d18e4724d/data" go.version="go1.19.13 X:strictfipsruntime" http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=d8f31bae-51c9-42ed-b787-e8acba109965 http.request.method=PUT http.request.remoteaddr="10.128.3.210:36788" http.request.uri="/v2/cpet-image-builder/ubi8-git/blobs/uploads/973f6caf-46d9-4a24-93c7-9b8d18e4724d?_state=Z_gvDRcc8qFSFA6nh5gNi0CKf7sk00oRgJsXvlFQdYp7Ik5hbWUiOiJjcGV0LWltYWdlLWJ1aWxkZXIvdWJpOC1naXQiLCJVVUlEIjoiOTczZjZjYWYtNDZkOS00YTI0LTkzYzctOWI4ZDE4ZTQ3MjRkIiwiT2Zmc2V0Ijo1MjkzOTU5MCwiU3RhcnRlZEF0IjoiMjAyNC0wNi0xMVQwNzo1MDo0MFoifQ%3D%3D&digest=sha256%3Aad66da9c7bb2643ee8bfadf6f34effb1dd8908c3031ba3393206e751beb1407b" http.request.useragent=Buildah/1.33.7 openshift.auth.user="system:serviceaccount:cpet-image-builder:pipeline" vars.name=cpet-image-builder/ubi8-git vars.uuid=973f6caf-46d9-4a24-93c7-9b8d18e4724d

4.14.z

Randomly, no consistent pattern

The image write returns an "error resolving upload: s3aws: Path not found" error, in random scenarios.

For the image write to success every time

- Customer-provided logs that the `s3aws.Writer` function calls the same `/docker/registry/v2/repositories/cog-tekton-generiek/cog-base-devspaces/_uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f/data`, but returning different `true` / `false` outputs
- Huawei S3 storage used, and internal customer S3 storage team confirmed there is no current limitation in place for the chunk size and total uploaded size of stream.
- Image-registry service sessionAffinity: clientIP 
- The customer's environment initially had 3 registry replicas, but they replicated the issue even when scaling down the image-registry replicas to 1.
- 

~~~
$ grep -r bf54079a312f  0020-ml01-image-registry.log | grep s3aws.Writer -A1
time="2024-07-01T09:29:14.031854297Z" level=debug msg="s3aws.Writer(\"/docker/registry/v2/repositories/cog-tekton-generiek/cog-base-devspaces/_uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f/data\", false)" go.version="go1.19.13 X:strictfipsruntime" http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=89ac9878-e070-49f3-82de-be705e512166 http.request.method=POST http.request.remoteaddr="10.128.5.61:38690" http.request.uri=/v2/cog-tekton-generiek/cog-base-devspaces/blobs/uploads/ http.request.useragent=Buildah/1.29.1 openshift.auth.user="system:serviceaccount:cog-tekton-generiek:tekton-sa-build" trace.duration=18.308178ms trace.file=/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=ab9fd935-8c22-423f-9ae3-29a5fae097d5 trace.line=142 vars.name=cog-tekton-generiek/cog-base-devspaces
time="2024-07-01T09:29:14.042154242Z" level=debug msg="s3aws.PutContent(\"/docker/registry/v2/repositories/cog-tekton-generiek/cog-base-devspaces/_uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f/hashstates/sha256/0\")" go.version="go1.19.13 X:strictfipsruntime" http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=89ac9878-e070-49f3-82de-be705e512166 http.request.method=POST http.request.remoteaddr="10.128.5.61:38690" http.request.uri=/v2/cog-tekton-generiek/cog-base-devspaces/blobs/uploads/ http.request.useragent=Buildah/1.29.1 openshift.auth.user="system:serviceaccount:cog-tekton-generiek:tekton-sa-build" trace.duration=10.216999ms trace.file=/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=d97a6952-f613-4b83-8df6-5f7430ec14ee trace.line=110 vars.name=cog-tekton-generiek/cog-base-devspaces
--
time="2024-07-01T09:29:14.148041107Z" level=debug msg="s3aws.Writer(\"/docker/registry/v2/repositories/cog-tekton-generiek/cog-base-devspaces/_uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f/data\", true)" go.version="go1.19.13 X:strictfipsruntime" http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=5495ab20-b49a-4dc1-96a4-3bfc9887094e http.request.method=PATCH http.request.remoteaddr="10.128.5.61:38726" http.request.uri="/v2/cog-tekton-generiek/cog-base-devspaces/blobs/uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f?_state=yxqltEIAQi3mKAuOTKOhSUNah4bOKdsVd5b44RqsmS97Ik5hbWUiOiJjb2ctdGVrdG9uLWdlbmVyaWVrL2NvZy1iYXNlLWRldnNwYWNlcyIsIlVVSUQiOiI5ZDJjNmRiMy00MTRhLTQ0YmYtOWNiNi1iZjU0MDc5YTMxMmYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDctMDFUMDk6Mjk6MTMuOTg3Mzk0NzQ2WiJ9" http.request.useragent=Buildah/1.29.1 openshift.auth.user="system:serviceaccount:cog-tekton-generiek:tekton-sa-build" trace.duration=19.031601ms trace.file=/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=8b223110-c91c-4ee3-8563-50373ea74c8a trace.line=142 vars.name=cog-tekton-generiek/cog-base-devspaces vars.uuid=9d2c6db3-414a-44bf-9cb6-bf54079a312f
time="2024-07-01T09:29:16.565474885Z" level=debug msg="s3aws.PutContent(\"/docker/registry/v2/repositories/cog-tekton-generiek/cog-base-devspaces/_uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f/hashstates/sha256/57452937\")" go.version="go1.19.13 X:strictfipsruntime" http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=5495ab20-b49a-4dc1-96a4-3bfc9887094e http.request.method=PATCH http.request.remoteaddr="10.128.5.61:38726" http.request.uri="/v2/cog-tekton-generiek/cog-base-devspaces/blobs/uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f?_state=yxqltEIAQi3mKAuOTKOhSUNah4bOKdsVd5b44RqsmS97Ik5hbWUiOiJjb2ctdGVrdG9uLWdlbmVyaWVrL2NvZy1iYXNlLWRldnNwYWNlcyIsIlVVSUQiOiI5ZDJjNmRiMy00MTRhLTQ0YmYtOWNiNi1iZjU0MDc5YTMxMmYiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDctMDFUMDk6Mjk6MTMuOTg3Mzk0NzQ2WiJ9" http.request.useragent=Buildah/1.29.1 openshift.auth.user="system:serviceaccount:cog-tekton-generiek:tekton-sa-build" trace.duration=9.442256ms trace.file=/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).PutContent" trace.id=d5eea3c0-00dc-4512-96e9-7bfc87f0b3df trace.line=110 vars.name=cog-tekton-generiek/cog-base-devspaces vars.uuid=9d2c6db3-414a-44bf-9cb6-bf54079a312f
--
time="2024-07-01T09:29:17.095452692Z" level=debug msg="s3aws.Writer(\"/docker/registry/v2/repositories/cog-tekton-generiek/cog-base-devspaces/_uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f/data\", true)" go.version="go1.19.13 X:strictfipsruntime" http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=a56814b5-e77b-4156-ae5f-60f18dc44c07 http.request.method=PUT http.request.remoteaddr="10.128.5.61:39122" http.request.uri="/v2/cog-tekton-generiek/cog-base-devspaces/blobs/uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f?_state=Zf90Mj-KIxLTiwsiypRdqs1ePVFyoXEnRuBbZi75WKZ7Ik5hbWUiOiJjb2ctdGVrdG9uLWdlbmVyaWVrL2NvZy1iYXNlLWRldnNwYWNlcyIsIlVVSUQiOiI5ZDJjNmRiMy00MTRhLTQ0YmYtOWNiNi1iZjU0MDc5YTMxMmYiLCJPZmZzZXQiOjU3NDUyOTM3LCJTdGFydGVkQXQiOiIyMDI0LTA3LTAxVDA5OjI5OjEzWiJ9&digest=sha256%3A0f6a5e636adda3abb12294895b25b3b44650810255ba4b40d9207937004dcf93" http.request.useragent=Buildah/1.29.1 openshift.auth.user="system:serviceaccount:cog-tekton-generiek:tekton-sa-build" trace.duration=7.120776ms trace.file=/go/src/github.com/openshift/image-registry/vendor/github.com/docker/distribution/registry/storage/driver/base/base.go trace.func="github.com/docker/distribution/registry/storage/driver/base.(*Base).Writer" trace.id=630b1ea7-b04b-4d5e-8d4b-c7c92198791a trace.line=142 vars.name=cog-tekton-generiek/cog-base-devspaces vars.uuid=9d2c6db3-414a-44bf-9cb6-bf54079a312f
time="2024-07-01T09:29:17.09552467Z" level=error msg="error resolving upload: s3aws: Path not found: /docker/registry/v2/repositories/cog-tekton-generiek/cog-base-devspaces/_uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f/data" go.version="go1.19.13 X:strictfipsruntime" http.request.contenttype=application/octet-stream http.request.host="image-registry.openshift-image-registry.svc:5000" http.request.id=a56814b5-e77b-4156-ae5f-60f18dc44c07 http.request.method=PUT http.request.remoteaddr="10.128.5.61:39122" http.request.uri="/v2/cog-tekton-generiek/cog-base-devspaces/blobs/uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f?_state=Zf90Mj-KIxLTiwsiypRdqs1ePVFyoXEnRuBbZi75WKZ7Ik5hbWUiOiJjb2ctdGVrdG9uLWdlbmVyaWVrL2NvZy1iYXNlLWRldnNwYWNlcyIsIlVVSUQiOiI5ZDJjNmRiMy00MTRhLTQ0YmYtOWNiNi1iZjU0MDc5YTMxMmYiLCJPZmZzZXQiOjU3NDUyOTM3LCJTdGFydGVkQXQiOiIyMDI0LTA3LTAxVDA5OjI5OjEzWiJ9&digest=sha256%3A0f6a5e636adda3abb12294895b25b3b44650810255ba4b40d9207937004dcf93" http.request.useragent=Buildah/1.29.1 openshift.auth.user="system:serviceaccount:cog-tekton-generiek:tekton-sa-build" vars.name=cog-tekton-generiek/cog-base-devspaces vars.uuid=9d2c6db3-414a-44bf-9cb6-bf54079a312f
~~~ 

s3aws.Writer(\"/docker/registry/v2/repositories/cog-tekton-generiek/cog-base-devspaces/_uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f/data\", false
s3aws.Writer(\"/docker/registry/v2/repositories/cog-tekton-generiek/cog-base-devspaces/_uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f/data\", true
s3aws.Writer(\"/docker/registry/v2/repositories/cog-tekton-generiek/cog-base-devspaces/_uploads/9d2c6db3-414a-44bf-9cb6-bf54079a312f/data\", true