[3734137365] Upstream Reporter: Yauhen
Upstream issue status: Open
Upstream description:

Original issue: https://github.com/containers/podman/issues/27197

I can't re-open issue and seems no notifications, so I create another one.

? podman --version
podman version 5.7.0

? podman info
host:
  arch: amd64
  buildahVersion: 1.42.1
...

and issue still here. I can't re-open issue b/o permissions.

REPRODUCING:

1. put the repro.sh script into an empty directory
2. adjust registry to some really existing registry you have read/write access
3. run script - it fails

repro.sh:

#!/usr/bin/env bash

# We need write access to this registry to reproduce
REGISTRY=myregistry.corp.local/bug

echo "Generate containers for multistage"
for name in one two three; do
    cat <<EOF > "Containerfile.$name"
FROM debian:trixie-20251117
EOF
    podman build -f "Containerfile.$name" -t "$REGISTRY/fake-image-$name:latest" .
    podman push "$REGISTRY/fake-image-$name:latest"
done

echo "List images"
podman image ls "$REGISTRY/fake-image*"

echo "Change containers locally: add file generation"
for name in one two three; do
    cat <<EOF >> "Containerfile.$name"
RUN echo "name $name" > /$name
EOF
    echo "build, but not push"
    podman build -f "Containerfile.$name" -t "$REGISTRY/fake-image-$name:latest" .
done

echo "List images"
podman image ls "$REGISTRY/fake-image*"

echo "Generate bug container. Expected FROM uses locally generated images"
cat <<EOF > Containerfile.bug
ARG VERSION=latest
FROM $REGISTRY/fake-image-one:${VERSION} as one
ARG VERSION=latest
FROM $REGISTRY/fake-image-two:${VERSION} as two
ARG VERSION=latest
FROM $REGISTRY/fake-image-three:${VERSION} as three

FROM debian:trixie-20251117
COPY --from=one /one /
COPY --from=two /two /
COPY --from=three /three /
RUN env|sort && cat /one /two
EOF

echo "Expect not to download anything"
podman build -f Containerfile.bug .

echo "List images finally"
podman image ls "$REGISTRY/fake-image*"

And here is my output:

Generate containers for multistage
STEP 1/1: FROM debian:trixie-20251117
COMMIT myregistry.corp.local/bug/fake-image-one:latest
--> 53f0b37b86c4
Successfully tagged myregistry.corp.local/bug/fake-image-one:latest
Successfully tagged myregistry.corp.local/bug/fake-image-three:latest
Successfully tagged myregistry.corp.local/bug/fake-image-two:latest
Successfully tagged docker.io/library/debian:trixie-20251117
53f0b37b86c4420b4f5e38bcc55fc08598c321dad8734a6823f36e355b91af42
Getting image source signatures
Copying blob af8838a67f3b skipped: already exists  
Copying config 53f0b37b86 done   | 
Writing manifest to image destination
STEP 1/1: FROM debian:trixie-20251117
COMMIT myregistry.corp.local/bug/fake-image-two:latest
--> 53f0b37b86c4
Successfully tagged myregistry.corp.local/bug/fake-image-two:latest
Successfully tagged myregistry.corp.local/bug/fake-image-one:latest
Successfully tagged myregistry.corp.local/bug/fake-image-three:latest
Successfully tagged docker.io/library/debian:trixie-20251117
53f0b37b86c4420b4f5e38bcc55fc08598c321dad8734a6823f36e355b91af42
Getting image source signatures
Copying blob af8838a67f3b skipped: already exists  
Copying config 53f0b37b86 done   | 
Writing manifest to image destination
STEP 1/1: FROM debian:trixie-20251117
COMMIT myregistry.corp.local/bug/fake-image-three:latest
--> 53f0b37b86c4
Successfully tagged myregistry.corp.local/bug/fake-image-three:latest
Successfully tagged myregistry.corp.local/bug/fake-image-two:latest
Successfully tagged myregistry.corp.local/bug/fake-image-one:latest
Successfully tagged docker.io/library/debian:trixie-20251117
53f0b37b86c4420b4f5e38bcc55fc08598c321dad8734a6823f36e355b91af42
Getting image source signatures
Copying blob af8838a67f3b skipped: already exists  
Copying config 53f0b37b86 done   | 
Writing manifest to image destination
List images
REPOSITORY                                                              TAG              IMAGE ID      CREATED      SIZE
myregistry.corp.local/bug/fake-image-three  latest           53f0b37b86c4  2 weeks ago  124 MB
myregistry.corp.local/bug/fake-image-two    latest           53f0b37b86c4  2 weeks ago  124 MB
myregistry.corp.local/bug/fake-image-one    latest           53f0b37b86c4  2 weeks ago  124 MB
docker.io/library/debian                                                trixie-20251117  53f0b37b86c4  2 weeks ago  124 MB
Change containers locally: add file generation
build, but not push
STEP 1/2: FROM debian:trixie-20251117
STEP 2/2: RUN echo "name one" > /one
--> Using cache f8928fb716d6f3dc7c8c371ff71f6fbc4cbfc85dd1b744dfd96a5a6ba6f0a09b
COMMIT myregistry.corp.local/bug/fake-image-one:latest
--> f8928fb716d6
Successfully tagged myregistry.corp.local/bug/fake-image-one:latest
f8928fb716d6f3dc7c8c371ff71f6fbc4cbfc85dd1b744dfd96a5a6ba6f0a09b
build, but not push
STEP 1/2: FROM debian:trixie-20251117
STEP 2/2: RUN echo "name two" > /two
--> Using cache 1f59ef0442b06b6690fe63ca40845f7dee85350e7591311cb79e330a70783114
COMMIT myregistry.corp.local/bug/fake-image-two:latest
--> 1f59ef0442b0
Successfully tagged myregistry.corp.local/bug/fake-image-two:latest
1f59ef0442b06b6690fe63ca40845f7dee85350e7591311cb79e330a70783114
build, but not push
STEP 1/2: FROM debian:trixie-20251117
STEP 2/2: RUN echo "name three" > /three
--> Using cache bbddb80ff26120f71d63a9d05765b3cba57066239f4872038ab5f90bba820db7
COMMIT myregistry.corp.local/bug/fake-image-three:latest
--> bbddb80ff261
Successfully tagged myregistry.corp.local/bug/fake-image-three:latest
bbddb80ff26120f71d63a9d05765b3cba57066239f4872038ab5f90bba820db7
List images
REPOSITORY                                                              TAG         IMAGE ID      CREATED        SIZE
myregistry.corp.local/bug/fake-image-three  latest      bbddb80ff261  3 minutes ago  124 MB
myregistry.corp.local/bug/fake-image-two    latest      1f59ef0442b0  3 minutes ago  124 MB
myregistry.corp.local/bug/fake-image-one    latest      f8928fb716d6  3 minutes ago  124 MB
Generate bug container. Expected from uses locally generated images
Expect not to download anything
[1/4] STEP 1/2: FROM myregistry.corp.local/bug/fake-image-one:latest AS one
Trying to pull myregistry.corp.local/bug/fake-image-one:latest...
Getting image source signatures
Copying blob af8838a67f3b skipped: already exists  
Copying config 53f0b37b86 done   | 
Writing manifest to image destination
[1/4] STEP 2/2: ARG VERSION=latest
--> Using cache 61351b606524c9f3eff94442a43db4118c00b16a7acd7be062acd92b233e2869
--> 61351b606524
[2/4] STEP 1/2: FROM myregistry.corp.local/bug/fake-image-two:latest AS two
Trying to pull myregistry.corp.local/bug/fake-image-two:latest...
Getting image source signatures
Copying blob af8838a67f3b skipped: already exists  
Copying config 53f0b37b86 done   | 
Writing manifest to image destination
[2/4] STEP 2/2: ARG VERSION=latest
--> Using cache 61351b606524c9f3eff94442a43db4118c00b16a7acd7be062acd92b233e2869
--> 61351b606524
[3/4] STEP 1/1: FROM myregistry.corp.local/bug/fake-image-three:latest AS three
Trying to pull myregistry.corp.local/bug/fake-image-three:latest...
Getting image source signatures
Copying blob af8838a67f3b skipped: already exists  
Copying config 53f0b37b86 done   | 
Writing manifest to image destination
--> 53f0b37b86c4
[4/4] STEP 1/5: FROM debian:trixie-20251117
[4/4] STEP 2/5: COPY --from=one /one /
Error: building at STEP "COPY --from=one /one /": checking on sources under "/home/sirex/.local/share/containers/storage/overlay/006f2bb8ac363a6ba4359a94299751239b5bdb43e6470cc2267f85ce11ab33a7/merged": copier: stat: "/one": no such file or directory
List images finally
REPOSITORY                                                              TAG              IMAGE ID      CREATED      SIZE
myregistry.corp.local/bug/fake-image-three  latest           53f0b37b86c4  2 weeks ago  124 MB
myregistry.corp.local/bug/fake-image-two    latest           53f0b37b86c4  2 weeks ago  124 MB
myregistry.corp.local/bug/fake-image-one    latest           53f0b37b86c4  2 weeks ago  124 MB
docker.io/library/debian                                                trixie-20251117  53f0b37b86c4  2 weeks ago  124 MB

Notice: we generated images and pushed them (id 53f0b37b86c4) Then changed locally and rebuilt (another uniq image id)

Tried to build container, all ids reverted back to 53f0b37b86c4 (images we pulled, local changes discarded).