[3034290471] Upstream Reporter: Dusty Mabe
Upstream issue status: Closed
Upstream description:

In the CoreOS team we found that cached layers that were getting re-used for later builds violated the principle of least surprise.

Here's a simple reproducer with example output:

[dustymabe@media podman-build-cache-bug]$ echo 'abc' > file
[dustymabe@media podman-build-cache-bug]$ cat Containerfile 
FROM quay.io/fedora/fedora:42
RUN <<EOF
    curl http://0.0.0.0:8000/file | tee /file
EOF
[dustymabe@media podman-build-cache-bug]$ podman build -t test .
STEP 1/2: FROM quay.io/fedora/fedora:42
STEP 2/2: RUN <<EOF (curl http://0.0.0.0:8000/file | tee /file)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     4  100     4    0     0   1291      0 --:--:-- --:--:-- --:--:--  1333
abc
COMMIT test
--> e318ca889b77
Successfully tagged localhost/test:latest
e318ca889b77318233d8a3df8ede75e1d506315d0da9951b1b412102cf6548ba
[dustymabe@media podman-build-cache-bug]$ 
[dustymabe@media podman-build-cache-bug]$ echo 'def' > file
[dustymabe@media podman-build-cache-bug]$ podman build --cache-ttl=.01s -t test .
STEP 1/2: FROM quay.io/fedora/fedora:42
STEP 2/2: RUN <<EOF (curl http://0.0.0.0:8000/file | tee /file)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     4  100     4    0     0   3241      0 --:--:-- --:--:-- --:--:--  4000
def
COMMIT test
--> a8e31050715f
Successfully tagged localhost/test:latest
a8e31050715fdff1df846ce087a11fb93e7129026324675670b4db14fb1b006d
[dustymabe@media podman-build-cache-bug]$ echo 'ghi' > file
[dustymabe@media podman-build-cache-bug]$ podman build -t test .
STEP 1/2: FROM quay.io/fedora/fedora:42
STEP 2/2: RUN <<EOF (curl http://0.0.0.0:8000/file | tee /file)
--> Using cache e318ca889b77318233d8a3df8ede75e1d506315d0da9951b1b412102cf6548ba
COMMIT test
--> e318ca889b77
Successfully tagged localhost/test:latest
e318ca889b77318233d8a3df8ede75e1d506315d0da9951b1b412102cf6548ba
[dustymabe@media podman-build-cache-bug]$ 
[dustymabe@media podman-build-cache-bug]$ podman run -it --rm localhost/test:latest cat /file
abc

There is a webserver on the local system hosting the file at http://0.0.0.0:8000/file so the RUN line never changes (i.e. is eligible for searching for previous cache layers).

The first run does a build with abc in it. The second run does a build, this time with --cache-ttl=0.1s to effectively disable searching for cached layers, so it gets def. The last build re-allows cached layers to be used and you can see that it chose cached layer e318ca889b77, which was from the very first build. Not the most recent build.

I believe what most people would expect is that the original cached layer wouldn't be used because there is a newer layer that matches.

[dustymabe@media podman-build-cache-bug]$ rpm -q podman
podman-5.4.1-1.fc41.x86_64