Description:

COMPOSER-1640 added the ability to embed containers into ostree images, such that containerised applications can be started without requiring to pull them over the network. This depends on embedded images behaving exactly as if they had just been `podman pull`ed.

However, at least some images referenced by digest get modified during embedding, with the result that the container engine tries to pull them once more over the network.

Reproducible: 100%

Reproducer:

1. Create a minimal blueprint with the Red Hat openssl image embedded and referenced by digest:

description = ""
version = "0.0.1"
modules = []
groups = []

[[containers]]
source = "registry.access.redhat.com/ubi8/openssl@sha256:9e743d947be073808f7f1750a791a3dbd81e694e37161e8c6c6057c2c342d671"

2. Build an edge-commit from that blueprint on a RHEL8.7 machine, deploy a machine with that edge-commit.

3. On the deployed machine, run `sudo podman images`:

$ sudo podman images

REPOSITORY                               TAG         IMAGE ID      CREATED       SIZE        R/O

registry.access.redhat.com/ubi8/openssl  latest      7ee48a73504f  2 months ago  42.1 MB     true

4. Try to start a container with that image:

$ sudo podman run -it registry.access.redhat.com/ubi8/openssl@sha256:9e743d947be073808f7f1750a791a3dbd81e694e37161e8c6c6057c2c342d671 help

Trying to pull registry.access.redhat.com/ubi8/openssl@sha256:9e743d947be073808f7f1750a791a3dbd81e694e37161e8c6c6057c2c342d671...

[...]

5. Run `sudo podman images` again:

$ sudo podman images

REPOSITORY                               TAG         IMAGE ID      CREATED       SIZE        R/O

registry.access.redhat.com/ubi8/openssl  <none>      7ee48a73504f  2 months ago  42.1 MB     false

registry.access.redhat.com/ubi8/openssl  latest      7ee48a73504f  2 months ago  42.1 MB     true

Experienced outcome:

Container doesn't start until image is once more pulled over the network.

Expected outcome:

Container should start immediately because the image is already embedded.

Additional information:

RHEL8.7 builder and images contain the cgwalters ostree white-out patches.

$ sudo dnf list | grep ostree

osbuild-ostree.noarch                                  65-1.el8                                    @rhel-8-for-x86_64-appstream-rpms
ostree.x86_64                                          2022.2.20.g00b173d0-1.el8                   @copr:copr.fedorainfracloud.org:walters:ostreerhel8
ostree-libs.x86_64                                     2022.2.20.g00b173d0-1.el8                   @copr:copr.fedorainfracloud.org:walters:ostreerhel8
rpm-ostree.x86_64                                      2022.10.99.g0049dbdd-3.el8                  @rhocp-4.12-for-rhel-8-x86_64-rpms
rpm-ostree-libs.x86_64                                 2022.10.99.g0049dbdd-3.el8                  @rhocp-4.12-for-rhel-8-x86_64-rpms