Uploaded image for project: 'Container Tools'
  1. Container Tools
  2. RUN-2575

[containers/buildah] buildah is creating broken filesystems between layers when symlinks overwrite existing symlinks on the same layer

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • rhel-container-tools
    • 3

      [2718924356] Upstream Reporter: Michael Merickel
      Upstream issue status: Closed
      Upstream description:

      Tested with:

      • podman desktop on macos: 5.2.5
      • buildah in eks: buildah version 1.38.0 (image-spec 1.1.0, runtime-spec 1.2.0)

      Examples

      I have 3 container files (broken-1, broken-2, and working).

      The broken-1 file tries to install the newer libpq (libpq.so.5.17) on top of a previously installed libpq (libpq.so.5.15). This fails only when using buildah. It does not fail when built with docker or with kaniko.

      The broken-2 file tries to remove libpq in the same RUN statement as where it tries to install the new libpq. Again this fails only when using buildah. It does not fail when built with docker or kaniko.

      The working file separates the remove and install into two separate RUN statements / layers.

      In the broken containers you'll see that the symlinks are incorrect in the final image.

      broken-1

      # Dockerfile.broken-1
      FROM public.ecr.aws/docker/library/python:3.9.20-bookworm
      
      CMD ["/bin/bash"]
      
      ENV DEBIAN_FRONTEND=noninteractive
      
      # bookworm ships with postgres 15 so we use the ppa to get >= 16
      RUN set -e 
          && echo "deb [signed-by=/etc/apt/keyrings/pgdg.gpg] http://apt.postgresql.org/pub/repos/apt bookworm-pgdg main" > /etc/apt/sources.list.d/pgdg.list 
          && mkdir -p /etc/apt/keyrings 
          && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc > /tmp/pgdg.asc 
          && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --import /tmp/pgdg.asc 
          && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --export > /etc/apt/keyrings/pgdg.gpg 
          && rm -rf /tmp/pgdg* 
          && apt-get update
      
      RUN apt-get install -y libpq-dev postgresql-client && ls -lah /lib/x86_64-linux-gnu/libpq*
      RUN ls -lah /lib/x86_64-linux-gnu/libpq*
      STEP 1/6: FROM public.ecr.aws/docker/library/python:3.9.20-bookworm
      WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
      STEP 2/6: CMD ["/bin/bash"]
      --> 80f6c94f2b39
      STEP 3/6: ENV DEBIAN_FRONTEND=noninteractive
      --> 8c1bbd37e493
      STEP 4/6: RUN set -e     && echo "deb [signed-by=/etc/apt/keyrings/pgdg.gpg] http://apt.postgresql.org/pub/repos/apt bookworm-pgdg main" > /etc/apt/sources.list.d/pgdg.list     && mkdir -p /etc/apt/keyrings     && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc > /tmp/pgdg.asc     && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --import /tmp/pgdg.asc     && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --export > /etc/apt/keyrings/pgdg.gpg     && rm -rf /tmp/pgdg*     && apt-get update
      gpg: keybox '/tmp/pgdg.gpg' created
      gpg: key 7FCC7D46ACCC4CF8: 2 signatures not checked due to missing keys
      gpg: directory '/root/.gnupg' created
      gpg: /root/.gnupg/trustdb.gpg: trustdb created
      gpg: key 7FCC7D46ACCC4CF8: public key "PostgreSQL Debian Repository" imported
      gpg: Total number processed: 1
      gpg:               imported: 1
      gpg: no ultimately trusted keys found
      Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
      Get:2 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease [129 kB]
      Get:3 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
      Get:4 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
      Get:5 http://deb.debian.org/debian bookworm/main amd64 Packages [8789 kB]
      Get:6 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [2712 B]
      Get:7 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [206 kB]
      Get:8 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 Packages [359 kB]
      Fetched 9740 kB in 4s (2653 kB/s)
      Reading package lists...
      --> 6bc9a5a76751
      STEP 5/6: RUN apt-get install -y libpq-dev postgresql-client && ls -lah /lib/x86_64-linux-gnu/libpq*
      Reading package lists...
      Building dependency tree...
      Reading state information...
      The following additional packages will be installed:
        libio-pty-perl libipc-run-perl libpq5 postgresql-client-17
        postgresql-client-common
      Suggested packages:
        postgresql-doc-17 postgresql-17
      The following NEW packages will be installed:
        libio-pty-perl libipc-run-perl postgresql-client postgresql-client-17
        postgresql-client-common
      The following packages will be upgraded:
        libpq-dev libpq5
      2 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
      Need to get 2563 kB of archives.
      After this operation, 11.0 MB of additional disk space will be used.
      Get:1 http://deb.debian.org/debian bookworm/main amd64 libio-pty-perl amd64 1:1.17-1 [34.9 kB]
      Get:2 http://deb.debian.org/debian bookworm/main amd64 libipc-run-perl all 20220807.0-1 [104 kB]
      Get:3 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 libpq-dev amd64 17.2-1.pgdg120+1 [146 kB]
      Get:4 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 libpq5 amd64 17.2-1.pgdg120+1 [223 kB]
      Get:5 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client-common all 267.pgdg120+1 [37.5 kB]
      Get:6 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client-17 amd64 17.2-1.pgdg120+1 [2005 kB]
      Get:7 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client all 17+267.pgdg120+1 [12.4 kB]
      debconf: delaying package configuration, since apt-utils is not installed
      Fetched 2563 kB in 5s (487 kB/s)
      Selecting previously unselected package libio-pty-perl.
      (Reading database ... 23969 files and directories currently installed.)
      Preparing to unpack .../0-libio-pty-perl_1%3a1.17-1_amd64.deb ...
      Unpacking libio-pty-perl (1:1.17-1) ...
      Selecting previously unselected package libipc-run-perl.
      Preparing to unpack .../1-libipc-run-perl_20220807.0-1_all.deb ...
      Unpacking libipc-run-perl (20220807.0-1) ...
      Preparing to unpack .../2-libpq-dev_17.2-1.pgdg120+1_amd64.deb ...
      Unpacking libpq-dev (17.2-1.pgdg120+1) over (15.10-0+deb12u1) ...
      Preparing to unpack .../3-libpq5_17.2-1.pgdg120+1_amd64.deb ...
      Unpacking libpq5:amd64 (17.2-1.pgdg120+1) over (15.10-0+deb12u1) ...
      Selecting previously unselected package postgresql-client-common.
      Preparing to unpack .../4-postgresql-client-common_267.pgdg120+1_all.deb ...
      Unpacking postgresql-client-common (267.pgdg120+1) ...
      Selecting previously unselected package postgresql-client-17.
      Preparing to unpack .../5-postgresql-client-17_17.2-1.pgdg120+1_amd64.deb ...
      Unpacking postgresql-client-17 (17.2-1.pgdg120+1) ...
      Selecting previously unselected package postgresql-client.
      Preparing to unpack .../6-postgresql-client_17+267.pgdg120+1_all.deb ...
      Unpacking postgresql-client (17+267.pgdg120+1) ...
      Setting up libio-pty-perl (1:1.17-1) ...
      Setting up libpq5:amd64 (17.2-1.pgdg120+1) ...
      Setting up libpq-dev (17.2-1.pgdg120+1) ...
      Setting up libipc-run-perl (20220807.0-1) ...
      Setting up postgresql-client-common (267.pgdg120+1) ...
      Setting up postgresql-client-17 (17.2-1.pgdg120+1) ...
      update-alternatives: using /usr/share/postgresql/17/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
      Setting up postgresql-client (17+267.pgdg120+1) ...
      Processing triggers for libc-bin (2.36-9+deb12u9) ...
      -rw-r--r--. 1 root root 348K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.a
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so -> libpq.so.5.17  # <-- CORRECT SYMLINK
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5 -> libpq.so.5.17  # <-- CORRECT SYMLINK
      -rw-r--r--. 1 root root 351K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5.17
      --> 9ba1606c83f2
      STEP 6/6: RUN ls -lah /lib/x86_64-linux-gnu/libpq*
      -rw-r--r--. 1 root root 348K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.a
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so -> libpq.so.5.15  # <-- BROKEN SYMLINK
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5 -> libpq.so.5.15  # <-- BROKEN SYMLINK
      -rw-r--r--. 1 root root 351K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5.17
      COMMIT
      --> bdfbd60cdb18
      bdfbd60cdb18714afb7b97b1fc453f3e68a9b454d3f35a11cee897dbe20b9d7f
      
      ? podman run --rm -it bdfbd60cdb18714afb7b97b1fc453f3e68a9b454d3f35a11cee897dbe20b9d7f /bin/bash
      root@8e81fdc46b49:/# ls -lah /lib/x86_64-linux-gnu/libpq*
      -rw-r--r--. 1 root root 348K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.a
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so -> libpq.so.5.15  # <-- BROKEN SYMLINK
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5 -> libpq.so.5.15  # <-- BROKEN SYMLINK
      -rw-r--r--. 1 root root 351K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5.17

      broken-2

      # Dockerfile.broken-2
      FROM public.ecr.aws/docker/library/python:3.9.20-bookworm
      
      CMD ["/bin/bash"]
      
      ENV DEBIAN_FRONTEND=noninteractive
      
      # bookworm ships with postgres 15 so we use the ppa to get >= 16
      RUN set -e 
          && echo "deb [signed-by=/etc/apt/keyrings/pgdg.gpg] http://apt.postgresql.org/pub/repos/apt bookworm-pgdg main" > /etc/apt/sources.list.d/pgdg.list 
          && mkdir -p /etc/apt/keyrings 
          && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc > /tmp/pgdg.asc 
          && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --import /tmp/pgdg.asc 
          && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --export > /etc/apt/keyrings/pgdg.gpg 
          && rm -rf /tmp/pgdg* 
          && apt-get update
      
      RUN set -e 
          && apt-get remove -y libpq5 libpq-dev 
          && apt-get install -y libpq-dev postgresql-client 
          && ls -lah /lib/x86_64-linux-gnu/libpq*
      RUN ls -lah /lib/x86_64-linux-gnu/libpq*
      ? podman build --no-cache --file Dockerfile.broken-2
      STEP 1/6: FROM public.ecr.aws/docker/library/python:3.9.20-bookworm
      WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
      STEP 2/6: CMD ["/bin/bash"]
      --> 13d947c35f20
      STEP 3/6: ENV DEBIAN_FRONTEND=noninteractive
      --> 8627d702eaa8
      STEP 4/6: RUN set -e     && echo "deb [signed-by=/etc/apt/keyrings/pgdg.gpg] http://apt.postgresql.org/pub/repos/apt bookworm-pgdg main" > /etc/apt/sources.list.d/pgdg.list     && mkdir -p /etc/apt/keyrings     && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc > /tmp/pgdg.asc     && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --import /tmp/pgdg.asc     && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --export > /etc/apt/keyrings/pgdg.gpg     && rm -rf /tmp/pgdg*     && apt-get update
      gpg: keybox '/tmp/pgdg.gpg' created
      gpg: key 7FCC7D46ACCC4CF8: 2 signatures not checked due to missing keys
      gpg: directory '/root/.gnupg' created
      gpg: /root/.gnupg/trustdb.gpg: trustdb created
      gpg: key 7FCC7D46ACCC4CF8: public key "PostgreSQL Debian Repository" imported
      gpg: Total number processed: 1
      gpg:               imported: 1
      gpg: no ultimately trusted keys found
      Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
      Get:2 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease [129 kB]
      Get:3 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
      Get:4 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
      Get:5 http://deb.debian.org/debian bookworm/main amd64 Packages [8789 kB]
      Get:6 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [2712 B]
      Get:7 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [206 kB]
      Get:8 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 Packages [359 kB]
      Fetched 9740 kB in 3s (2835 kB/s)
      Reading package lists...
      --> 190f38ce488b
      STEP 5/6: RUN set -e     && apt-get remove -y libpq5 libpq-dev     && apt-get install -y libpq-dev postgresql-client     && ls -lah /lib/x86_64-linux-gnu/libpq*
      Reading package lists...
      Building dependency tree...
      Reading state information...
      The following packages will be REMOVED:
        libpq-dev libpq5
      0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
      After this operation, 1464 kB disk space will be freed.
      (Reading database ... 23969 files and directories currently installed.)
      Removing libpq-dev (15.10-0+deb12u1) ...
      Removing libpq5:amd64 (15.10-0+deb12u1) ...
      Processing triggers for libc-bin (2.36-9+deb12u9) ...
      Reading package lists...
      Building dependency tree...
      Reading state information...
      The following additional packages will be installed:
        libio-pty-perl libipc-run-perl libpq5 postgresql-client-17
        postgresql-client-common
      Suggested packages:
        postgresql-doc-17 postgresql-17
      The following NEW packages will be installed:
        libio-pty-perl libipc-run-perl libpq-dev libpq5 postgresql-client
        postgresql-client-17 postgresql-client-common
      0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
      Need to get 2563 kB of archives.
      After this operation, 12.4 MB of additional disk space will be used.
      Get:1 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 libpq5 amd64 17.2-1.pgdg120+1 [223 kB]
      Get:2 http://deb.debian.org/debian bookworm/main amd64 libio-pty-perl amd64 1:1.17-1 [34.9 kB]
      Get:3 http://deb.debian.org/debian bookworm/main amd64 libipc-run-perl all 20220807.0-1 [104 kB]
      Get:4 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 libpq-dev amd64 17.2-1.pgdg120+1 [146 kB]
      Get:5 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client-common all 267.pgdg120+1 [37.5 kB]
      Get:6 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client-17 amd64 17.2-1.pgdg120+1 [2005 kB]
      Get:7 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client all 17+267.pgdg120+1 [12.4 kB]
      debconf: delaying package configuration, since apt-utils is not installed
      Fetched 2563 kB in 1s (2436 kB/s)
      Selecting previously unselected package libio-pty-perl.
      (Reading database ... 23922 files and directories currently installed.)
      Preparing to unpack .../0-libio-pty-perl_1%3a1.17-1_amd64.deb ...
      Unpacking libio-pty-perl (1:1.17-1) ...
      Selecting previously unselected package libipc-run-perl.
      Preparing to unpack .../1-libipc-run-perl_20220807.0-1_all.deb ...
      Unpacking libipc-run-perl (20220807.0-1) ...
      Selecting previously unselected package libpq5:amd64.
      Preparing to unpack .../2-libpq5_17.2-1.pgdg120+1_amd64.deb ...
      Unpacking libpq5:amd64 (17.2-1.pgdg120+1) ...
      Selecting previously unselected package libpq-dev.
      Preparing to unpack .../3-libpq-dev_17.2-1.pgdg120+1_amd64.deb ...
      Unpacking libpq-dev (17.2-1.pgdg120+1) ...
      Selecting previously unselected package postgresql-client-common.
      Preparing to unpack .../4-postgresql-client-common_267.pgdg120+1_all.deb ...
      Unpacking postgresql-client-common (267.pgdg120+1) ...
      Selecting previously unselected package postgresql-client-17.
      Preparing to unpack .../5-postgresql-client-17_17.2-1.pgdg120+1_amd64.deb ...
      Unpacking postgresql-client-17 (17.2-1.pgdg120+1) ...
      Selecting previously unselected package postgresql-client.
      Preparing to unpack .../6-postgresql-client_17+267.pgdg120+1_all.deb ...
      Unpacking postgresql-client (17+267.pgdg120+1) ...
      Setting up libio-pty-perl (1:1.17-1) ...
      Setting up libpq5:amd64 (17.2-1.pgdg120+1) ...
      Setting up libpq-dev (17.2-1.pgdg120+1) ...
      Setting up libipc-run-perl (20220807.0-1) ...
      Setting up postgresql-client-common (267.pgdg120+1) ...
      Setting up postgresql-client-17 (17.2-1.pgdg120+1) ...
      update-alternatives: using /usr/share/postgresql/17/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
      Setting up postgresql-client (17+267.pgdg120+1) ...
      Processing triggers for libc-bin (2.36-9+deb12u9) ...
      -rw-r--r--. 1 root root 348K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.a
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so -> libpq.so.5.17  # <-- CORRECT SYMLINK
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5 -> libpq.so.5.17  # <-- CORRECT SYMLINK
      -rw-r--r--. 1 root root 351K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5.17
      --> a6b5a1c4f04c
      STEP 6/6: RUN ls -lah /lib/x86_64-linux-gnu/libpq*
      -rw-r--r--. 1 root root 348K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.a
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so -> libpq.so.5.15  # <-- BROKEN SYMLINK
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5 -> libpq.so.5.15  # <-- BROKEN SYMLINK
      -rw-r--r--. 1 root root 351K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5.17
      COMMIT
      --> f16a24fb54f8
      f16a24fb54f8401ca0271840ddf427e14a362e25f3261a577f9be389374a4a10
      
      ? podman run --rm -it f16a24fb54f8401ca0271840ddf427e14a362e25f3261a577f9be389374a4a10 /bin/bash
      root@959cba30995c:/# ls -lah /lib/x86_64-linux-gnu/libpq*
      -rw-r--r--. 1 root root 348K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.a
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so -> libpq.so.5.15  # <-- BROKEN SYMLINK
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5 -> libpq.so.5.15  # <-- BROKEN SYMLINK
      -rw-r--r--. 1 root root 351K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5.17

      working

      # Dockerfile.working
      FROM public.ecr.aws/docker/library/python:3.9.20-bookworm
      
      ENV DEBIAN_FRONTEND=noninteractive
      
      # bookworm ships with postgres 15 so we use the ppa to get >= 16
      RUN set -e 
          && echo "deb [signed-by=/etc/apt/keyrings/pgdg.gpg] http://apt.postgresql.org/pub/repos/apt bookworm-pgdg main" > /etc/apt/sources.list.d/pgdg.list 
          && mkdir -p /etc/apt/keyrings 
          && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc > /tmp/pgdg.asc 
          && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --import /tmp/pgdg.asc 
          && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --export > /etc/apt/keyrings/pgdg.gpg 
          && rm -rf /tmp/pgdg* 
          && apt-get update
      
      RUN apt-get remove -y libpq-dev libpq5
      RUN apt-get install -y libpq-dev postgresql-client
      RUN ls -lah /lib/x86_64-linux-gnu/libpq*
      ? podman build --no-cache --file Dockerfile.working
      STEP 1/5: FROM public.ecr.aws/docker/library/python:3.9.20-bookworm
      WARNING: image platform (linux/amd64) does not match the expected platform (linux/arm64)
      STEP 2/5: ENV DEBIAN_FRONTEND=noninteractive
      --> 2c8ed75e2b08
      STEP 3/5: RUN set -e     && echo "deb [signed-by=/etc/apt/keyrings/pgdg.gpg] http://apt.postgresql.org/pub/repos/apt bookworm-pgdg main" > /etc/apt/sources.list.d/pgdg.list     && mkdir -p /etc/apt/keyrings     && wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc > /tmp/pgdg.asc     && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --import /tmp/pgdg.asc     && gpg --no-default-keyring --keyring /tmp/pgdg.gpg --export > /etc/apt/keyrings/pgdg.gpg     && rm -rf /tmp/pgdg*     && apt-get update
      gpg: keybox '/tmp/pgdg.gpg' created
      gpg: key 7FCC7D46ACCC4CF8: 2 signatures not checked due to missing keys
      gpg: directory '/root/.gnupg' created
      gpg: /root/.gnupg/trustdb.gpg: trustdb created
      gpg: key 7FCC7D46ACCC4CF8: public key "PostgreSQL Debian Repository" imported
      gpg: Total number processed: 1
      gpg:               imported: 1
      gpg: no ultimately trusted keys found
      Get:1 http://deb.debian.org/debian bookworm InRelease [151 kB]
      Get:2 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease [129 kB]
      Get:3 http://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]
      Get:4 http://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]
      Get:5 http://deb.debian.org/debian bookworm/main amd64 Packages [8789 kB]
      Get:6 http://deb.debian.org/debian bookworm-updates/main amd64 Packages [2712 B]
      Get:7 http://deb.debian.org/debian-security bookworm-security/main amd64 Packages [206 kB]
      Get:8 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 Packages [359 kB]
      Fetched 9740 kB in 4s (2687 kB/s)
      Reading package lists...
      --> bebe3557cc7a
      STEP 4/5: RUN apt-get remove -y libpq-dev libpq5
      Reading package lists...
      Building dependency tree...
      Reading state information...
      The following packages will be REMOVED:
        libpq-dev libpq5
      0 upgraded, 0 newly installed, 2 to remove and 0 not upgraded.
      After this operation, 1464 kB disk space will be freed.
      (Reading database ... 23969 files and directories currently installed.)
      Removing libpq-dev (15.10-0+deb12u1) ...
      Removing libpq5:amd64 (15.10-0+deb12u1) ...
      Processing triggers for libc-bin (2.36-9+deb12u9) ...
      --> dd0f95ea2e2b
      STEP 5/5: RUN apt-get install -y libpq-dev postgresql-client
      Reading package lists...
      Building dependency tree...
      Reading state information...
      The following additional packages will be installed:
        libio-pty-perl libipc-run-perl libpq5 postgresql-client-17
        postgresql-client-common
      Suggested packages:
        postgresql-doc-17 postgresql-17
      The following NEW packages will be installed:
        libio-pty-perl libipc-run-perl libpq-dev libpq5 postgresql-client
        postgresql-client-17 postgresql-client-common
      0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
      Need to get 2563 kB of archives.
      After this operation, 12.4 MB of additional disk space will be used.
      Get:1 http://deb.debian.org/debian bookworm/main amd64 libio-pty-perl amd64 1:1.17-1 [34.9 kB]
      Get:2 http://deb.debian.org/debian bookworm/main amd64 libipc-run-perl all 20220807.0-1 [104 kB]
      Get:3 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 libpq5 amd64 17.2-1.pgdg120+1 [223 kB]
      Get:4 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 libpq-dev amd64 17.2-1.pgdg120+1 [146 kB]
      Get:5 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client-common all 267.pgdg120+1 [37.5 kB]
      Get:6 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client-17 amd64 17.2-1.pgdg120+1 [2005 kB]
      Get:7 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main amd64 postgresql-client all 17+267.pgdg120+1 [12.4 kB]
      debconf: delaying package configuration, since apt-utils is not installed
      Fetched 2563 kB in 4s (659 kB/s)
      Selecting previously unselected package libio-pty-perl.
      (Reading database ... 23922 files and directories currently installed.)
      Preparing to unpack .../0-libio-pty-perl_1%3a1.17-1_amd64.deb ...
      Unpacking libio-pty-perl (1:1.17-1) ...
      Selecting previously unselected package libipc-run-perl.
      Preparing to unpack .../1-libipc-run-perl_20220807.0-1_all.deb ...
      Unpacking libipc-run-perl (20220807.0-1) ...
      Selecting previously unselected package libpq5:amd64.
      Preparing to unpack .../2-libpq5_17.2-1.pgdg120+1_amd64.deb ...
      Unpacking libpq5:amd64 (17.2-1.pgdg120+1) ...
      Selecting previously unselected package libpq-dev.
      Preparing to unpack .../3-libpq-dev_17.2-1.pgdg120+1_amd64.deb ...
      Unpacking libpq-dev (17.2-1.pgdg120+1) ...
      Selecting previously unselected package postgresql-client-common.
      Preparing to unpack .../4-postgresql-client-common_267.pgdg120+1_all.deb ...
      Unpacking postgresql-client-common (267.pgdg120+1) ...
      Selecting previously unselected package postgresql-client-17.
      Preparing to unpack .../5-postgresql-client-17_17.2-1.pgdg120+1_amd64.deb ...
      Unpacking postgresql-client-17 (17.2-1.pgdg120+1) ...
      Selecting previously unselected package postgresql-client.
      Preparing to unpack .../6-postgresql-client_17+267.pgdg120+1_all.deb ...
      Unpacking postgresql-client (17+267.pgdg120+1) ...
      Setting up libio-pty-perl (1:1.17-1) ...
      Setting up libpq5:amd64 (17.2-1.pgdg120+1) ...
      Setting up libpq-dev (17.2-1.pgdg120+1) ...
      Setting up libipc-run-perl (20220807.0-1) ...
      Setting up postgresql-client-common (267.pgdg120+1) ...
      Setting up postgresql-client-17 (17.2-1.pgdg120+1) ...
      update-alternatives: using /usr/share/postgresql/17/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
      Setting up postgresql-client (17+267.pgdg120+1) ...
      Processing triggers for libc-bin (2.36-9+deb12u9) ...
      COMMIT
      --> 538fc3926181
      538fc3926181a6bf06c5d68190f2a56f00839787bc97cacb4387fddca417e5d4
      
      ? podman run --rm -it 538fc3926181a6bf06c5d68190f2a56f00839787bc97cacb4387fddca417e5d4 /bin/bash
      root@123a1b28405e:/# ls -lah /lib/x86_64-linux-gnu/libpq*
      -rw-r--r--. 1 root root 348K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.a
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so -> libpq.so.5.17
      lrwxrwxrwx. 1 root root   13 Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5 -> libpq.so.5.17
      -rw-r--r--. 1 root root 351K Nov 19 14:36 /lib/x86_64-linux-gnu/libpq.so.5.17

      Analysis

      All 3 of these docker files work fine on docker and on kaniko (rootless) when building an image. I have tested combinations of the default settings as well as BUILDAH_ISOLATION=chroot and STORAGE_DRIVER=vfs and I'm still seeing the same issue.

      Please note that within the layer itself the filesystem looks correct. But when I start a new layer then you will see the issue. So I suspect it is something to do potentially with symlinks between layers and how they are being tracked.

      I'd consider this a critical bug.


      Upstream URL: https://github.com/containers/buildah/issues/5861

              gscrivan@redhat.com Giuseppe Scrivano
              upstream-sync Upstream Sync
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: