[2853841336] Upstream Reporter: Sean Lafferty
Upstream issue status: Closed
Upstream description:

Issue Description

I suspect this was never an intended feature, but our use case is CI runners. We have hundreds of replicas, each of which has podman installed. Jobs always start with container pulls (which is the slowest part of the job). If a job pulls an image, we'd like for it to be written to a shared imagestore for subsequent jobs to reuse.

I created a imgs volume on the host, which I mounted into all of the runners. Then, all the runners were given a storage.conf with the contents:

[storage]
  driver = "overlay"
  imagestore = "/home/runner/imgs"

My expectation was that any podman pull inside of a runner would read/write this shared imagestore. However, the pulls step on each other, corrupting the imagestore/root.

Steps to reproduce the issue

# pull with root1, shared imgstore (works)
podman --root /tmp/root1 --imagestore /tmp/imgs pull mirror.gcr.io/library/bash

# pull with root2, shared imgstore (doesn't work)
podman --root /tmp/root2 --imagestore /tmp/imgs pull mirror.gcr.io/library/bash

ERRO[0000] Image mirror.gcr.io/library/bash exists in local storage but may be corrupted (remove the image to resolve the issue): layer not known

Describe the results you received

Corrupted imagestore

Describe the results you expected

Non-corrupted imagestore

podman info output

host:   arch: amd64
  buildahVersion: 1.37.6
  cgroupControllers:   - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:     package: conmon-2.1.12-1.el9.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: c0564282e9befb7804c3642230f8e94f1b2ba9f8'
  cpuUtilization:     idlePercent: 95.6
    systemPercent: 1.94
    userPercent: 2.47
  cpus: 2
  databaseBackend: sqlite
  distribution:     distribution: rhel
    version: "9.4"
  eventLogger: journald
  freeLocks: 2033
  hostname: slaffy-github1.fyre.ibm.com
  idMappings:     gidmap: null
    uidmap: null
  kernel: 5.14.0-427.42.1.el9_4.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 357359616
  memTotal: 3837394944
  networkBackend: netavark
  networkBackendInfo:     backend: netavark
    dns:       package: aardvark-dns-1.12.1-1.el9.x86_64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.1
    package: netavark-1.12.2-1.el9.x86_64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:     name: crun
    package: crun-1.16.1-1.el9.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.16.1
      commit: afa829ca0122bd5e1d67f1f38e6cc348027e3c32
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:     executable: /usr/bin/pasta
    package: passt-0^20240806.gee36266-2.el9.x86_64
    version: |
      pasta 0^20240806.gee36266-2.el9.x86_64
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:     exists: false
    path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:     apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:     executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.1-1.el9.x86_64
    version: |-
      slirp4netns version 1.3.1
      commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 17118425088
  swapTotal: 17175670784
  uptime: 453h 46m 52.00s (Approximately 18.88 days)
  variant: ""
plugins:   authorization: null
  log:   - k8s-file
  - none
  - passthrough
  - journald
  network:   - bridge
  - macvlan
  - ipvlan
  volume:   - local
registries:   search:   - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:   configFile: /etc/containers/storage.conf
  containerStore:     number: 7
    paused: 0
    running: 7
    stopped: 0
  graphDriverName: overlay
  graphOptions:     overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 250059685888
  graphRootUsed: 124696444928
  graphStatus:     Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:     number: 1970
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:   APIVersion: 5.2.2
  Built: 1737721907
  BuiltTime: Fri Jan 24 04:31:47 2025
  GitCommit: ""
  GoVersion: go1.22.9 (Red Hat 1.22.9-2.el9_5)
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.2

Podman in a container

Yes

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

No response

Additional information

No response