[3830393082] Upstream Reporter: Han-Wen Nienhuys
Upstream issue status: Closed
Upstream description:

Issue Description

While debugging a race condition, I tried running podman 6-dev (1090a09da81d5b41d8bd48bb969d49ec795ec866) with -race.

I got the following stacktrace,

==================
WARNING: DATA RACE
Write at 0x00c0000bedc9 by goroutine 984:
  go.podman.io/storage/pkg/lockfile.(*LockFile).lock()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/pkg/lockfile/lockfile.go:404 +0x245
  go.podman.io/storage/pkg/lockfile.(*LockFile).RLock()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/pkg/lockfile/lockfile.go:132 +0x99
  go.podman.io/storage.(*imageStore).startReadingWithReload()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/images.go:262 +0x81
  go.podman.io/storage.(*imageStore).startReading()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/images.go:349 +0x2b
  go.podman.io/storage.readAllImageStores[go.shape.[]uint8]()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/store.go:1369 +0x218
  go.podman.io/storage.(*store).ImageBigData()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/store.go:2148 +0x11a
  go.podman.io/image/v5/storage.(*storageImageSource).GetBlob()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/image/v5/storage/storage_src.go:147 +0x412
  go.podman.io/image/v5/internal/image.(*manifestOCI1).ConfigBlob()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/image/v5/internal/image/oci.go:67 +0x47d
  go.podman.io/image/v5/internal/image.(*manifestOCI1).OCIConfig()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/image/v5/internal/image/oci.go:92 +0x164
  go.podman.io/image/v5/internal/image.(*SourcedImage).OCIConfig()
      <autogenerated>:1 +0x67
  go.podman.io/image/v5/internal/image.(*imageCloser).OCIConfig()
      <autogenerated>:1 +0x63
  go.podman.io/image/v5/storage.(*storageImageCloser).OCIConfig()
      <autogenerated>:1 +0x63
  github.com/containers/podman/v6/libpod.(*storageService).CreateContainerStorage()
      /home/hanwen/vc/containers/podman/libpod/storage.go:90 +0x2f3
  github.com/containers/podman/v6/libpod.(*Container).setupStorage()
      /home/hanwen/vc/containers/podman/libpod/container_internal.go:515 +0x1117
  github.com/containers/podman/v6/libpod.(*Runtime).setupContainer()
      /home/hanwen/vc/containers/podman/libpod/runtime_ctr.go:456 +0x2ea6
  github.com/containers/podman/v6/libpod.(*Runtime).newContainer()
      /home/hanwen/vc/containers/podman/libpod/runtime_ctr.go:242 +0x205
  github.com/containers/podman/v6/libpod.(*Runtime).NewContainer()
      /home/hanwen/vc/containers/podman/libpod/runtime_ctr.go:58 +0x25a
  github.com/containers/podman/v6/pkg/specgen/generate.ExecuteCreate()
      /home/hanwen/vc/containers/podman/pkg/specgen/generate/container_create.go:317 +0xb2
  github.com/containers/podman/v6/pkg/domain/infra/abi.(*ContainerEngine).ContainerCreate()
      /home/hanwen/vc/containers/podman/pkg/domain/infra/abi/containers.go:848 +0x23b
  github.com/containers/podman/v6/pkg/api/handlers/compat.CreateContainer()
      /home/hanwen/vc/containers/podman/pkg/api/handlers/compat/containers_create.go:128 +0x1104
  github.com/containers/podman/v6/pkg/api/server.(*APIServer).apiWrapper()
      /home/hanwen/vc/containers/podman/pkg/api/server/handler_api.go:66 +0xcf8
  github.com/containers/podman/v6/pkg/api/server.(*APIServer).registerContainersHandlers.(*APIServer).APIHandler.func1()
      /home/hanwen/vc/containers/podman/pkg/api/server/handler_api.go:26 +0x59
  net/http.HandlerFunc.ServeHTTP()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:2294 +0x47
  github.com/containers/podman/v6/pkg/api/server.newServer.referenceIDHandler.func7.1()
      /home/hanwen/vc/containers/podman/pkg/api/server/handler_rid.go:40 +0x1c1
  net/http.HandlerFunc.ServeHTTP()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:2294 +0x47
  github.com/gorilla/handlers.loggingHandler.ServeHTTP()
      /home/hanwen/vc/containers/podman/vendor/github.com/gorilla/handlers/logging.go:47 +0x143
  github.com/gorilla/handlers.(*loggingHandler).ServeHTTP()
      <autogenerated>:1 +0xae
  github.com/containers/podman/v6/pkg/api/server.newServer.panicHandler.func6.1()
      /home/hanwen/vc/containers/podman/pkg/api/server/handler_panic.go:31 +0xd4
  net/http.HandlerFunc.ServeHTTP()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:2294 +0x47
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /home/hanwen/vc/containers/podman/vendor/github.com/gorilla/mux/mux.go:212 +0x366
  net/http.serverHandler.ServeHTTP()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:3301 +0x2a1
  net/http.(*conn).serve()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:2102 +0x1304
  net/http.(*Server).Serve.gowrap3()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:3454 +0x4f

Previous read at 0x00c0000bedc9 by goroutine 897:
  go.podman.io/storage/pkg/lockfile.(*LockFile).AssertLocked()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/pkg/lockfile/lockfile.go:192 +0x44
  go.podman.io/storage/pkg/lockfile.(*LockFile).ModifiedSince()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/pkg/lockfile/lockfile.go:230 +0x3b
  go.podman.io/storage.(*imageStore).modified()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/images.go:364 +0x24a
  go.podman.io/storage.(*imageStore).startReadingWithReload()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/images.go:274 +0x201
  go.podman.io/storage.(*imageStore).startReading()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/images.go:349 +0x2b
  go.podman.io/storage.readAllImageStores[go.shape.*uint8]()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/store.go:1369 +0x215
  go.podman.io/storage.(*store).Image()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/storage/store.go:3565 +0xca
  go.podman.io/image/v5/storage.(*storageReference).resolveImage()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/image/v5/storage/storage_reference.go:160 +0x2c1
  go.podman.io/image/v5/storage.newImageSource()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/image/v5/storage/storage_src.go:67 +0x44
  go.podman.io/image/v5/storage.newImage()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/image/v5/storage/storage_image.go:55 +0x69
  go.podman.io/image/v5/storage.storageReference.NewImage()
      /home/hanwen/vc/containers/podman/vendor/go.podman.io/image/v5/storage/storage_reference.go:271 +0x22f
  github.com/containers/podman/v6/libpod.(*storageService).CreateContainerStorage()
      /home/hanwen/vc/containers/podman/libpod/storage.go:83 +0x1c0
  github.com/containers/podman/v6/libpod.(*Container).setupStorage()
      /home/hanwen/vc/containers/podman/libpod/container_internal.go:515 +0x1117
  github.com/containers/podman/v6/libpod.(*Runtime).setupContainer()
      /home/hanwen/vc/containers/podman/libpod/runtime_ctr.go:456 +0x2ea6
  github.com/containers/podman/v6/libpod.(*Runtime).newContainer()
      /home/hanwen/vc/containers/podman/libpod/runtime_ctr.go:242 +0x205
  github.com/containers/podman/v6/libpod.(*Runtime).NewContainer()
      /home/hanwen/vc/containers/podman/libpod/runtime_ctr.go:58 +0x25a
  github.com/containers/podman/v6/pkg/specgen/generate.ExecuteCreate()
      /home/hanwen/vc/containers/podman/pkg/specgen/generate/container_create.go:317 +0xb2
  github.com/containers/podman/v6/pkg/domain/infra/abi.(*ContainerEngine).ContainerCreate()
      /home/hanwen/vc/containers/podman/pkg/domain/infra/abi/containers.go:848 +0x23b
  github.com/containers/podman/v6/pkg/api/handlers/compat.CreateContainer()
      /home/hanwen/vc/containers/podman/pkg/api/handlers/compat/containers_create.go:128 +0x1104
  github.com/containers/podman/v6/pkg/api/server.(*APIServer).apiWrapper()
      /home/hanwen/vc/containers/podman/pkg/api/server/handler_api.go:66 +0xcf8
  github.com/containers/podman/v6/pkg/api/server.(*APIServer).registerContainersHandlers.(*APIServer).APIHandler.func1()
      /home/hanwen/vc/containers/podman/pkg/api/server/handler_api.go:26 +0x59
  net/http.HandlerFunc.ServeHTTP()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:2294 +0x47
  github.com/containers/podman/v6/pkg/api/server.newServer.referenceIDHandler.func7.1()
      /home/hanwen/vc/containers/podman/pkg/api/server/handler_rid.go:40 +0x1c1
  net/http.HandlerFunc.ServeHTTP()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:2294 +0x47
  github.com/gorilla/handlers.loggingHandler.ServeHTTP()
      /home/hanwen/vc/containers/podman/vendor/github.com/gorilla/handlers/logging.go:47 +0x143
  github.com/gorilla/handlers.(*loggingHandler).ServeHTTP()
      <autogenerated>:1 +0xae
  github.com/containers/podman/v6/pkg/api/server.newServer.panicHandler.func6.1()
      /home/hanwen/vc/containers/podman/pkg/api/server/handler_panic.go:31 +0xd4
  net/http.HandlerFunc.ServeHTTP()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:2294 +0x47
  github.com/gorilla/mux.(*Router).ServeHTTP()
      /home/hanwen/vc/containers/podman/vendor/github.com/gorilla/mux/mux.go:212 +0x366
  net/http.serverHandler.ServeHTTP()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:3301 +0x2a1
  net/http.(*conn).serve()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:2102 +0x1304
  net/http.(*Server).Serve.gowrap3()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:3454 +0x4f

Goroutine 984 (running) created at:
  net/http.(*Server).Serve()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:3454 +0x8c9
  github.com/containers/podman/v6/pkg/api/server.(*APIServer).Serve.func3()
      /home/hanwen/vc/containers/podman/pkg/api/server/server.go:266 +0x306

Goroutine 897 (running) created at:
  net/http.(*Server).Serve()
      /home/hanwen/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.6.linux-amd64/src/net/http/server.go:3454 +0x8c9
  github.com/containers/podman/v6/pkg/api/server.(*APIServer).Serve.func3()
      /home/hanwen/vc/containers/podman/pkg/api/server/server.go:266 +0x306
==================

Steps to reproduce the issue

This happened while running sudo podman system service unix://$SOCKET and running docker-ce-client 29.1 against said socket.

Describe the results you received

Describe the results you received

Describe the results you expected

Describe the results you expected

podman info output

I self-compiled with: 
`go build -race       -buildvcs=false         -o /tmp/podman         -ldflags "-X github.com/containers/podman/v6/libpod/define.gitCommit=engflow-superproject-$(git rev-parse HEAD) -X github.com/containers/podman/v5/libpod/define.engflowVersion=${DEBIAN_VERSION} -X github.com/containers/podman/v5/libpod/config._installPrefix=/usr/local -X github.com/containers/podman/v5/libpod/config._etcDir=/etc -X github.com/containers/podman/v5/pkg/systemd/quadlet._binDir=/usr/local/bin -X github.com/containers/common/pkg/config.additionalHelperBinariesDir= "         -tags "apparmor systemd exclude_graphdriver_devicemapper seccomp "         ./cmd/podman; sudo cp /tmp/podman /usr/local/bin/podman`

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting