[1991527459] Upstream Reporter: Miloslav Trmač
Upstream issue status: Open
Upstream description:

• Zstd in general:
  • ☐ Switch to Zstd default, earlier than a Zstd:chunked default? Testing in https://github.com/containers/podman/pull/21571 ; https://github.com/containers/podman/pull/21793
  • ☒ Not a bug: pull of gzip + push of zstd changes image ID due to triggering a conversion to OCI: https://github.com/containers/podman/issues/20611
  • ☒ Pushes with Zstd compression prefer using schema1 manifests: https://github.com/containers/image/issues/2181 , https://github.com/containers/image/pull/2196
  • ☒ Pushes with Zstd compression to schema-agnostic transports don?t trigger a conversion: https://github.com/containers/image/issues/2195 , https://github.com/containers/image/pull/2204- Zstd:chunked:
  • ☒ Blob reuse is not restricted by possible manifest schema options: https://github.com/containers/image/issues/2205 , https://github.com/containers/image/pull/2213
  • ☒ Uncompressed pushes of originally-compressed layers don?t correctly update MIME types (always a bug, but now breaks pushes of originally-Zstd to docker-archive: destinations): https://github.com/containers/image/issues/2182
• Zstd:chunked:
  • ☐ copy.Options.EnsureCompressionVariantsExist would never stop adding zstd:chunked variants: https://github.com/containers/image/issues/2318
  • ☐ See if the BlobCache needs updating for Zstd / Zstd:chunked: https://issues.redhat.com/browse/RUN-1124
  • ☐ The chunked manifest size has a hard-coded limit, and pulling large layers outright fails: https://github.com/containers/podman/issues/24885
  • ☒ Exercise Podman tests with Zstd:chunked default: https://github.com/containers/podman/pull/21903
  • ☒ ?view ambiguity?: https://issues.redhat.com/browse/RHEL-66492 , https://github.com/containers/storage/pull/2155 , https://github.com/containers/image/pull/2613 , https://github.com/containers/podman/pull/25007
  • ☒ Pulls with missing parent directories fail: https://issues.redhat.com/browse/RUN-2364: Fixed in https://github.com/containers/storage/pull/2194
  • ☒ Reusing already-present data can be optimized: https://github.com/containers/storage/pull/1830, https://github.com/containers/image/pull/2299, https://github.com/containers/image/pull/2583
  • ☒ Podman?s zstd:chunked-specific tests are not representative: https://github.com/containers/podman/pull/24686
  • ☒ Pulls on VFS fail outright: https://github.com/containers/podman/issues/24308 , https://github.com/containers/storage/pull/2140
  • ☒ expectedLayerDiffIDFlag seems to use mismatching types: https://github.com/containers/image/issues/2602 , https://github.com/containers/image/issues/2602
  • ☒ Document zstd:chunked and encryption interaction: https://github.com/containers/common/issues/2117, https://github.com/containers/common/pull/2176, https://github.com/containers/skopeo/pull/2427, https://github.com/containers/buildah/pull/5759, https://github.com/containers/podman/pull/24113
  • ☒ Propose iterateTarSplit to upstream tar-split: https://github.com/containers/storage/pull/2035#discussion_r1680130116 . Filed https://github.com/vbatts/tar-split/pull/71 , https://github.com/containers/storage/pull/2116 .
  • ☒ blobPipelineDetectCompressionStep detects zstd:chunked as zstd, causing unnecessary recompression: around https://github.com/containers/image/pull/1980#discussion_r1388492171 , https://github.com/containers/image/pull/2487 .
  • ☒ Pushes reusing chunked layers don?t create the required annotations, making the layers impossible to pull chunked: https://github.com/containers/image/pull/2183#pullrequestreview-1725228215 / https://github.com/containers/image/pull/2185 , https://github.com/containers/image/pull/2487
  • ☒ zstd:chunked and layer encryption don?t make sense together: #2485
  • ☒ Blocker? TOC data and tar-split may be ambiguous: https://github.com/containers/storage/issues/2014 , https://github.com/containers/storage/pull/2035
  • ☒ c/storage AdditionalLayerStore seems to be used for TOC-identified layers, incorrectly: https://github.com/containers/storage/pull/795#issuecomment-2007355572 , https://github.com/containers/storage/pull/1924
  • ☒ TarSplitChecksumKey not used in a layer ID: https://github.com/containers/storage/issues/1888 , https://github.com/containers/storage/pull/1902
  • ☒ blobPipelineCompressionStep would trigger a recompression of zstd:chunked if the user asks for zstd (except that we don?t currently detect zstd:chunked): around https://github.com/containers/image/pull/1980#discussion_r1388493306 , https://github.com/containers/image/pull/2317
  • ☒ The c/storage ?binary footer? code path does not work: https://github.com/containers/storage/issues/1886
  • ☒ Outstanding items from https://github.com/containers/storage/pull/1627#pullrequestreview-1600645581 : https://github.com/containers/storage/pull/1844
  • ☒ Copies of chunked layers don?t mark the image as requiring Zstd support: https://github.com/containers/image/issues/2077 , #2302
  • ☒ Chunked layers are visible in c/storage, and usable by other processes, while they are still being populated: https://github.com/containers/image/issues/1979, https://github.com/containers/storage/pull/1826 + #2301 .
  • ☒ PutBlobPartial with a non-chunked input and conversion in c/storage enabled doesn?t work: https://github.com/containers/image/pull/2288#issuecomment-1933549444 , fixed there
  • ☒ Pushes of chunked layers fail because the UncompressedDigest fields stores a TOC digest, and layer digest validation fails: Fixed in https://github.com/containers/image/pull/1980
  • ☒ Pulls of chunked layers do not reuse locally-existing layers: Part of https://github.com/containers/image/pull/1980
  • ☒ In c/storage, we don?t sufficiently differentiate layers pulled by blob digest vs. TOC digest, and reuse is unclear: Part of https://github.com/containers/image/pull/1980 , and https://github.com/containers/storage/pull/1627#discussion_r1309255199 . Fixed in #2288 .
  • ☒ Pulls may reuse layers pulled by blob digests vs. TOC digests: Part of https://github.com/containers/image/pull/1980 . Fixed in #2288 .
  • ☒ Image deduplication contains a (required!) sanity check that TopLayer matches, but that might not be true if layers pulled by blob vs. TOC digests have different IDs: Part of https://github.com/containers/image/pull/1980 . Fixed in #2288 .
  • ☒ Reuse of local layers by TOC may trigger inconsistent metadata updates: #2294
  • ☒ A pull by TOC + conversion to non-OCI might parse data incorrectly: #2295