Corrupted rpm packages in some CentOS 9 Stream repositories

Description of problem:

The following rpm files are currently corrupted in some of CentOS 9 Stream mirrors

• AppStream/x86_64/os/Packages/rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm
• CRB/x86_64/debug/tree/Packages/python3-qt5-debuginfo-5.15.6-1.el9.i686.rpm
• CRB/x86_64/os/Packages/postgresql-test-13.7-1.el9.x86_64.rpm

For example, the following mirrors have rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm corrupted

• 108.61.16.227
• 147.75.199.55
• 208.100.23.71
• 198.15.78.42

In our case, as a result, Katello fails to synchronize CentOS 9 Stream repositories.

How reproducible:

Steps to Reproduce:

1.1 Random approach - repeat wget few times with 2 minutes interval to get the file from different mirrors (mirror.stream.centos.org DNS A RR has 60 seconds TTL).

$ while :; do
wget http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm
sleep 120
done

1.2 Use curl to download from the the known bad mirrors.

$ rpm=AppStream/x86_64/os/Packages/rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm
for ip in 108.61.16.227 147.75.199.55 208.100.23.71 198.15.78.42; do
curl -v --connect-to ::$ip:80 -o ${rpm##*/}.$ip http://mirror.stream.centos.org/9-stream/$rpm
done

2. See that the copies corrupted
$ file rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm*
rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm.108.61.16.227: data
rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm.147.75.199.55: data
rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm.198.15.78.42: data
rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm.208.100.23.71: data

Actual results:

$ file rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm.108.61.16.227
rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm: data

Expected results:

$ file rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm.108.61.16.227
rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm.108.61.16.227: RPM v3.0 bin i386/x86_64 rust-std-static-wasm32-wasi-1.61.0-1.el9

Additional info:

To download from a specific mirror use a curl command similar to one below (use --connect-to option to specify the mirror IP address). If downloading multiple copies, adjust -o option output pathname every time.

curl -v --connect-to ::108.61.16.227:80 -o postgresql-test-13.7-1.el9.x86_64.rpm.7 http://mirror.stream.centos.org/9-stream/CRB/x86_64/os/Packages/postgresql-test-13.7-1.el9.x86_64.rpm

Here is wget output example

$ wget http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm
-2022-09-29 19:54:01- http://mirror.stream.centos.org/9-stream/AppStream/x86_64/os/Packages/rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm
Resolving mirror.stream.centos.org (mirror.stream.centos.org)... 198.15.78.42, 2600:1f16:c1:5e03:d700:d81:de56:d84a
Connecting to mirror.stream.centos.org (mirror.stream.centos.org)|198.15.78.42|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26190130 (25M) [application/x-rpm]
Saving to: ‘rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm’

rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm 100%[============================================================================================================================>] 24.98M 28.2MB/s in 0.9s

2022-09-29 19:54:02 (28.2 MB/s) - ‘rust-std-static-wasm32-wasi-1.61.0-1.el9.noarch.rpm’ saved [26190130/26190130]