Major Description
During installation, skopeo fails to push images to automation hub, i.e. this command fails:
skopeo copy --retry-times 3 --dest-creds admin:password --dest-tls-verify=false docker-archive:///tmp/ansible.jm1hs8qiee_images/ee-supported-rhel8.tar docker://customer.domain/ee-supported-rhel8:latest Getting image source signatures Copying blob 5a3ebb577b7b done Copying blob 647a854c512b done Copying blob c888267f26c8 done Copying blob 8e0e04b5c700 done Copying blob e979e22b4ff2 done FATA[0073] writing blob: uploading layer chunked: received unexpected HTTP status: 504 Gateway Time-out
I am using installer 2.2.0.8.
In nginx error log:
2022/08/09 10:54:02 [error] 2104#0: *154 upstream timed out (110: Connection timed out) while sending request to upstream, client: 10.99.216.108, server: customer.domain, request: "PATCH /v2/ee-supported-rhel8/blobs/uploads/3efe29e1-2d1f-4d73-a3fb-4d9594e58666 HTTP/1.1", upstream: "http://unix:/var/run/pulpcore-api/pulpcore-api.sock/v2/ee-supported-rhel8/blobs/uploads/3efe29e1-2d1f-4d73-a3fb-4d9594e58666", host: "customer.domain"
In pulpcore api service:
Aug 09 10:55:54 customer.domain gunicorn[1973]: [2022-08-09 10:55:54 +0200] [1973] [CRITICAL] WORKER TIMEOUT (pid:9064) Aug 09 10:55:55 customer.domain gunicorn[1973]: [2022-08-09 10:55:55 +0200] [1973] [WARNING] Worker with pid 9064 was terminated due to signal 9
Increasing the values for proxy_send_timeout, proxy_read_timeout in nginx.conf and timeout in pulpcore-api.service only delay the issue from happening.
IOPS numbers appears to be correct:
fiotest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.19
Starting 1 process
fiotest: Laying out IO file (1 file / 8192MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=127MiB/s,w=42.0MiB/s][r=32.4k,w=10.8k IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=1): err= 0: pid=213541: Fri Aug 12 09:14:51 2022
read: IOPS=28.1k, BW=110MiB/s (115MB/s)(6141MiB/55862msec)
bw ( KiB/s): min=71401, max=152416, per=100.00%, avg=112707.02, stdev=19715.17, samples=111
iops : min=17850, max=38104, avg=28176.72, stdev=4928.80, samples=111
write: IOPS=9398, BW=36.7MiB/s (38.5MB/s)(2051MiB/55862msec); 0 zone resets
bw ( KiB/s): min=23416, max=50920, per=100.00%, avg=37638.22, stdev=6538.05, samples=111
iops : min= 5854, max=12730, avg=9409.52, stdev=1634.53, samples=111
cpu : usr=6.47%, sys=21.97%, ctx=332450, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=1572145,525007,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64Run status group 0 (all jobs):
READ: bw=110MiB/s (115MB/s), 110MiB/s-110MiB/s (115MB/s-115MB/s), io=6141MiB (6440MB), run=55862-55862msec
WRITE: bw=36.7MiB/s (38.5MB/s), 36.7MiB/s-36.7MiB/s (38.5MB/s-38.5MB/s), io=2051MiB (2150MB), run=55862-55862msec
Configuring the file /etc/pulp/settings.py with the PAH FQDN instead of LB address did not change this behavior.
Key Dependency Versions
automation-hub-4.5.0-1.el8pc.noarch Tue Jul 26 10:13:02 2022 nginx-1.20.1-1.module+el8.6.0+13722+f063ea60.x86_64 Tue Jul 26 10:10:22 2022 nginx-filesystem-1.20.1-1.module+el8.6.0+13722+f063ea60.noarch Tue Jul 26 10:10:22 2022 pulpcore-selinux-1.3.2-1.el8ap.x86_64 Mon Aug 8 18:43:03 2022 python39-pulpcore-3.18.4-4.el8pc.noarch Tue Jul 26 02:32:13 2022 python39-pulp-ansible-0.13.0-3.el8pc.noarch Tue Jul 26 02:33:25 2022 python39-pulp-container-2.10.3-4.el8pc.noarch Tue Jul 26 02:32:46 2022
Steps to Reproduce
Default parameters for installer
Actual Behavior
Skopeo timeouts
Expected Behavior
Successful EE upload to PAH.