Quay Image Repository Mirror was stuck

Description:

This is an issue when performing image Repository Mirroring, after enable image repository to mirror status, click Mirroring Tab, provide a public external Repository, tags, Sync interval, select robot account with write permissions of this image repository, enable Mirror and Click "Sync Now", the result is Repository Mirroring is stuck, check two repomirror  PODS, get message "skipped: maximum number of running instances reached (1)", refer to the POD logs

Note: In this Quay ENV, only one image repository was set as Mirror status.

Quay Image: quay.io/projectquay/quay:50ba6130

Steps:

1. Login Quay Console
2. Create new Image repository
3. Create new robot account, give write permissions to repository
4. Click "Settings" tab and change Image repository to mirror status
5. Click "Mirroring" tab, input required configurations, here use a public repository of dockerHub as external repository
6. Choose correct robot account, and specify Sync internal
7. Click "Enable Mirror"
8. Wait for scheduled Sync to complete

Expected Results:

Quay repository mirror should complete successfully, all source images matching specified tags should be sync to target image repository.

Actual Results:

Quay repository mirror is Stuck.

Quay Repomirror POD logs:

repomirrorworker stdout | 2020-08-05 12:00:00,651 [60] [INFO] [apscheduler.scheduler] Scheduler started

repomirrorworker stdout | 2020-08-05 12:00:00,721 [60] [INFO] [apscheduler.scheduler] Added job "RepoMirrorWorker._process_mirrors" to job store "default"

repomirrorworker stdout | 2020-08-05 12:00:21,654 [60] [INFO] [apscheduler.executors.default] Running job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:00:21 UTC)" (scheduled at 2020-08-05 12:00:21.653586+00:00)

repomirrorworker stdout | 2020-08-05 12:00:21,667 [60] [INFO] [util.migrate.allocator] Found 1 candidates, processing block

repomirrorworker stdout | 2020-08-05 12:00:51,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:00:51 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:01:21,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:01:21 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:01:51,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:01:51 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:02:21,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:02:21 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:02:51,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:02:51 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:03:21,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:03:21 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:03:51,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:03:51 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:04:21,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:04:21 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:04:51,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:04:51 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:05:21,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:05:21 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:05:51,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:05:51 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:06:21,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:06:21 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:06:51,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:06:51 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:07:21,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:07:21 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:07:51,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:07:51 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:08:21,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:08:21 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:08:51,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:08:51 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:09:21,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:09:21 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:09:51,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:09:51 UTC)" skipped: maximum number of running instances reached (1)

repomirrorworker stdout | 2020-08-05 12:10:21,654 [60] [WARNING] [apscheduler.scheduler] Execution of job "RepoMirrorWorker._process_mirrors (trigger: interval[0:00:30], next run at: 2020-08-05 12:10:21 UTC)" skipped: maximum number of running instances reached (1)