Description of problem:

As part of the Satellite 6.9 Pulp 3 High-Touch Beta, several duplicate key errors were observed on Red Hat repository syncs.

During this HTB, a Satellite 6.9 instance went through the process of migration to Pulp 3, a switchover to Pulp 3 was performed and content management workflows executed.

Version-Release number of selected component (if applicable):
6.9.1

Errors observed:

root@hostname:/var/opt/rh/rh-postgresql12/lib/pgsql/data/log# grep -c ERROR *.log
postgresql-Fri.log:39
postgresql-Mon.log:7
postgresql-Sat.log:0
postgresql-Sun.log:0
postgresql-Thu.log:35071
postgresql-Tue.log:5955
postgresql-Wed.log:33

The errors on Thursday are between 03:00am-04:00am that was the first time I was running the RedHat Repositories sync after the pulp3 switchover

To give an idea on the tables and error i summarized the count per hour and error

root@hostname:/var/opt/rh/rh-postgresql12/lib/pgsql/data/log# grep 'UTC ERROR:' .log | cut -c-1000 | sed 's/(2021-. [0-9].:).*UTC/\1/' | sort | uniq -c
21 postgresql-Fri.log:2021-07-02 03: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
5 postgresql-Fri.log:2021-07-02 09: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
4 postgresql-Fri.log:2021-07-02 12: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
1 postgresql-Fri.log:2021-07-02 13: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
1 postgresql-Fri.log:2021-07-02 13: ERROR: duplicate key value violates unique constraint "dynflow_coordinator_records_pkey"
10 postgresql-Fri.log:2021-07-02 14: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
1 postgresql-Mon.log:2021-06-07 14: ERROR: duplicate key value violates unique constraint "dynflow_coordinator_records_pkey"
1 postgresql-Mon.log:2021-06-14 13: ERROR: duplicate key value violates unique constraint "dynflow_coordinator_records_pkey"
1 postgresql-Mon.log:2021-06-28 07: ERROR: duplicate key value violates unique constraint "dynflow_coordinator_records_pkey"
1 postgresql-Mon.log:2021-06-28 08: ERROR: duplicate key value violates unique constraint "dynflow_coordinator_records_pkey"
2 postgresql-Mon.log:2021-06-28 08: ERROR: duplicate key value violates unique constraint "index_katello_rpms_on_pulp_id"
1 postgresql-Mon.log:2021-06-28 13: ERROR: duplicate key value violates unique constraint "dynflow_coordinator_records_pkey"
1 postgresql-Thu.log:2021-07-01 03: ERROR: could not extend file "base/16926/640886": No space left on device
2 postgresql-Thu.log:2021-07-01 03: ERROR: could not write to file "base/pgsql_tmp/pgsql_tmp12388.4": No space left on device
2 postgresql-Thu.log:2021-07-01 03: ERROR: could not write to file "base/pgsql_tmp/pgsql_tmp14002.4": No space left on device
2 postgresql-Thu.log:2021-07-01 03: ERROR: could not write to file "base/pgsql_tmp/pgsql_tmp14313.2": No space left on device
263 postgresql-Thu.log:2021-07-01 03: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
843 postgresql-Thu.log:2021-07-01 03: ERROR: duplicate key value violates unique constraint "index_katello_errata_on_pulp_id"
2111 postgresql-Thu.log:2021-07-01 03: ERROR: duplicate key value violates unique constraint "index_katello_erratum_cves_on_erratum_id_and_cve_id_and_href"
3990 postgresql-Thu.log:2021-07-01 03: ERROR: duplicate key value violates unique constraint "katello_erratum_bz_eid_bid_href"
18171 postgresql-Thu.log:2021-07-01 03: ERROR: duplicate key value violates unique constraint "katello_erratum_packages_eid_nvrea_n_f"
1932 postgresql-Thu.log:2021-07-01 03: ERROR: duplicate key value violates unique constraint "rpm_modulemd_name_stream_version_context_arch_f6598e4e_uniq"
6 postgresql-Thu.log:2021-07-01 03: ERROR: duplicate key value violates unique constraint "rpm_package_pkgId_key"
7662 postgresql-Thu.log:2021-07-01 03: ERROR: duplicate key value violates unique constraint "rpm_updaterecord_digest_key"
37 postgresql-Thu.log:2021-07-01 05: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
11 postgresql-Thu.log:2021-07-01 08: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
15 postgresql-Thu.log:2021-07-01 11: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
4 postgresql-Thu.log:2021-07-01 12: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
4 postgresql-Thu.log:2021-07-01 15: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
4 postgresql-Thu.log:2021-07-01 16: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
99 postgresql-Tue.log:2021-06-29 10: ERROR: duplicate key value violates unique constraint "core_artifact_pkey"
164 postgresql-Tue.log:2021-06-29 11: ERROR: duplicate key value violates unique constraint "core_artifact_pkey"
705 postgresql-Tue.log:2021-06-29 11: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
185 postgresql-Tue.log:2021-06-29 12: ERROR: duplicate key value violates unique constraint "core_artifact_pkey"
913 postgresql-Tue.log:2021-06-29 12: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
25 postgresql-Tue.log:2021-06-29 13: ERROR: duplicate key value violates unique constraint "core_artifact_pkey"
1422 postgresql-Tue.log:2021-06-29 13: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
3 postgresql-Tue.log:2021-06-29 13: ERROR: duplicate key value violates unique constraint "rpm_distributiontree_header_version_release_n_5404b190_uniq"
539 postgresql-Tue.log:2021-06-29 13: ERROR: duplicate key value violates unique constraint "rpm_modulemddefaults_digest_key"
90 postgresql-Tue.log:2021-06-29 13: ERROR: duplicate key value violates unique constraint "rpm_packagecategory_digest_key"
87 postgresql-Tue.log:2021-06-29 13: ERROR: duplicate key value violates unique constraint "rpm_packageenvironment_digest_key"
389 postgresql-Tue.log:2021-06-29 13: ERROR: duplicate key value violates unique constraint "rpm_packagegroup_digest_key"
20 postgresql-Tue.log:2021-06-29 13: ERROR: duplicate key value violates unique constraint "rpm_packagelangpacks_digest_key"
133 postgresql-Tue.log:2021-06-29 13: ERROR: duplicate key value violates unique constraint "rpm_repometadatafile_data_type_checksum_relat_c9d7364a_uniq"
584 postgresql-Tue.log:2021-06-29 13: ERROR: duplicate key value violates unique constraint "rpm_updaterecord_digest_key"
259 postgresql-Tue.log:2021-06-29 14: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
338 postgresql-Tue.log:2021-06-29 15: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
1 postgresql-Wed.log:2021-06-30 06: ERROR: duplicate key value violates unique constraint "dynflow_coordinator_records_pkey"
1 postgresql-Wed.log:2021-06-30 07: ERROR: duplicate key value violates unique constraint "dynflow_coordinator_records_pkey"
1 postgresql-Wed.log:2021-06-30 16: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
5 postgresql-Wed.log:2021-06-30 17: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
1 postgresql-Wed.log:2021-06-30 17: ERROR: duplicate key value violates unique constraint "dynflow_coordinator_records_pkey"
1 postgresql-Wed.log:2021-06-30 17: ERROR: update or delete on table "core_reservedresource" violates foreign key constraint "core_taskreservedres_resource_id_ee0b7c62_fk_core_rese" on table "core_taskreservedresource"
4 postgresql-Wed.log:2021-06-30 18: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
8 postgresql-Wed.log:2021-06-30 19: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
11 postgresql-Wed.log:2021-06-30 20: ERROR: duplicate key value violates unique constraint "core_artifact_sha256_key"
root@hostname:/var/opt/rh/rh-postgresql12/lib/pgsql/data/log#