Description of problem:
Running incremental update of multiple CVs (something click-able when applying errata to multiple Hosts that belong to different CVs) can generate increental update with wrong content.

For the sake of simplicity, I will incrementally add just packages, but the same applies to errata.

Assume CV containing a small repo that was gradually published, such that:

• version 1.0 contains one package P1
• version 2.0 contains packages P1 and P2
• the repo itself contains also package P3

Now, adding P3 to both CVs altogether via an incremental update, we will always get versions 1.1 and 2.1 with same content - either the one corresponding to version 1.1 or to version 2.1. The outcome depends on ordering.

So, we can easily do an incremental update that ends up in much less content.

Version-Release number of selected component (if applicable):
Sat 6.13 or newer (I bet older versions as well)

How reproducible:
100%

Steps to Reproduce:
I wrote very limited reproducer. The original one is "we added an errata to two CCVs that shared a CV in different versions, and incremental updates of the component CV got already bad". A fix should treat this scenario as well (Composite CV, errata and not only packages).

1. Have in /var/tmp/sos_packages/ a few sos packages for a custom product:
sos-4.2-1.el8.noarch.rpm
sos-4.2-3.el8.noarch.rpm
sos-4.3-1.el8.noarch.rpm
sos-4.3-2.el8.noarch.rpm
sos-4.6.1-1.el8.noarch.rpm

2. Have reproducer script that follows the "Assume CV" above, just adds two packages to every CV version (for a clear evidence what incr.update did):

~~~
get_cv_version_id() {
cv=$1
vers=$2
hammer content-view version list --organization-id 1 --content-view $cv | grep $vers | awk '

{ print $1 }'
}

# create product and repo, add many sos packages there - iteratively, while creating CV versions for it
hammer product create --organization-id 1 --name sos_product
hammer repository create --organization-id 1 --product sos_product --name sos_repo --content-type yum

hammer repository upload-content --organization-id 1 --product sos_product --name sos_repo --content-type rpm --path /var/tmp/sos_packages/sos-4.2-1.el8.noarch.rpm
hammer repository upload-content --organization-id 1 --product sos_product --name sos_repo --content-type rpm --path /var/tmp/sos_packages/sos-4.2-3.el8.noarch.rpm

repoid=$(hammer repository info --organization-id 1 --product sos_product --name sos_repo | grep -m1 "^Id:" | awk '{ print $2 }')

hammer content-view create --organization-id 1 --name cv_base_sos --repository-ids ${repoid}
hammer content-view publish --organization-id 1 --name cv_base_sos # version 1.0 to contain just sos-4.2-1 and sos-4.2-3
cv_sos_vers1=$(get_cv_version_id cv_base_sos "1\.0")

hammer repository upload-content --organization-id 1 --product sos_product --name sos_repo --content-type rpm --path /var/tmp/sos_packages/sos-4.3-1.el8.noarch.rpm
hammer repository upload-content --organization-id 1 --product sos_product --name sos_repo --content-type rpm --path /var/tmp/sos_packages/sos-4.3-2.el8.noarch.rpm
hammer content-view publish --organization-id 1 --name cv_base_sos # version 2.0 to contain sos-4.2-1, sos-4.5.4-1, sos-4.3-1 and sos-4.3-2
cv_sos_vers2=$(get_cv_version_id cv_base_sos "2\.0")

hammer repository upload-content --organization-id 1 --product sos_product --name sos_repo --content-type rpm --path /var/tmp/sos_packages/sos-4.6.1-1.el8.noarch.rpm # repo contains also sos-4.6.1-1

# package ID of the last RPM, to incrementally add to both CV versions
pkgid=$(hammer package list --organization-id 1 --product sos_product --repository sos_repo | grep sos-4.6.1-1.el8.noarch.rpm | awk '{ print $1 }

')

cat > incremental_update.json << EOF
{"add_content": {
"package_ids": ["${pkgid}"]
},
"content_view_version_environments": [
{
"content_view_version_id": ${cv_sos_vers2},
"environment_ids": []
},
{
"content_view_version_id": ${cv_sos_vers1},
"environment_ids": []
}
],
"resolve_dependencies": true,
"organization_id": 1
}
EOF

curl -u admin:redhat -X POST -H "Accept: application/json" -H "Content-Type: application/json" -d @incremental_update.json "https://$(hostname -f)/katello/api/v2/content_view_versions/incremental_update?organization_id=1"

while [ $(hammer task list --search "label = Actions::Katello::ContentView::IncrementalUpdates AND state = running" | grep -c running) -gt 0 ]; do
sleep 5
done

su - postgres c "psql foreman -c \"SELECT major,minor,content_counts FROM katello_content_view_versions WHERE content_view_id = (SELECT id FROM katello_content_views WHERE name = 'cv_base_sos') ORDER BY major,minor;\"" | grep -e major -e "--" -e " rpm: "
~~~

3. Now, RESET the reproducer by removing both CVs and the product:
~~~
hammer content-view remove-from-environment --organization-id 1 --name cv_base_sos --lifecycle-environment Library
hammer content-view delete --organization-id 1 --name cv_base_sos
hammer product delete --organization-id 1 --name sos_product
~~~

4. Re-run the reproducer, but with swapped content_view_version_id, i.e.:

~~~
cat > incremental_update.json << EOF
{"add_content": {
"package_ids": ["${pkgid}"]
},
"content_view_version_environments": [
{
"content_view_version_id": ${cv_sos_vers1},
"environment_ids": []
},
{
"content_view_version_id": ${cv_sos_vers2},
"environment_ids": []
}
],
"resolve_dependencies": true,
"organization_id": 1
}
EOF
~~~

Actual results:
2. shows both incremental versions having the expected content of 2.1:

major | minor | content_counts
-----------------------------------
1 | 0 | — +

(we wanted to add just one RPM to 1.0, but 1.1 got more!)

4. shows both incremental versions having the expected content of 1.1:

major | minor | content_counts
-----------------------------------
1 | 0 | — +

(2.1 has less content than 2.0, see?)

Expected results:
2. and 4. to show:

major | minor | content_counts
-----------------------------------
1 | 0 | — +

Additional info:
Again, this is a very limited reproducer. A fix must cover composite CVs on top of this, and applying errata also.