Description of problem:
Removing imageContentSources from HostedCluster does not update IDMS for the cluster.
Version-Release number of selected component (if applicable):
Tested with 4.15.14
How reproducible:
100%
Steps to Reproduce:
1. add imageContentSources to HostedCluster
2. verify it is applied to IDMS
3. remove imageContentSources from HostedCluster
Actual results:
IDMS is not updated to remove imageDigestMirrors contents
Expected results:
IDMS is updated to remove imageDigestMirrors contents
Additional info:
Workaround, set imageContentSources=[]
- blocks
-
OCPBUGS-36766 Removing imageContentSources from HostedCluster does not update IDMS
-
- Closed
-
- is cloned by
-
-
- Closed
-
- links to
-
RHEA-2024:3718
OpenShift Container Platform 4.17.z bug fix update
[OCPBUGS-34820] Removing imageContentSources from HostedCluster does not update IDMS
Verified on 4.17.
Create an HC (here we use an ARO-HCP-compatible HC)
fxie-mac:hypershift fxie$ oc get hc -A
NAMESPACE NAME VERSION KUBECONFIG PROGRESS AVAILABLE PROGRESSING MESSAGE
clusters 989407ecec28e8fef4bf 4.17.0-0.nightly-2024-07-31-035751 989407ecec28e8fef4bf-admin-kubeconfig Completed True False The hosted control plane is available
fxie-mac:hypershift fxie$ hcp oc get idms cluster -o yaml
apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
creationTimestamp: "2024-07-31T10:52:28Z"
generation: 1
labels:
hypershift.openshift.io/managed: "true"
machineconfiguration.openshift.io/role: worker
name: cluster
resourceVersion: "1278"
uid: 2d3e1d51-bb58-4c49-97af-a249395a6d81
spec:
imageDigestMirrors: []
Patch HC so hc.spec.imageContentSources =
- mirrors:
- brew.registry.redhat.io
source: registry.redhat.io - mirrors:
- brew.registry.redhat.io
source: registry.stage.redhat.io - mirrors:
- brew.registry.redhat.io
source: registry-proxy.engineering.redhat.com
fxie-mac:hypershift fxie$ hcp oc get idms cluster -o yaml
apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
creationTimestamp: "2024-07-31T10:52:28Z"
generation: 2
labels:
hypershift.openshift.io/managed: "true"
machineconfiguration.openshift.io/role: worker
name: cluster
resourceVersion: "21050"
uid: 2d3e1d51-bb58-4c49-97af-a249395a6d81
spec:
imageDigestMirrors:
- mirrors:
- brew.registry.redhat.io
source: registry.redhat.io
- mirrors:
- brew.registry.redhat.io
source: registry.stage.redhat.io
- mirrors:
- brew.registry.redhat.io
source: registry-proxy.engineering.redhat.com
Edit HC, remove hc.spec.imageContentSources
fxie-mac:hypershift fxie$ hcp oc get idms cluster -o yaml
apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
creationTimestamp: "2024-07-31T10:52:28Z"
generation: 3
labels:
hypershift.openshift.io/managed: "true"
machineconfiguration.openshift.io/role: worker
name: cluster
resourceVersion: "21236"
uid: 2d3e1d51-bb58-4c49-97af-a249395a6d81
spec:
imageDigestMirrors: []
Juan Manuel Parrilla Madrid
added a comment - For the QE team:
In order to validate this, you need to:
- Deploy a normal hostedCluster with or without ImageContentSource in the HostedCluster object
- Add the ImageContentSource to the HostedCluster
- Load as KUBECONFIG the hostedCluster kubeconfig one and check that a new IDMS entry is added to the existent "cluster" one
- Load as KUBECONFIG the Management cluster kubeconfig one and now remove the ImageContentSource field from HostedCluster object
- Load as KUBECONFIG the hostedCluster kubeconfig one and check the IDMS entry has disappear from the HostedCluster.
Juan Manuel Parrilla Madrid
added a comment - For the QE team:
In order to validate this, you need to:
Deploy a normal hostedCluster with or without ImageContentSource in the HostedCluster object
Add the ImageContentSource to the HostedCluster
Load as KUBECONFIG the hostedCluster kubeconfig one and check that a new IDMS entry is added to the existent "cluster" one
Load as KUBECONFIG the Management cluster kubeconfig one and now remove the ImageContentSource field from HostedCluster object
Load as KUBECONFIG the hostedCluster kubeconfig one and check the IDMS entry has disappear from the HostedCluster.
Juan Manuel Parrilla Madrid
added a comment - Yes, you were right, I've been investigating the fix and the issue and I've submitted a new PR to fix both issues.
I will include both PRs in the backport path to 4.14 too.
Juan Manuel Parrilla Madrid
added a comment - Yes, you were right, I've been investigating the fix and the issue and I've submitted a new PR to fix both issues.
https://github.com/openshift/hypershift/pull/4412
I will include both PRs in the backport path to 4.14 too. That's weird, did you install a fresh HO and the deploy the cluster?
Yes. This error message is also found in the periodic-ci-openshift-hypershift-release-4.17-periodics-e2e-aks job triggered today. See https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-hypershift-release-4.17-periodics-e2e-aks/1815235649062572032/artifacts/e2e-aks/hypershift-azure-run-e2e/artifacts/TestCreateCluster/namespaces/e2e-clusters-4x6dh-example-58v8h/core/pods/logs/hosted-cluster-config-operator-59858b467-czxw7-hosted-cluster-config-operator.log
Juan Manuel Parrilla Madrid
added a comment - fxierh That's weird, did you install a fresh HO and the deploy the cluster?
Juan Manuel Parrilla Madrid
added a comment - fxierh That's weird, did you install a fresh HO and the deploy the cluster? I Installed a cluster with the latest Hypershift build and without specifying --image-content-sources. The installation succeeded yet I found this from HCCO logs:
{"level":"error","ts":"2024-07-17T17:13:42Z","msg":"Reconciler error","controller":"resources","object":{"name":""},"namespace":"","name":"","reconcileID":"6a071a7d-30f0-419b-bce7-735a2353256a","error":"failed to reconcile global configuration: failed to delete image digest mirror set: error deleting *v1.ImageDigestMirrorSet: imagedigestmirrorsets.config.openshift.io \"cluster\" is forbidden: deleting required imagedigestmirrorsets.config.openshift.io resource, named cluster, is not allowed","errorCauses":[{"error":"failed to reconcile global configuration: failed to delete image digest mirror set: error deleting *v1.ImageDigestMirrorSet: imagedigestmirrorsets.config.openshift.io \"cluster\" is forbidden: deleting required imagedigestmirrorsets.config.openshift.io resource, named cluster, is not allowed"}],"stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/hypershift/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:222"}
OpenShift Jira Bot
added a comment - Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified.
OpenShift Jira Bot
added a comment - Hi jparrill@redhat.com ,
Bugs should not be moved to Verified without first providing a Release Note Type("Bug Fix" or "No Doc Update") and for type "Bug Fix" the Release Note Text must also be provided. Please populate the necessary fields before moving the Bug to Verified. 
Since the problem described in this issue should be resolved in a recent advisory, it has been closed.
For information on the advisory (Moderate: OpenShift Container Platform 4.17.0 bug fix and security update), and where to find the updated files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:3718