Seed generation fails due to:

seedimage   81m   SeedGenInProgress   failed to cleanup resources: timed out waiting for ACM CRD deletion

LCA logs show:

2024-11-13T18:12:16Z	INFO	controllers.SeedGenerator	Deleting ACM CRDs
2024-11-13T18:12:16Z	INFO	controllers.SeedGenerator	Deleting CRD clusterclaims.cluster.open-cluster-management.io
2024-11-13T18:12:16Z	INFO	controllers.SeedGenerator	Deleting CRD appliedmanifestworks.work.open-cluster-management.io
2024-11-13T18:12:16Z	INFO	controllers.SeedGenerator	Deleting CRD configurationpolicies.policy.open-cluster-management.io
2024-11-13T18:12:16Z	INFO	controllers.SeedGenerator	Deleting CRD policies.policy.open-cluster-management.io
2024-11-13T18:12:16Z	INFO	controllers.SeedGenerator	Deleting CRD operatorpolicies.policy.open-cluster-management.io
2024-11-13T18:12:16Z	INFO	controllers.SeedGenerator	Waiting until ACM CRDs are deleted
2024-11-13T18:27:18Z	ERROR	controllers.SeedGenerator	Seed generation failed	{"error": "failed to cleanup resources: timed out waiting for ACM CRD deletion"}
github.com/openshift-kni/lifecycle-agent/controllers.(*SeedGeneratorReconciler).Reconcile
	/remote-source/app/controllers/seedgen_controller.go:1003
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:119
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:316
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:227
2024-11-13T18:27:18Z	INFO	controllers.SeedGenerator	Finish reconciling SeedGen	{"name": {"name":"seedimage"}, "requeueRightAway": false}


The blocking CRD is stuck finalizing:

kind: CustomResourceDefinition
  name: appliedmanifestworks.work.open-cluster-management.io
status:
  - lastTransitionTime: "2024-11-19T23:47:21Z"
    message: CustomResource deletion is in progress
    reason: InstanceDeletionInProgress
    status: "True"
    type: Terminating

This has happened multiple times in QE regression tests. It is always the same CRD which gets stuck and blocks seed generation.

Spoke: OCP 4.16.23 & LCA 4.16.2
Hub: OCP 4.17.5  ACM 2.12.1 MCE 2.7.1

    Frequently but not consistently.

    1. Deploy hub cluster with OCP 4.17.5, ACM 2.12., MCE 2.7.1, 
    2. Deploy spoke cluster with 4.16.23 and LCA 4.16.2. QE uses GitOps/ZTP to deploy spoke in this test.
    3.Detach spoke cluster from ACM:
      3a. On hub cluster delete ArgoCD clusters and policies apps
      3b. On hub cluster delete managed cluster 
      3c. Wait for spoke to reboot
    4. Generate seed image from the spoke cluster.
    5. Seed generation fails with the error shown above.


    

seed generation fails due to blocking ACM CRD    

Seed generation completes successfully.