+++ This bug was initially created as a clone of Bug #2106935 +++

Description of problem:

The nmstate operator fails to install with error

"message": "no channel heads (entries not replaced by another entry) found in channel \"stable\" of package \"kubernetes-nmstate-operator\"",

Version-Release number of selected component (if applicable):

On a newly installed OCP 4.10.15 bare metal cluster

How reproducible:

Always. Deleting and reinstalling the operator does not correct the issue.

Steps to Reproduce:
1.Create the subscription and wait

Here is the customer's subscription

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"operators.coreos.com/v1alpha1","kind":"Subscription","metadata":{"annotations":{},"labels":

{"app.kubernet es.io/instance":"nerc-ocp-infra-cluster-scope","nerc.mghpcc.org/kustomized":"true"}

,"name":"kubernetes-nmstate-operator"
,"namespace":"openshift-nmstate"},"spec":{"channel":"stable","installPlanApproval":"Automatic","name":"kubernetes-nmstat
e-operator","source":"redhat-operators","sourceNamespace":"openshift-marketplace"}}
creationTimestamp: "2022-06-27T23:02:49Z"
generation: 3
labels:
app.kubernetes.io/instance: nerc-ocp-infra-cluster-scope
nerc.mghpcc.org/kustomized: "true"
operators.coreos.com/kubernetes-nmstate-operator.openshift-nmstate: ""
name: kubernetes-nmstate-operator
namespace: openshift-nmstate
resourceVersion: "61147251"
uid: f6920e77-4a12-40dd-8410-e7a9914aa4e1
spec:
channel: stable
installPlanApproval: Automatic
name: kubernetes-nmstate-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
startingCSV: kubernetes-nmstate-operator.4.10.0-202205311508
status:
catalogHealth:

• catalogSourceRef:
  apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  name: certified-operators
  namespace: openshift-marketplace
  resourceVersion: "48033282"
  uid: f619ca87-386c-4d16-a667-bb81c48d1801
  healthy: true
  lastUpdated: "2022-06-27T23:02:51Z"
• catalogSourceRef:
  apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  name: community-operators
  namespace: openshift-marketplace
  resourceVersion: "48034491"
  uid: d681196c-a6da-436a-a277-6d145b40672c
  healthy: true
  lastUpdated: "2022-06-27T23:02:51Z"
• catalogSourceRef:
  apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  name: redhat-marketplace
  namespace: openshift-marketplace
  resourceVersion: "48024056"
  uid: 8f344b25-8b5a-4e04-8f9a-20a16a91021a
  healthy: true
  lastUpdated: "2022-06-27T23:02:51Z"
• catalogSourceRef:
  apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  name: redhat-operators
  namespace: openshift-marketplace
  resourceVersion: "48034492"
  uid: eaba30bc-7e51-49c2-b084-4122e9c717be
  healthy: true
  lastUpdated: "2022-06-27T23:02:51Z"
  conditions:
• lastTransitionTime: "2022-06-27T23:02:51Z"
  message: all available catalogsources are healthy
  reason: AllCatalogSourcesHealthy
  status: "False"
  type: CatalogSourcesUnhealthy
• message: no channel heads (entries not replaced by another entry) found in channel
  "stable" of package "kubernetes-nmstate-operator"
  reason: ErrorPreventedResolution
  status: "True"
  type: ResolutionFailed
  currentCSV: kubernetes-nmstate-operator.4.10.0-202205311508
  installPlanGeneration: 1
  installPlanRef:
  apiVersion: operators.coreos.com/v1alpha1
  kind: InstallPlan
  name: install-h4dbm
  namespace: openshift-nmstate
  resourceVersion: "48044077"
  uid: 77dee7cc-5629-43a3-a0cb-99324ab8b70d
  installedCSV: kubernetes-nmstate-operator.4.10.0-202205311508
  installplan:
  apiVersion: operators.coreos.com/v1alpha1
  kind: InstallPlan
  name: install-h4dbm
  uuid: 77dee7cc-5629-43a3-a0cb-99324ab8b70d
  lastUpdated: "2022-07-01T18:10:24Z"
  state: AtLatestKnown

Actual results:

type: CatalogSourcesUnhealthy

• message: no channel heads (entries not replaced by another entry) found in channel
  "stable" of package "kubernetes-nmstate-operator"
  reason: ErrorPreventedResolution
  status: "True"
  type: ResolutionFailed

Expected results:

The operator installs correctly

Additional info:

— Additional comment from dguthrie@redhat.com on 2022-07-13 22:07:31 UTC —

Description of problem:

The nmstate operator fails to install with error

"message": "no channel heads (entries not replaced by another entry) found in channel \"stable\" of package \"kubernetes-nmstate-operator\"",

Version-Release number of selected component (if applicable):

On a newly installed OCP 4.10.15 bare metal cluster

How reproducible:

Always. Deleting and reinstalling the operator does not correct the issue.

Steps to Reproduce:
1.Create the subscription and wait

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
annotations:
name: kubernetes-nmstate-operator
namespace: openshift-nmstate
spec:
channel: stable
installPlanApproval: Automatic
name: kubernetes-nmstate-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
startingCSV: kubernetes-nmstate-operator.4.10.0-202205311508

Actual results:

type: CatalogSourcesUnhealthy

• message: no channel heads (entries not replaced by another entry) found in channel
  "stable" of package "kubernetes-nmstate-operator"
  reason: ErrorPreventedResolution
  status: "True"
  type: ResolutionFailed

Expected results:

The operator installs correctly

Additional info:

— Additional comment from dguthrie@redhat.com on 2022-07-20 22:38:21 UTC —

Confusingly, the only issue the appears to happening in the operation of the cluster is that argocd shows degraded resource. It's possible this is masking more operationally significant problems, but we haven't found any other networking issues on the cluster since some version of the nmstate operator did install. We are hoping to get a response before any other problems appear.

— Additional comment from cstabler@redhat.com on 2022-07-25 08:05:49 UTC —

Hello dguthrie@redhat.com,
could you provide the must-gathers so we can have to the other resources?

— Additional comment from dguthrie@redhat.com on 2022-07-25 17:49:33 UTC —

Hello @cstabler@redhat.com

Here is a link to the must-gather in support hub. It's much too large to upload here.

https://access.redhat.com/support/cases/#/case/03257697/discussion?attachmentId=a096R00002orxBhQAI

— Additional comment from roarora@redhat.com on 2022-08-01 09:23:29 UTC —

Another affected customer case.

Subscription :
-----------
spec:
channel: stable
installPlanApproval: Automatic
name: kubernetes-nmstate-operator
source: redhat-operators
sourceNamespace: openshift-marketplace
startingCSV: kubernetes-nmstate-operator.4.10.0-202207140916
status:
catalogHealth:

• catalogSourceRef:
  apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  name: certified-operators
  namespace: openshift-marketplace
  resourceVersion: "114039328"
  uid: 8fd4aa80-1fce-4cf9-b19a-18c5be55c8d4
  healthy: true
  lastUpdated: "2022-07-29T07:01:48Z"
• catalogSourceRef:
  apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  name: community-operators
  namespace: openshift-marketplace
  resourceVersion: "114046184"
  uid: 90ef809d-ee18-498b-b95f-3d5633b535ea
  healthy: true
  lastUpdated: "2022-07-29T07:01:48Z"
• catalogSourceRef:
  apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  name: redhat-marketplace
  namespace: openshift-marketplace
  resourceVersion: "114034591"
  uid: 9380b26a-bf1c-4ea1-9c97-4d152a75d287
  healthy: true
  lastUpdated: "2022-07-29T07:01:48Z"
• catalogSourceRef:
  apiVersion: operators.coreos.com/v1alpha1
  kind: CatalogSource
  name: redhat-operators
  namespace: openshift-marketplace
  resourceVersion: "114036727"
  uid: 85323cff-29cc-4ca0-aed7-589ca240d433
  healthy: true
  lastUpdated: "2022-07-29T07:01:48Z"
  conditions:
• lastTransitionTime: "2022-07-29T07:01:48Z"
  message: all available catalogsources are healthy
  reason: AllCatalogSourcesHealthy
  status: "False"
  type: CatalogSourcesUnhealthy
• message: no channel heads (entries not replaced by another entry) found in channel
  "stable" of package "kubernetes-nmstate-operator"
  reason: ErrorPreventedResolution
  status: "True"
  type: ResolutionFailedSubscriotion
  currentCSV: kubernetes-nmstate-operator.4.10.0-202207140916
  installPlanGeneration: 2
  installPlanRef:
  apiVersion: operators.coreos.com/v1alpha1
  kind: InstallPlan
  name: install-fvqgq
  namespace: openshift-nmstate
  resourceVersion: "114048697"
  uid: 833567a8-1e36-48ac-8618-8eab1ea956d9
  installedCSV: kubernetes-nmstate-operator.4.10.0-202207140916
  installplan:
  apiVersion: operators.coreos.com/v1alpha1
  kind: InstallPlan
  name: install-fvqgq
  uuid: 833567a8-1e36-48ac-8618-8eab1ea956d9
  lastUpdated: "2022-07-29T09:15:03Z"
  state: AtLatestKnown
  -----------

Must gather : https://attachments.access.redhat.com/hydra/rest/cases/03278246/attachments/d86b3d46-fa88-4c6a-908c-4b73ce9f23e2

— Additional comment from alosingh@redhat.com on 2022-08-03 16:38:22 UTC —

Hi Team,

Another customer is facing the same issue on 4.10.24.

Must gather: https://attachments.access.redhat.com/hydra/rest/cases/03281536/attachments/8f481003-f2a1-424c-85b5-66e028f64b34?usePresignedUrl=true

— Additional comment from amalykhi@redhat.com on 2022-08-08 12:54:33 UTC —

Polarion TC https://polarion.engineering.redhat.com/polarion/#/project/OSE/workitem?id=OCP-47088

— Additional comment from aos-team-art-private@bot.bugzilla.redhat.com on 2022-08-10 08:05:10 UTC —

Elliott changed bug status from MODIFIED to ON_QA.
This bug is expected to ship in the next 4.12 release.

— Additional comment from amalykhi@redhat.com on 2022-08-10 10:56:43 UTC —

There are no knmstate builds. The fix will be verified on OCP 4.11