Major Description: Customer installed Red Hat OpenShift Gitops Operator .
Issue looks like a bug
- Originally installed the operator on August 5, 2022, the version installed at that time was 1.6.0.
- Customer want to upgrade to 1.7.0.
The upgrade channel is set to "latest", which points to 1.7.0. Update approval is set to automatic.
Upgrade status is set to 'up to date' , but the currently installed version is 1.6.2, not 1.7.0.
- Suggested change channel path to v1.7 instead Latest but
Attempt to change the channnel to gitops-1.7, customer received this error when hitting Save:
Danger alert:An error occurred a: Operation cannot be fulfilled on subscriptions.operators.coreos.com "openshift-gitops-operator": the object has been modified; please apply your changes to the latest version and try again
$ oc get events LAST SEEN TYPE REASON OBJECT MESSAGE 2m43s Normal Pulling pod/gitlab-runner-controller-manager-66c9dc9749-d79gw Pulling image "registry.gitlab.com/gitlab-org/gl-openshift/gitlab-runner-operator/gitlab-runner-operator@sha256:ead726da6f75d80ff7b389dac853096d8680b9910cd5079d3aaf77a31846bf7b" 7m43s Normal BackOff pod/gitlab-runner-controller-manager-66c9dc9749-d79gw Back-off pulling image "registry.gitlab.com/gitlab-org/gl-openshift/gitlab-runner-operator/gitlab-runner-operator@sha256:ead726da6f75d80ff7b389dac853096d8680b9910cd5079d3aaf77a31846bf7b" 178m Warning FailedToUpdateEndpointSlices service/gitlab-runner-controller-manager-service Error updating Endpoint Slices for Service openshift-operators/gitlab-runner-controller-manager-service: failed to delete gitlab-runner-controller-manager-service-l99gj EndpointSlice for Service openshift-operators/gitlab-runner-controller-manager-service: endpointslices.discovery.k8s.io "gitlab-runner-controller-manager-service-l99gj" not found
$oc get csv openshift-gitops-operator.v1.6.2 Red Hat OpenShift GitOps 1.6.2 openshift-gitops-operator.v1.6.1-0.1666124167.p Succeeded
$ oc get ip NAME CSV APPROVAL APPROVED install-44wwd openshift-gitops-operator.v1.6.1-0.1666124167.p Automatic true <<< install-b56dm devspacesoperator.v3.2.0 Automatic true install-mrbdv servicemeshoperator.v2.2.3 Automatic true install-v7s65 gatekeeper-operator-product.v0.2.4-0.1666670065.p Automatic true install-xtp8b devworkspace-operator.v0.16.0 Automatic true
Also found this additional information when running `oc describe` on the subscription.
Conditions: Last Transition Time: 2022-09-09T02:24:53Z Message: all available catalogsources are healthy Reason: AllCatalogSourcesHealthy Status: False Type: CatalogSourcesUnhealthy Message: constraints not satisfiable: subscription openshift-gitops-operator requires at least one of redhat-operators/openshift-marketplace/latest/openshift-gitops-operator.v1.7.0, redhat-operators/openshift-marketplace/latest/openshift-gitops-operator.v1.6.3 or @existing/openshift-operatorsopenshift-gitops-operator.v1.6.2, redhat-operators/openshift-marketplace/latest/openshift-gitops-operator.v1.7.0, redhat-operators/openshift-marketplace/latest/openshift-gitops-operator.v1.6.3, @existing/openshift-operatorsopenshift-gitops-operator.v1.6.1-0.1666124167.p and @existing/openshift-operators//openshift-gitops-operator.v1.6.2 provide AppProject (argoproj.io/v1alpha1), clusterserviceversion openshift-gitops-operator.v1.6.1-0.1666124167.p exists and is not referenced by a subscription, subscription openshift-gitops-operator exists Reason: ConstraintsNotSatisfiable Status: True Type: ResolutionFailed
- As per error, looks like OLM issue so, Suggested customer to restart OLM but it doesn't work.
We do not see any clusterserviceversion resources for openshift-gitops-operator.v1.6.1-0.1666124167.p
Here, we can see when I look at the v1.6.2 operator.
$oc get clusterserviceversion -n openshift-gitops openshift-gitops-operator.v1.6.2 NAME DISPLAY VERSION REPLACES PHASE openshift-gitops-operator.v1.6.2 Red Hat OpenShift GitOps 1.6.2 openshift-gitops-operator.v1.6.1-0.1666124167.p Succeeded
I did find the 1.6.1 version referenced in an install plan.
$oc get installplan -n openshift-operators NAME CSV APPROVAL APPROVED install-44wwd openshift-gitops-operator.v1.6.1-0.1666124167.p Automatic true
Something interesting I discovered though is that the install plan listed for the openshift-gitops-operator, actually belongs to the gatekeeper-operator-product operator. That seems strange.
$oc get sub -n openshift-operators openshift-gitops-operator -o json | jq .status.installplan { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "InstallPlan", "name": "install-v7s65", "uuid": "a5402b3a-7c57-4b85-85cb-38bdb6baa7f3" } install-v7s65 gatekeeper-operator-product.v0.2.4-0.1666670065.p Automatic true
$oc get subs -n openshift-operators openshift-gitops-operator -o jsonpath='{.status.installplan}' | jq { "apiVersion": "operators.coreos.com/v1alpha1", "kind": "InstallPlan", "name": "install-v7s65", "uuid": "a5402b3a-7c57-4b85-85cb-38bdb6baa7f3" }
$oc get installplan -n openshift-operators -o wide NAME CSV APPROVAL APPROVED install-44wwd openshift-gitops-operator.v1.6.1-0.1666124167.p Automatic true install-b56dm devspacesoperator.v3.2.0 Automatic true install-mrbdv servicemeshoperator.v2.2.3 Automatic true install-v7s65 gatekeeper-operator-product.v0.2.4-0.1666670065.p Automatic true install-xtp8b devworkspace-operator.v0.16.0 Automatic true
