-
Bug
-
Resolution: Unresolved
-
Undefined
-
None
-
None
-
False
-
None
-
False
-
-
We sometimes see this issue when using RHOAI with gitops. What basically happens when deploying with gitops is that you install both the modelmesh and the RHOAI subscriptions. RHOAI then creates an `SMCP` object without the version field set.
Now if this SMCP creation happens after the SMCP CRD is installed but before the webhooks (specifically the conversion webhooks), the installation of ServiceMesh will never complete successfully because the subscription will be stuck with the following error:
oc get subscription.operators.coreos.com -n openshift-operators servicemeshoperator -o yaml|grep -A2 "error" message: 'error validating existing CRs against new CRD''s schema for "servicemeshcontrolplanes.maistra.io": conversion webhook for maistra.io/v2, Kind=ServiceMeshControlPlane failed: invalid version: '
The Istio operator pod will log the following:
{{{"level":"error","ts":1730305106.8801115,"logger":"conversion-webhook","msg":"failed to convert","request":"cb2b4ecb-b0cc-482f-a96a-c2508e8d5c0d","error":"invalid version: ","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/istio-operator/app/vendor/github.com/go-logr/zapr/zapr.go:132\nsigs.k8s.io/controller-runtime/pkg/webhook/conversion.(*Webhook).ServeHTTP\n\t/remote-source/istio-operator/app/vendor/sigs.k8s.io/controller-runtime/pkg/webhook/conversion/conversion.go:76\nsigs.k8s.io/controller-runtime/pkg/webhook.instrumentedHook.InstrumentHandlerInFlight.func1\n\t/remote-source/istio-operator/app/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:40\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2136\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/remote-source/istio-operator/app/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:100\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2136\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2\n\t/remote-source/istio-operator/app/vendor/github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:76\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2136\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2514\nnet/http.serverHandler.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2938\nnet/http.(*conn).serve\n\t/usr/lib/golang/src/net/http/server.go:2009"}}}
Note, that this only happens when the version field in the SMCP is either empty or left out entirely. When set to a correct value, this issue does not present itself.
To reproduce this you can run the following on a fresh cluster:
curl -L https://acksyn.org/files/mesh-issue/smcp-crd-noconversion.yaml | oc apply -f -
oc new-project istio-system
sleep 5
curl -L https://acksyn.org/files/mesh-issue/smcp.yaml | oc apply -f -
cat > sub.yaml<<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: servicemeshoperator
namespace: openshift-operators
spec:
channel: stable
installPlanApproval: Automatic
name: servicemeshoperator
source: redhat-operators
sourceNamespace: openshift-marketplace
EOF
oc apply -f sub.yaml
# Now wait a few minutes and then see that the subscription will be in failed state:
oc get subscription.operators.coreos.com -n openshift-operators servicemeshoperator -o yaml |grep -C2 error
type: CatalogSourcesUnhealthy
- lastTransitionTime: "2024-11-06T08:34:38Z"
message: 'error validating existing CRs against new CRD''s schema for "servicemeshcontrolplanes.maistra.io":
conversion webhook for maistra.io/v2, Kind=ServiceMeshControlPlane failed: invalid
version: '
This basically installs the SMCP CRD (without the webhooks), create an SMCP CR without the version field. Then after installing the Istio subscription and waiting a bit the subscription itself will be in errored state due to an existing CR failing the conversion webhook.