Description of problem: 

After installing rhods 2.4.0 and creating DSC with KServe enabled, Kserve-controller-manager crashes when trying to apply InferenceService manifest to namespace.

Prerequisites (if any, like setup, operators/versions):

Openshift cluster 4.12

Operators:
Red Hat Openshift Pipelines - version: 1.12.2
Red Hat Openshift Service Mesh - 2.4.5-0
Red Hat Openshift Data Science - 2.4.0

Steps to Reproduce

1. Go to demo.redhat.com and order Openshift 4.12 workshop

2. Install operators:
Red Hat Openshift Pipelines - version: 1.12.2
Red Hat Openshift Service Mesh - 2.4.5-0
Red Hat Openshift Data Science - 2.4.0
 
3. Once RHODS is successfully installed, create Datascience cluster with components:
codeflare: Removed
dashboard: Managed
datasciencepipelines: Managed
kserve: Managed
modelmeshserving: Managed
ray: Removed
trustyai: Removed
workbenches: Managed
 
Wait for Datascience cluster to successfully come up.

 

4. Open RHODS workbenches and create a new Datascience project named "kserve-test".

5. Go back to the openshift console and go to the "kserve-test" project.

Apply the yaml manifests in openshift:

Serving runtime: please find attached

and the Inference Service: please find attached

 

7. InferenceService pods/services do not come up. Go to "redhat-ods-applications" namespace and observe that the `kserve-controller-manager` pod crashes with this error:

 

{"level":"error","ts":"2023-11-27T20:37:32Z","logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start", "kind":"Service.serving.knative.dev","error":"no matches for kind \"Service\" in version \"serving.knative.dev/v1\"", "stacktrace":"sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6 /pkg/source/source.go:143\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.4 /pkg/util/wait/wait.go:235\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.4/pkg/util/wait/wait.go:582 \nk8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.4/pkg/util/wait/wait.go:547 \nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/source/source.go:136"}

 

Actual results:

kserve-controller-manager crash loops and inference service fails to spin up

Expected results:

InferenceService to successful spin up.

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details:

N/A

Workaround:

None

Additional info: