demo.redhat.com

order Openshift 4.12 workshop

Install operators:
Red Hat Openshift Pipelines - version: 1.12.2
Red Hat Openshift Service Mesh - 2.4.5-0
Red Hat Openshift Data Science - 2.4.0

Once RHODS is successfully installed, create Datascience cluster.

components:
  codeflare:
    managementState: Removed
  dashboard:
    managementState: Managed
  datasciencepipelines:
    managementState: Managed
  kserve:
    managementState: Managed
  modelmeshserving:
    managementState: Managed
  ray:
    managementState: Removed
  trustyai:
    managementState: Removed
  workbenches:
    managementState: Managed


Wait for Datascience cluster to successfully come up.

Open RHODS workbenches and create new Datascience project named "kserve-test".
Go back to openshift console and go to "kserve-test" project.

apply these yaml manifests:

apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  name: kserve-mlserver
spec:
  containers:
  - env:
    - name: MLSERVER_MODEL_IMPLEMENTATION
      value: '{{.Labels.modelClass}}'
    - name: MLSERVER_HTTP_PORT
      value: "8080"
    - name: MLSERVER_GRPC_PORT
      value: "9000"
    - name: MODELS_DIR
      value: /mnt/models
    image: docker.io/seldonio/mlserver:1.0.0
    name: kserve-container
    resources:
      limits:
        cpu: "1"
        memory: 2Gi
      requests:
        cpu: "1"
        memory: 2Gi
  protocolVersions:
  - v2
  supportedModelFormats:
  - autoSelect: true
    name: sklearn
    version: "0"
  - autoSelect: true
    name: xgboost
    version: "1"
  - autoSelect: true
    name: lightgbm
    version: "3"
  - autoSelect: true
    name: mlflow
    version: "1"

apiVersion: "serving.kserve.io/v1beta1"
kind: "InferenceService"
metadata:
  name: "sklearn-irisv2"
  annotations:
    serving.knative.openshift.io/enablePassthrough: "true"
    sidecar.istio.io/inject: "true"
    sidecar.istio.io/rewriteAppHTTPProbers: "true"
spec:
  predictor:
    model:
      modelFormat:
        name: sklearn
      runtime: kserve-mlserver
      storageUri: "gs://seldon-models/sklearn/mms/lr_model"


InferenceService pods/services do not come up. Go to "redhat-ods-applications" namespace and observe that the `kserve-controller-manager` pod crashes wit this error:

`Error "failed calling webhook "inferenceservice.kserve-webhook-server.defaulter": failed to call webhook: Post "https://kserve-webhook-server-service.redhat-ods-applications.svc:443/mutate-serving-kserve-io-v1beta1-inferenceservice?timeout=10s": no endpoints available for service "kserve-webhook-server-service"" for field "undefined".`

{"level":"error","ts":"2023-11-27T20:37:32Z","logger":"controller-runtime.source","msg":"if kind is a CRD, it should be installed before calling Start","kind":"Service.serving.knative.dev","error":"no matches for kind \"Service\" in version \"serving.knative.dev/v1\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/source/source.go:143\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.4/pkg/util/wait/wait.go:235\nk8s.io/apimachinery/pkg/util/wait.poll\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.4/pkg/util/wait/wait.go:582\nk8s.io/apimachinery/pkg/util/wait.PollImmediateUntilWithContext\n\t/go/pkg/mod/k8s.io/apimachinery@v0.26.4/pkg/util/wait/wait.go:547\nsigs.k8s.io/controller-runtime/pkg/source.(*Kind).Start.func1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/source/source.go:136"}