Loading...

Type: Bug
Resolution: Done-Errata
Priority: Blocker
Fix Version/s: RHOAI_2.8.0
Affects Version/s: RHOAI_2.8.0
Component/s: IDE
Labels:
- RC2
- RC3
- found_in_RC1

Story Points:
1
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Regression:
No
Test Blocker:
No
Git Pull Request:
https://github.com/opendatahub-io/kubeflow/pull/264, https://github.com/red-hat-data-services/kubeflow/pull/43
Intelligence Requested:
Market:

Sprint:
RHOAI IDE 2.9

Target Version:

RHOAI_2.8.0

Affects Testing:

Testable

SFDC Cases Links:
SFDC Cases Counter:
SFDC Cases Open:

The notebook-controller-deployment pod logs errors when a workbench via DS project is created and spinned-up.

Go to Data Science projects and create a new project.
Create a new workbench using Minimal image.

See following errors:

notebook-controller-deployment pod log excerpt

2024-03-04T17:18:21Z INFO controllers.Notebook Reconciliation loop started {"notebook": {"name":"def","namespace":"abc"}}
2024-03-04T17:18:21Z INFO controllers.Notebook Updating StatefulSet {"notebook": {"name":"def","namespace":"abc"}, "namespace": "abc", "name": "def"}
2024-03-04T17:18:21Z INFO controllers.Notebook Initializing Notebook CR Status {"notebook": {"name":"def","namespace":"abc"}}
2024-03-04T17:18:21Z INFO controllers.Notebook Calculating Notebook's containerState {"notebook": {"name":"def","namespace":"abc"}}
2024-03-04T17:18:21Z INFO controllers.Notebook Updating Notebook CR state: {"notebook": {"name":"def","namespace":"abc"}, "state": {"running":{"startedAt":"2024-03-04T17:18:08Z"}}}
2024-03-04T17:18:21Z INFO controllers.Notebook Calculating Notebook's Conditions {"notebook": {"name":"def","namespace":"abc"}}
2024-03-04T17:18:21Z INFO controllers.Notebook Updating Notebook CR Status {"notebook": {"name":"def","namespace":"abc"}, "status": {"conditions":[{"type":"Initialized","status":"True","lastProbeTime":"2024-03-04T17:18:21Z","lastTransitionTime":"2024-03-04T17:18:00Z"},{"type":"Ready","status":"True","lastProbeTime":"2024-03-04T17:18:21Z","lastTransitionTime":"2024-03-04T17:18:21Z"},{"type":"ContainersReady","status":"True","lastProbeTime":"2024-03-04T17:18:21Z","lastTransitionTime":"2024-03-04T17:18:21Z"},{"type":"PodScheduled","status":"True","lastProbeTime":"2024-03-04T17:18:21Z","lastTransitionTime":"2024-03-04T17:18:00Z"}],"readyReplicas":1,"containerState":{"running":{"startedAt":"2024-03-04T17:18:08Z"}}}}
2024-03-04T17:18:21Z INFO controllers.Notebook Reconciliation loop started {"notebook": {"name":"def","namespace":"abc"}}
2024-03-04T17:18:21Z INFO controllers.Notebook Updating StatefulSet {"notebook": {"name":"def","namespace":"abc"}, "namespace": "abc", "name": "def"}
2024-03-04T17:18:21Z INFO controllers.Notebook Initializing Notebook CR Status {"notebook": {"name":"def","namespace":"abc"}}
2024-03-04T17:18:21Z INFO controllers.Notebook Calculating Notebook's containerState {"notebook": {"name":"def","namespace":"abc"}}
2024-03-04T17:18:21Z INFO controllers.Notebook Updating Notebook CR state: {"notebook": {"name":"def","namespace":"abc"}, "state": {"running":{"startedAt":"2024-03-04T17:18:08Z"}}}
2024-03-04T17:18:21Z INFO controllers.Notebook Calculating Notebook's Conditions {"notebook": {"name":"def","namespace":"abc"}}
2024-03-04T17:18:21Z INFO controllers.Notebook Updating Notebook CR Status {"notebook": {"name":"def","namespace":"abc"}, "status": {"conditions":[{"type":"Initialized","status":"True","lastProbeTime":"2024-03-04T17:18:21Z","lastTransitionTime":"2024-03-04T17:18:00Z"},{"type":"Ready","status":"True","lastProbeTime":"2024-03-04T17:18:21Z","lastTransitionTime":"2024-03-04T17:18:21Z"},{"type":"ContainersReady","status":"True","lastProbeTime":"2024-03-04T17:18:21Z","lastTransitionTime":"2024-03-04T17:18:21Z"},{"type":"PodScheduled","status":"True","lastProbeTime":"2024-03-04T17:18:21Z","lastTransitionTime":"2024-03-04T17:18:00Z"}],"readyReplicas":1,"containerState":{"running":{"startedAt":"2024-03-04T17:18:08Z"}}}}
2024-03-04T17:18:21Z ERROR Reconciler error {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"def","namespace":"abc"}, "namespace": "abc", "name": "def", "reconcileID": "5e482b15-470d-429b-89c6-a30f0a08e1e9", "error": "Operation cannot be fulfilled on notebooks.kubeflow.org \"def\": the object has been modified; please apply your changes to the latest version and try again"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226

odh-notebook-controller-manager log excerpt

{"level":"info","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"Checking and mounting CA certificate bundle","notebook":"def","namespace":"abc"}
{"level":"info","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"ConfigMap found on the given Namespace","notebook":"def","namespace":"abc"}
{"level":"info","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"Validating certificates for ca-bundle.crt","notebook":"def","namespace":"abc"}
{"level":"error","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"Error validating certificates for odh-ca-bundle.crt","notebook":"def","namespace":"abc","error":"Warning: odh-ca-bundle.crt data is empty","stacktrace":"github.com/opendatahub-io/kubeflow/components/odh-notebook-controller/controllers.CheckAndMountCACertBundle\n\t/opt/rhods/odh-notebook-controller/controllers/notebook_webhook.go:325\ngithub.com/opendatahub-io/kubeflow/components/odh-notebook-controller/controllers.(*NotebookWebhook).Handle\n\t/opt/rhods/odh-notebook-controller/controllers/notebook_webhook.go:246\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/webhook.go:169\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/http.go:98\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1\n\t/remote-source/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2109\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1\n\t/remote-source/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:147\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2109\ngithub.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2\n\t/remote-source/deps/gomod/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:109\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2109\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2487\nnet/http.serverHandler.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2947\nnet/http.(*conn).serve\n\t/usr/lib/golang/src/net/http/server.go:1991"}
{"level":"info","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"Creating Network Policy","name":"def-ctrl-np"}
{"level":"info","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"Creating Network Policy","name":"def-oauth-np"}
{"level":"info","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"Creating Service Account","notebook":"def","namespace":"abc"}
{"level":"info","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"Creating OAuth Service","notebook":"def","namespace":"abc"}
{"level":"info","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"Creating OAuth Secret","notebook":"def","namespace":"abc"}
{"level":"info","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"Creating Route","notebook":"def","namespace":"abc"}
{"level":"info","ts":"2024-03-04T17:17:58Z","logger":"controllers.Notebook","msg":"Removing reconciliation lock","notebook":"def","namespace":"abc"}

The value of the `odh-ca-bundle.crt` in `odh-trusted-ca-bundle` ConfigMap is empty indeed. But - is this a valid state? In case it is, do we want to print error in the log about it? In case the value shouldn't be empty, what did I do wrong? Am I supposed to update this config manually everytime?

relates to

RHOAIENG-1848 Setup workbench to use the certs set by the platform certs manager