Uploaded image for project: 'Red Hat OpenShift AI Engineering'
  1. Red Hat OpenShift AI Engineering
  2. RHOAIENG-4159

[RHOAI 2.8RC1] Notebooks spinned up by Jupyter are being created under wrong namespace

XMLWordPrintable

    • Dashboard - General-2
    • Critical
    • Testable

      Looks like we're not able to spin up the Notebook with current RHOAI 2.8 RC1 build candidate.

      Go to Jupyter app, spin up e.g. Minimal Python notebook with default settings.
      The Starting server pop-up window shows up but nothing more happens.

      This can be seen in the controller pod:

      {"level":"info","ts":"2024-03-04T11:49:47Z","msg":"Starting EventSource","controller":"notebook","controllerGroup":"kubeflow.org","controllerKind":"Notebook","source":"kind source: *v1.Notebook"}
{"level":"info","ts":"2024-03-04T11:49:47Z","msg":"Starting EventSource","controller":"notebook","controllerGroup":"kubeflow.org","controllerKind":"Notebook","source":"kind source: *v1.Route"}
{"level":"info","ts":"2024-03-04T11:49:47Z","msg":"Starting EventSource","controller":"notebook","controllerGroup":"kubeflow.org","controllerKind":"Notebook","source":"kind source: *v1.ServiceAccount"}
{"level":"info","ts":"2024-03-04T11:49:47Z","msg":"Starting EventSource","controller":"notebook","controllerGroup":"kubeflow.org","controllerKind":"Notebook","source":"kind source: *v1.Service"}
{"level":"info","ts":"2024-03-04T11:49:47Z","msg":"Starting EventSource","controller":"notebook","controllerGroup":"kubeflow.org","controllerKind":"Notebook","source":"kind source: *v1.Secret"}
{"level":"info","ts":"2024-03-04T11:49:47Z","msg":"Starting EventSource","controller":"notebook","controllerGroup":"kubeflow.org","controllerKind":"Notebook","source":"kind source: *v1.NetworkPolicy"}
{"level":"info","ts":"2024-03-04T11:49:47Z","msg":"Starting Controller","controller":"notebook","controllerGroup":"kubeflow.org","controllerKind":"Notebook"}
{"level":"info","ts":"2024-03-04T11:49:47Z","logger":"controller-runtime.webhook","msg":"Serving webhook server","host":"","port":8443}
{"level":"info","ts":"2024-03-04T11:49:48Z","msg":"Starting workers","controller":"notebook","controllerGroup":"kubeflow.org","controllerKind":"Notebook","worker count":1}
{"level":"info","ts":"2024-03-04T12:16:23Z","logger":"controllers.Notebook","msg":"Checking and mounting CA certificate bundle","notebook":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications"}
{"level":"info","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"ConfigMap found on the given Namespace","notebook":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications"}
{"level":"info","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"Validating certificates for ca-bundle.crt","notebook":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications"}
{"level":"error","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"Error validating certificates for odh-ca-bundle.crt","notebook":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications","error":"Warning: odh-ca-bundle.crt data is empty","stacktrace":"github.com/opendatahub-io/kubeflow/components/odh-notebook-controller/controllers.CheckAndMountCACertBundle\n\t/opt/rhods/odh-notebook-controller/controllers/notebook_webhook.go:325\ngithub.com/opendatahub-io/kubeflow/components/odh-notebook-controller/controllers.(*NotebookWebhook).Handle\n\t/opt/rhods/odh-notebook-controller/controllers/notebook_webhook.go:246\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/webhook.go:169\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/...
{"level":"info","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"Creating Network Policy","name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser-ctrl-np"}
{"level":"info","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"Creating Network Policy","name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser-oauth-np"}
{"level":"info","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"Creating Service Account","notebook":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications"}
{"level":"info","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"Creating OAuth Service","notebook":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications"}
{"level":"info","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"Creating OAuth Secret","notebook":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications"}
{"level":"info","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"Creating Route","notebook":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications"}
{"level":"error","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"Unable to create the Route","notebook":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications","error":"Route.route.openshift.io \"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser\" is invalid: spec.host: Invalid value: \"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser-redhat-ods-applications.apps.ods-qe-psi-07.osp.rh-ods.com\": must be no more than 63 characters","stacktrace":"github.com/opendatahub-io/kubeflow/components/odh-notebook-controller/controllers.(*OpenshiftNotebookReconciler).reconcileRoute\n\t/opt/rhods/odh-notebook-controller/controllers/notebook_route.go:104\ngithub.com/opendatahub-io/kubeflow/components/odh-notebook-controller/controllers.(*OpenshiftNotebookReconciler).ReconcileOAuthRoute\n\t/opt/rhods/odh-notebook-controller/controllers/notebook_oauth.go:262\ngithub.com/opendatahub-io/kubeflow/components/odh-notebook-controller/controllers.(*OpenshiftNotebookReconciler).Reconcile\n\t/...
{"level":"error","ts":"2024-03-04T12:16:24Z","msg":"Reconciler error","controller":"notebook","controllerGroup":"kubeflow.org","controllerKind":"Notebook","Notebook":{"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications"},"namespace":"redhat-ods-applications","name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","reconcileID":"6bbe037e-bf7c-4d46-8192-2a443dc4475b","error":"Route.route.openshift.io \"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser\" is invalid: spec.host: Invalid value: \"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser-redhat-ods-applications.apps.ods-qe-psi-07.osp.rh-ods.com\": must be no more than 63 characters","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k...
{"level":"info","ts":"2024-03-04T12:16:24Z","logger":"controllers.Notebook","msg":"Creating Route","notebook":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"redhat-ods-applications"}

      Update for reopen:

      In the log above, there is a mention that the hostname is too long - longer than 63 characters. But we worked with this setup just fine in the past so I didn't understand why it stopped to work suddenly and though that it was my bad setup. Since I lost my cluster yesterday, I wasn't able to perform more checks peacefully. This morning, on a new cluster, I was able to reproduce this and play with it more so I was able to realize that the actual issue here is the change of the namespace under which the notebooks is created - it can also be seen from the hostname in the log above:

      on RHOAI 2.8RC1 - one notebook created for user ldap-admin2 (not sure why there is also ldap-admin1 though)
      $ oc get pods -n rhods-notebooks                                                                                            
No resources found in rhods-notebooks namespace.

$ oc get pods -n redhat-ods-applications | grep jupyter  
jupyter-nb-ldap-2dadmin1-0                                        2/2     Running     0             24m
jupyter-nb-ldap-2dadmin2-0                                        2/2     Running     0             9m39s
      on RHOAI 2.6.0.GA - one notebook created for htpasswd-cluster-admin-user
      $ oc get pods -n rhods-notebooks                                                                                                  
NAME                                             READY   STATUS    RESTARTS   AGE
jupyter-nb-htpasswd-2dcluster-2dadmin-2duser-0   2/2     Running   0          14h

$ oc get pods -n redhat-ods-applications | grep jupyter

        1. rhoai-2.8rc1-missing-upsage-data-connection-setting.png
          141 kB
          Jorge Garcia Oncins

            lferrnan@redhat.com Lucas Fernandez Aragon
            jstourac@redhat.com Jan Stourac
            RHOAI Dashboard
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

              Created:
              Updated:
              Resolved: