New JH spawner (1.3) always tries to pull s2i-minimal-notebook:3.6

Description of problem:

The upcoming changes in 1.3 (RHODS-1585) seem to break the spawner. Whatever image the user tries to spawn will result in the following error:

2021-11-22T15:18:20Z [Normal] Pulling image 
"s2i-minimal-notebook:3.6"
2021-11-22T15:18:20Z [Warning] Failed to pull image 
"s2i-minimal-notebook:3.6": rpc error: code = Unknown desc = reading 
manifest 3.6 in docker.io/library/s2i-minimal-notebook: errors: denied: 
requested access to the resource is denied unauthorized: authentication 
required
2021-11-22T15:18:20Z [Warning] Error: ErrImagePull
2021-11-22T15:18:22Z [Normal] Back-off pulling image 
"s2i-minimal-notebook:3.6"
2021-11-22T15:18:22Z [Warning] Error: ImagePullBackOff
Spawn failed: pod/jupyterhub-nb-ldap-2duser4 did not start in 600 
seconds!

Note that each of the four default images will result in JH trying to pull s2i-minimal-notebook:3.6, as if the value is a default hardcoded somewhere.

The POST request, as captured from the browser dev console, instead contains the correct image tag. The user's CM is also correctly updated to reflect their last chosen options.

There is an additional issue highlighted in the first Warning message, which is the fact that OCP is trying to resolve the image in docker.io; this is a known behaviour, filed here: https://bugzilla.redhat.com/show_bug.cgi?id=2000216

However, the root cause is that the image being pulled does not exist in the internal registry of the cluster, where it's actually tagged as s2i-minimal-notebook:v0.0.15

Prerequisites (if any, like setup, operators/versions):

RHODS 1.2 with this PR applied on top. Reproduced both in PSI and OSD, both on OCP 4.9

Steps to Reproduce

1. Open JH spawner page
2. Try spawning any of the images

Actual results:

Faliure due to ErrImagePull/ImagePullBackOff. Image is not the correct one, since s2i-minimal-notebook:3.6 is always being pulled, regardless of chosen image.

Expected results:

Able to spawn any image.

Reproducibility (Always/Intermittent/Only Once):

Always

Build Details:

RHODS 1.2 with this PR applied on top. Reproduced both in PSI and OSD, both on OCP 4.9

Workaround:

No known workaround. Spawning is broken.

Additional info:

A possible solution from a conversation with vpavlin@redhat.com :

 
I think the solution is to not rely on https://github.com/opendatahub-io/jupyterhub-odh/blob/master/.jupyter/jupyterhub_config.py#L208, but actually load the image https://github.com/opendatahub-io/jupyterhub-odh/blob/master/.jupyter/jupyterhub_config.py#L231 here
So instead of using spawner.image https://github.com/opendatahub-io/jupyterhub-odh/blob/master/.jupyter/jupyterhub_config.py#L233
We would call https://github.com/opendatahub-io/jupyterhub-singleuser-profiles/blob/master/jupyterhub_singleuser_profiles/user.py#L104 and extract last_selected_image
And use it there