Task Description (Required)

Based on timings, momentary unavailability, or human ordering of inputs, it is possible for the reconciler to discover a running model but initially not be able to map model registry metadata to that model.

When a Kserve only discover occurs, the namespace and name of the KServe inference service, instead of the name of the Kubeflow Registered Model and Model Version.

If we later are able to correlate by accessing the model registry, we can end up with two entries for the same running model, as they'll have different keys/indexes in the location and storage services.

Not a breaking scenario, but conceivably one we would want to clean up.

Similarly, what happens if the kserve inference service is left running but the corresponding registered model etc. are deleted.  Again, perhaps an opportunity for migration of the keys/indexes in location/storage.

We will address this in the following way

• when we do not find any matching model registry information for a new kserve inferenceservice via a controller event, we will defer on publishing model information until the kserve inference service has reached a stable state, and is up and running with a route or service endpoint and all the status conditions ready; this typically takes minutes even for small models
• this will give the model registry controller ample time to connect information
• the model registry controller now adds well known labels to the kserve inferenceservice as part of the cross referencing it does; the presence of these label will allow our background polling to further delineate between kserve only vs. kubeflow+kserve
• lastly, similar to the use of the model registry k/v custom property pairs on the registered model and model version, we will allow platform engineers to define well know k/v pairs on the annotation map of the kserve inferenceservice CRD instance to add the additional metadata we would like to display in the backstage UI