Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-9656

Support for Gateway API Inference Extension

XMLWordPrintable

    • Gateway API Inference Extension
    • False
    • Hide

      None

      Show
      None
    • True
    • Done
    • 0% To Do, 0% In Progress, 100% Done
    • Hide
      This release includes technology preview support for Kubernetes Gateway API Inference Extensions. These extensions build on Kubernetes Gateway API to provide inference-specific routing capabilities that optimize for self-hosted generative-AI workloads. This implementation has been backported into OpenShift Service Mesh from Istio 1.27

      Reference: https://gateway-api-inference-extension.sigs.k8s.io/
      Show
      This release includes technology preview support for Kubernetes Gateway API Inference Extensions. These extensions build on Kubernetes Gateway API to provide inference-specific routing capabilities that optimize for self-hosted generative-AI workloads. This implementation has been backported into OpenShift Service Mesh from Istio 1.27 Reference: https://gateway-api-inference-extension.sigs.k8s.io/

      The Gateway API Inference Extension (GIE) provides routing features for AI Inference with VLLM model serving via Gateway API. Support for GIE in OSSM is a hard requirement and blocker for supporting llm-d on OpenShift, as we will be using the native Gateway API support introduced in 4.19 (which is based on OSSM) for llm-d.

      Today, preliminary support for this exists in upstream Istio, but it's currently an alpha build that is released separately from the main releases. There is an upstream tracking issue for the work needed to release it, and there is an Istio RFC to get it fully baked in.

      The purpose of this issue is to follow and deliver on the upstream tracking issue to deliver GIE support into mainline Istio, but ultimately into OSSM.

          1.
          		 [upstream] Rebase experimental-gwapi-inference-extension on master Sub-task Closed Undefined Aslak Knutsen
          2.
          		 [upstream] InferencePool Controller alignment Sub-task Closed Undefined Aslak Knutsen
          3.
          		 [upstream] Gateway API Inference Extension v0.5.0 support Sub-task Closed Critical Aslak Knutsen
          4.
          		 [upstream] Test scripts and reports for conformance tests Sub-task Closed Minor Aslak Knutsen
          5.
          		 [upstream] resolve 1.27 backport merge conflict Sub-task Closed Normal Aslak Knutsen
          6.
          		 [upstream] avoid panic on invalid backend ref Sub-task Closed Undefined Aslak Knutsen
          7.
          		 [downstream] Collect Gateway API Inference Extension CRDs in istio-must-gather Sub-task Closed Undefined Aslak Knutsen
          8.
          		 Proxy not working, missing override_host extension Sub-task Closed Blocker Zuzana Miklánková

              aknutsen@redhat.com Aslak Knutsen
              rh-ee-sutt Shane Utt
              Aslak Knutsen, Daniel Grimm, Hayk Hovsepyan, Tim Walsh
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

                Created:
                Updated:
                Resolved: