Description

Summary of the Use Case

Customers require the ability to configure multiple mTLS certificates for Red Hat OpenShift AI (RHOAI) InferenceServices, with each mTLS certificate mapped to a dedicated namespace and subdomain. This request originates from a need to isolate traffic for multiple external application clients (multi-tenancy), where each client’s InferenceServices are hosted in a specific namespace. By assigning unique wildcard certificates (e.g., *.namespaceA.apps.cluster.domain) per namespace, external clients can be managed and secured independently.

Key Points from Support Case

• Issue Description:
  • Currently, RHOAI only supports a single mTLS certificate for all InferenceServices across namespaces. The customer wants to enable multiple mTLS certs, one per external client/namespace, to achieve stricter traffic isolation and access control.
• Symptoms / Error Messages:
  • No specific error messages were provided. The main concern is lack of out-of-the-box support for multiple subdomain routing and certificate management at the Knative and Istio Gateway levels.
• Impact:
  • Inability to neatly segregate InferenceService traffic for different external clients, complicating resource quota management, compliance requirements, and ongoing operational support.
  • Potential security risk if all tenants must share a single mTLS certificate

Current Discussion Highlights

• Proposed Solutions:
  • Configure multiple hosts entries within the same Istio Gateway CR, each referencing a unique credentialName (mTLS certificate).
  • Potentially leverage Knative DomainMapping or OpenShift IngressController sharding for more control.

  • # example
    apiVersion: networking.istio.io/v1beta1
    kind: Gateway
    metadata:
      name: knative-ingress-gateway
      namespace: knative-serving
    spec:
      selector:
        knative: ingressgateway 
      servers:
        - port:
            name: https-namespaceA
            number: 443
            protocol: HTTPS
          hosts:
            - '*.namespaceA.apps.rits.fmaas.res.ibm.com'
          tls:
            mode: MUTUAL
            credentialName: mtls-cert-A
        - port:
            name: https-namespaceB
            number: 443
            protocol: HTTPS
          hosts:
            - '*.namespaceB.apps.rits.fmaas.res.ibm.com'
          tls:
            mode: MUTUAL
            credentialName: mtls-cert-B 

• Feasibility and Priority:
  • Engineers and CEE identified that this configuration is not currently documented or fully supported.
• Technical Constraints:
  • KServe/Knative automatically generates routes in a single system namespace (e.g., istio-system or knative-serving), which complicates namespace-specific or subdomain-specific routing.
  • Using wildcard subdomains (e.g., *.namespaceA.apps.cluster.domain) requires ensuring the underlying Gateway and Ingress controllers can route traffic solely to resources in that namespace.

Acceptance Criteria

• Multiple mTLS Certificates
  • The RHOAI serving components can be configured to utilize multiple mTLS secrets (one per subdomain) without manual patching of system-managed resources.
• Automated or Well-Documented Setup
  • Clear instructions or automated mechanisms exist to generate and map each subdomain/certificate pair to the correct namespace in the Gateway.
• Test Coverage & Validation
  • Engineering test plan covers scenarios with multiple external clients, each assigned a dedicated namespace, and verifies successful mTLS connections per subdomain.

References & Links

• Slack Thread (internal)
• Support Case: 04025889{}{{}}
• Knative DomainMapping and wildcard domains:
  • https://github.com/knative/serving/issues/11321
  • Official docs mention domainMapping for single host names, not wildcard subdomains.
• OpenShift Ingress Sharding:
  • https://docs.openshift.com/serverless/1.34/knative-serving/external-ingress-routing/using-serving-with-ingress-sharding.html