Details
-
Bug
-
Resolution: Done
-
Critical
-
None
-
OSSM 2.4.5
-
None
-
False
-
None
-
False
-
Release Notes
-
Release Notes
-
In a cluster-wide deployment with discovery selectors and gateways in the control plane namespace, users must label control plane namespace to be matched with discovery selectors to avoid skipping Gateway configurations.
Description
When I was testing a migration from multi-tenant to cluster-wide mode, I noticed that a Gateway, which uses privileged port, e.g. 80, is skipped by istiod during building listeners. However, it only happens when `spec.meshConfig.discoverySelectors` are specified and the gateway is deployed in the namespace, which labels do not match discoverySelectors.
How to reproduce:
1. Create SMCP:
apiVersion: maistra.io/v2 kind: ServiceMeshControlPlane metadata: name: basic spec: mode: ClusterWide meshConfig: discoverySelectors: - matchLabels: istio-discovery: enabled
2. Deploy bookinfo and create a Gateway:
oc new-project bookinfo oc label namespace bookinfo istio-discovery=enabled oc label namespace bookinfo istio-injection=enabled oc apply -f https://raw.githubusercontent.com/maistra/istio/maistra-2.4/samples/bookinfo/platform/kube/bookinfo.yaml -n bookinfo oc apply -f https://raw.githubusercontent.com/maistra/istio/maistra-2.4/samples/bookinfo/networking/bookinfo-gateway.yaml -n bookinfo
And then you will see the following log in istiod:
2023-12-10T15:53:51.488810Z warn buildGatewayListeners: skipping privileged gateway port 80 for node istio-ingressgateway-75b47579b5-xmjch.istio-system as it is an unprivileged pod 2023-12-10T15:53:51.488815Z warn gateway has zero listeners for node istio-ingressgateway-75b47579b5-xmjch.istio-system
istioctl will show that the istio-ingressgateway has no listener for the created Gateway:
istioctl pc listeners deploy/istio-ingressgateway -n istio-system ADDRESSES PORT MATCH DESTINATION 0.0.0.0 15021 ALL Inline Route: /healthz/ready* 0.0.0.0 15090 ALL Inline Route: /stats/prometheus*
istioctl analyze warning:
istioctl analyze -n bookinfo
Warning [IST0162] (Gateway bookinfo/bookinfo-gateway) The gateway is listening on a target port (port 80) that is not defined in the Service associated with its workload instances (Pod selector istio=ingressgateway). If you need to access the gateway port through the gateway Service, it will not be available.
Additional notes:
1) `istioctl analyze` shows the warning also when discovery selectors are not specified, but the Gateway is configured as expected.
2) It seems that this issue only occurs on OCP, because it works fine on KinD with upstream Istio, but fails on OCP.
apiVersion: install.istio.io/v1alpha1 kind: IstioOperator metadata: name: test spec: profile: default meshConfig: accessLogFile: /dev/stdout discoverySelectors: - matchLabels: istio-discovery: enabled
Impact:
This issue may cause an outage during a migration from multi-tenant to cluster-wide mode, because Gateway resources will be rejected.
Reason:
When discovery selectors are enabled, istiod does not know the targetPort of the istio-ingressgateway, which does not match discovery selectors:
2024-03-04T12:48:17.241319Z debug buildGatewayListeners: gateways after merging: &{[{80 HTTP }] map[{80 HTTP }:0xc0039a6630] map[] map[] map[port:{number:80 protocol:"HTTP" name:"http"} hosts:"*":bookinfo/bookinfo-gateway] map[http.80:[port:{number:80 protocol:"HTTP" name:"http"} hosts:"*"]] map[] false map[80:map[80:{}]] map[]}
2024-03-04T12:48:17.241331Z warn buildGatewayListeners: skipping privileged gateway port 80 for node istio-ingressgateway-7c6fcf8c7-ttrh4.istio-system as it is an unprivileged pod
2024-03-04T12:48:17.241335Z warn gateway has zero listeners for node istio-ingressgateway-7c6fcf8c7-ttrh4.istio-system
2024-03-04T12:48:17.241363Z info ads LDS: PUSH request for node:istio-ingressgateway-7c6fcf8c7-ttrh4.istio-system resources:0 size:0B
2024-03-04T12:48:17.241407Z debug buildGatewayRoutes: gateways after merging: &{[{80 HTTP }] map[{80 HTTP }:0xc0039a6630] map[] map[] map[port:{number:80 protocol:"HTTP" name:"http"} hosts:"*":bookinfo/bookinfo-gateway] map[http.80:[port:{number:80 protocol:"HTTP" name:"http"} hosts:"*"]] map[] false map[80:map[80:{}]] map[]}
2024-03-04T12:48:17.241465Z warn Gateway missing for route http.8080. This is normal if gateway was recently deleted.
2024-03-04T12:48:17.241519Z info ads RDS: PUSH request for node:istio-ingressgateway-7c6fcf8c7-ttrh4.istio-system resources:1 size:13B cached:0/0
When the istio-ingressgateway matches discovery selectors, istiod knows the target port and the Gateway is matched correctly:
kubectl label namespace istio-system istio-discovery=enabled
2024-03-04T12:49:33.146297Z debug buildGatewayListeners: gateways after merging: &{[{8080 HTTP }] map[{8080 HTTP }:0xc004174c60] map[] map[] map[port:{number:80 protocol:"HTTP" name:"http"} hosts:"*":bookinfo/bookinfo-gateway] map[http.8080:[port:{number:80 protocol:"HTTP" name:"http"} hosts:"*"]] map[] false map[8080:map[80:{}]] map[]}
2024-03-04T12:49:33.146363Z debug buildGatewayListeners: marshaling listener "0.0.0.0_8080" with 1 filter chains
2024-03-04T12:49:33.146511Z debug attached HTTP filter with 5 http_filter options to listener "0.0.0.0_8080" filter chain 0
2024-03-04T12:49:33.146615Z info ads LDS: PUSH for node:istio-ingressgateway-7c6fcf8c7-ttrh4.istio-system resources:1 size:3.1kB
2024-03-04T12:49:33.150350Z debug buildGatewayRoutes: gateways after merging: &{[{8080 HTTP }] map[{8080 HTTP }:0xc004174c60] map[] map[] map[port:{number:80 protocol:"HTTP" name:"http"} hosts:"*":bookinfo/bookinfo-gateway] map[http.8080:[port:{number:80 protocol:"HTTP" name:"http"} hosts:"*"]] map[] false map[8080:map[80:{}]] map[]}
2024-03-04T12:49:33.150514Z info ads RDS: PUSH request for node:istio-ingressgateway-7c6fcf8c7-ttrh4.istio-system resources:1 size:2.4kB cached:0/0
2024-03-04T12:49:33.245739Z info ads Push debounce stable[4] 4 for config ServiceEntry/istio-system/istio-ingressgateway.istio-system.svc.cluster.local and 1 more configs: 100.318318ms since last change, 100.457631ms since last push, full=true
2024-03-04T12:49:33.246153Z info ads XDS: Pushing:2024-03-04T12:49:33Z/3 Services:6 ConnectedEndpoints:7 Version:2024-03-04T12:49:33Z/3
2024-03-04T12:49:33.246940Z info ads CDS: PUSH for node:istio-ingressgateway-7c6fcf8c7-ttrh4.istio-system resources:13 size:12.9kB cached:0/12
2024-03-04T12:49:33.247024Z info ads CDS: PUSH for node:reviews-v2-5fdd9c7b59-smwg4.bookinfo resources:16 size:13.7kB cached:6/12
2024-03-04T12:49:33.247113Z debug buildGatewayListeners: gateways after merging: &{[{8080 HTTP }] map[{8080 HTTP }:0xc004174c60] map[] map[] map[port:{number:80 protocol:"HTTP" name:"http"} hosts:"*":bookinfo/bookinfo-gateway] map[http.8080:[port:{number:80 protocol:"HTTP" name:"http"} hosts:"*"]] map[] false map[8080:map[80:{}]] map[]}
2024-03-04T12:49:33.247167Z debug buildGatewayListeners: marshaling listener "0.0.0.0_8080" with 1 filter chains
Workaround:
oc label namespace istio-system istio-discovery=enabled
Solution:
The simplest solution is not using unprivileged port as target ports in gateway Service. This approach requires to set `net.ipv4.ip_unprivileged_port_start=0` in the security context, and it's worth to note that it's already used in the gateway chart - used for gateway injection - so this seems to be safe solution.
Attachments
Issue Links
- mentioned on
