-
Ticket
-
Resolution: Won't Do
-
Major
-
None
-
OSSM 2.6.6
-
None
-
False
-
-
False
-
-
A Service with 2 or more ports cannot be reached from the federated cluster, even though it is correctly exposed and imported correctly using ExposedServiceSet and ImportedServiceSet.
If the Service only defines one port, it can be accessed
❯ k exec -n mesh-sleep -ti sleep-57b97966d7-n6pdw -- curl -v nginx.red-webserver.svc.red-mesh-imports.local:8080
* About to connect() to nginx.red-webserver.svc.red-mesh-imports.local port 8080 (#0)
* Trying 240.241.0.2...
* Connected to nginx.red-webserver.svc.red-mesh-imports.local (240.241.0.2) port 8080 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: nginx.red-webserver.svc.red-mesh-imports.local:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< server: envoy
< date: Wed, 26 Mar 2025 12:35:33 GMT
< content-type: text/html
< content-length: 13
< last-modified: Tue, 25 Mar 2025 10:14:20 GMT
< etag: "67e281fc-d"
< accept-ranges: bytes
< x-envoy-upstream-service-time: 19
<
Hello world!
* Connection #0 to host nginx.red-webserver.svc.red-mesh-imports.local left intact
When a new port is added to the service, the traffic stops working:
apiVersion: v1
kind: Service
metadata:
labels:
app: nginx
app.kubernetes.io/component: nginx
app.kubernetes.io/instance: nginx
name: nginx
namespace: mesh-webserver
spec:
clusterIP: 172.30.8.145
clusterIPs:
- 172.30.8.145
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: 8080-tcp
port: 8080
protocol: TCP
targetPort: 8080
- name: 9999-tcp
port: 9999
protocol: TCP
targetPort: 8080
selector:
deployment: nginx
sessionAffinity: None
type: ClusterIP
status:
loadBalancer: {}
Now the service can't be accessed in either port:
❯ k exec -n mesh-sleep -ti sleep-57b97966d7-n6pdw -- curl -v nginx.red-webserver.svc.red-mesh-imports.local:8080 * About to connect() to nginx.red-webserver.svc.red-mesh-imports.local port 8080 (#0) * Trying 240.241.0.2... * Connected to nginx.red-webserver.svc.red-mesh-imports.local (240.241.0.2) port 8080 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: nginx.red-webserver.svc.red-mesh-imports.local:8080 > Accept: */* > < HTTP/1.1 503 Service Unavailable < content-length: 121 < content-type: text/plain < date: Wed, 26 Mar 2025 12:36:51 GMT < server: envoy < * Connection #0 to host nginx.red-webserver.svc.red-mesh-imports.local left intact upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure% ❯ k exec -n mesh-sleep -ti sleep-57b97966d7-n6pdw -- curl -v nginx.red-webserver.svc.red-mesh-imports.local:9999 * About to connect() to nginx.red-webserver.svc.red-mesh-imports.local port 9999 (#0) * Trying 240.241.0.2... * Connected to nginx.red-webserver.svc.red-mesh-imports.local (240.241.0.2) port 9999 (#0) > GET / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: nginx.red-webserver.svc.red-mesh-imports.local:9999 > Accept: */* > < HTTP/1.1 503 Service Unavailable < content-length: 121 < content-type: text/plain < date: Wed, 26 Mar 2025 12:37:02 GMT < server: envoy < * Connection #0 to host nginx.red-webserver.svc.red-mesh-imports.local left intact upstream connect error or disconnect/reset before headers. retried and the latest reset reason: remote connection failure%
The envoy configuration has been created (apparently) correct:
Listeners:
ADDRESSES PORT MATCH DESTINATION [...] 240.241.0.2 8080 Trans: raw_buffer; App: http/1.1,h2c Route: nginx.red-webserver.svc.red-mesh-imports.local:8080 240.241.0.2 8080 ALL Cluster: outbound|8080||nginx.red-webserver.svc.red-mesh-imports.local 240.241.0.2 9999 Trans: raw_buffer; App: http/1.1,h2c Route: nginx.red-webserver.svc.red-mesh-imports.local:9999 240.241.0.2 9999 ALL Cluster: outbound|9999||nginx.red-webserver.svc.red-mesh-imports.local
Routes:
NAME VHOST NAME DOMAINS MATCH VIRTUAL SERVICE [...] nginx.red-webserver.svc.red-mesh-imports.local:8080 nginx.red-webserver.svc.red-mesh-imports.local:8080 * /* nginx.red-webserver.svc.red-mesh-imports.local:9999 nginx.red-webserver.svc.red-mesh-imports.local:9999 * /*
Clusters:
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE [...] nginx.red-webserver.svc.red-mesh-imports.local 8080 - outbound EDS nginx.red-webserver.svc.red-mesh-imports.local 9999 - outbound EDS
Listeners:
ENDPOINT STATUS OUTLIER CHECK CLUSTER [...] 10.129.2.32:15443 HEALTHY OK outbound|8080||nginx.red-webserver.svc.red-mesh-imports.local 10.129.2.32:15443 HEALTHY OK outbound|9999||nginx.red-webserver.svc.red-mesh-imports.local
The istio-proxy logs in the pod that initiates the connection when the traffic works:
[2025-03-26T12:57:58.758Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 13 15 14 "-" "curl/7.29.0" "978148f3-c5e4-4b65-9efd-b9e46a0af555" "nginx.red-webserver.svc.red-mesh-imports.local:8080" "10.129.2.32:15443" outbound|8080||nginx.red-webserver.svc.red-mesh-imports.local 10.129.2.33:56358 240.241.0.2:8080 10.129.2.33:54384 - default [2025-03-26T12:58:40.833Z] "GET / HTTP/1.1" 200 - via_upstream - "-" 0 13 14 13 "-" "curl/7.29.0" "67c57b07-ac33-42b4-a18f-50fc8ad05c9a" "nginx.red-webserver.svc.red-mesh-imports.local:8080" "10.129.2.32:15443" outbound|8080||nginx.red-webserver.svc.red-mesh-imports.local 10.129.2.33:35552 240.241.0.2:8080 10.129.2.33:56938 - default
The logs in the same container when the traffic fails:
[2025-03-26T12:59:43.576Z] "GET / HTTP/1.1" 503 UF,URX upstream_reset_before_response_started{remote_connection_failure} - "-" 0 121 50 - "-" "curl/7.29.0" "38666438-81fc-46ed-9490-753ec7b385da" "nginx.red-webserver.svc.red-mesh-imports.local:8080" "10.129.2.32:15443" outbound|8080||nginx.red-webserver.svc.red-mesh-imports.local - 240.241.0.2:8080 10.129.2.33:40514 - default [2025-03-26T13:01:16.432Z] "GET / HTTP/1.1" 503 UF,URX upstream_reset_before_response_started{remote_connection_failure} - "-" 0 121 36 - "-" "curl/7.29.0" "502517ab-9cee-4033-8fe0-00eb3f1bfd70" "nginx.red-webserver.svc.red-mesh-imports.local:9999" "10.129.2.32:15443" outbound|9999||nginx.red-webserver.svc.red-mesh-imports.local - 240.241.0.1:9999 10.129.2.33:36342 - default