Major After upgrading the Service Mesh from 1.0 to 2.0 some of the API calls on MTLS egress failed with error 503.
The customer is getting an error 503 when connecting to a specific endpoint.
Below are the logs from istio-ingresgateway-pod:
[2022-10-25T11:22:46.873Z] "GET /icare/v1.1/routes?date=20221025 HTTP/1.1" 503 UC upstream_reset_before_response_started\{connection_termination}
We noticed in istio-ingressgateway pods that the connection is failing because of the below errors.
2022-10-03T10:01:37.308716524Z 2022-10-03T10:01:37.308560Z debug envoy client [external/envoy/source/common/http/codec_client.cc:127] [C6993781] protocol error: http/1.1 protocol error: unsupported transfer encoding 2022-10-03T10:02:13.378502995Z 2022-10-03T10:02:13.377886Z debug envoy client [external/envoy/source/common/http/codec_client.cc:127] [C6994045] protocol error: http/1.1 protocol error: unsupported transfer encoding
Also, observed that the issue is caused because of key=Transfer-Encoding value=chunked being added twice. one is added by istio and another which we suspect is because of Tomcat/application.
2022-11-02T12:02:07.514402594Z 2022-11-02T12:02:07.514387Z trace envoy http [C4596785] completed header: key=Transfer-Encoding value=chunked 2022-11-02T12:02:07.514402594Z 2022-11-02T12:02:07.514391Z trace envoy http [C4596785] completed header: key=Content-Type value=application/json;charset=UTF-8 2022-11-02T12:02:07.514402594Z 2022-11-02T12:02:07.514394Z trace envoy http [C4596785] onHeadersCompleteBase 2022-11-02T12:02:07.514402594Z 2022-11-02T12:02:07.514395Z trace envoy http [C4596785] completed header: key=Transfer-Encoding value=chunked 2022-11-02T12:02:07.514413709Z 2022-11-02T12:02:07.514401Z debug envoy client [C4596785] Error dispatching received data: http/1.1 protocol error: unsupported transfer encoding
Also, found some similar issues:
https://github.com/istio/istio/issues/24753
https://github.com/istio/istio/issues/39706
