Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-5965

Segmentation fault in the latest 2.5 istio-proxy

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Blocker Blocker
    • OSSM 2.5.0
    • OSSM 2.5.0
    • Envoy
    • None
    • False
    • None
    • False
    • Hide
      • Install the latest OSSM metadata bundle ( openshift-service-mesh-istio-rhel8-operator-metadata:2.5.0-37 )
      • Install all required operators
      • Create smcp and namespaces from the attachments 
        oc create -f smcp.yaml
      • Install book info app 
        oc project bookinfo
oc create -f https://raw.githubusercontent.com/Maistra/istio/maistra-2.5/samples/bookinfo/platform/kube/bookinfo.yaml
oc create -f https://raw.githubusercontent.com/Maistra/istio/maistra-2.5/samples/bookinfo/networking/bookinfo-gateway.yaml
      • Create testssl pod 
        oc create -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: testssl
  namespace: bookinfo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: testssl
  template:
    metadata:
      labels:
        app: testssl
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - name: testssl
        image: quay.io/maistra/testssl:2.5
        command: ["tail", "-f", "/dev/null"]
EOF
      • Open istio-proxy log in different terminal 
         oc logs -f $(oc get pod -l app=productpage -o custom-columns=NAME:.metadata.name --no-headers) -c istio-proxy
      • Run testssl script inside that pod (and in the different terminal, look at istio-) 
        oc rsh $(oc get pod -l app=testssl -o custom-columns=NAME:.metadata.name --no-headers)
./testssl.sh -P -6 productpage:9080 || true
      • There will be Segmentation fault in istio-proxy
      Show
      Install the latest OSSM metadata bundle ( openshift-service-mesh-istio-rhel8-operator-metadata:2.5.0-37 ) Install all required operators Create smcp and namespaces from the attachments oc create -f smcp.yaml Install book info app oc project bookinfo oc create -f https: //raw.githubusercontent.com/Maistra/istio/maistra-2.5/samples/bookinfo/platform/kube/bookinfo.yaml oc create -f https: //raw.githubusercontent.com/Maistra/istio/maistra-2.5/samples/bookinfo/networking/bookinfo-gateway.yaml Create testssl pod oc create -f - <<EOF apiVersion: apps/v1 kind: Deployment metadata: name: testssl namespace: bookinfo spec: replicas: 1 selector: matchLabels: app: testssl template: metadata: labels: app: testssl spec: terminationGracePeriodSeconds: 0 containers: - name: testssl image: quay.io/maistra/testssl:2.5 command: [ "tail" , "-f" , "/dev/ null " ] EOF Open istio-proxy log in different terminal oc logs -f $(oc get pod -l app=productpage -o custom-columns=NAME:.metadata.name --no-headers) -c istio-proxy Run testssl script inside that pod (and in the different terminal, look at istio-) oc rsh $(oc get pod -l app=testssl -o custom-columns=NAME:.metadata.name --no-headers) ./testssl.sh -P -6 productpage:9080 || true There will be Segmentation fault in istio-proxy

      When I tried to run testssl against the service (productpage from the bookinfo example) with the latest istio-proxy container, the container failed with Segmentation fault.

      2024-02-20T09:32:07.727518Z	info	cache	returned workload trust anchor from cache	ttl=23h59m59.272485027s
[2024-02-20T09:33:45.512Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 8 - "-" "-" "-" "-" "-" - - 10.128.2.146:9080 10.129.3.147:53968 - -
2024-02-20T09:33:45.731817Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:104	Caught Segmentation fault, suspect faulting address 0x0	thread=28
2024-02-20T09:33:45.731857Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:91	Backtrace (use tools/stack_decode.py to get line numbers):	thread=28
2024-02-20T09:33:45.731860Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:92	Envoy version: ae3bbc4313b45af63777a2588388796d74221cfd/1.26.8-dev/OSSM 2.5.0-1/RELEASE/OpenSSL	thread=28
2024-02-20T09:33:45.732123Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:96	#0: __restore_rt [0x7f5e9fec9cf0]	thread=28
2024-02-20T09:33:45.743869Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:96	#1: Envoy::Extensions::TransportSockets::Tls::TlsContext::isCipherEnabled() [0x55ec4937e91a]	thread=28
2024-02-20T09:33:45.755545Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:96	#2: Envoy::Extensions::TransportSockets::Tls::ServerContextImpl::isClientEcdsaCapable() [0x55ec4937e8cf]	thread=28
2024-02-20T09:33:45.766882Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:96	#3: Envoy::Extensions::TransportSockets::Tls::ServerContextImpl::selectTlsContext() [0x55ec4937f03c]	thread=28
2024-02-20T09:33:45.766986Z	critical	envoy backtrace external/envoy/source/server/backtrace.h:98	#4: [0x7f5ea0db186e]thread=28
ConnectionImpl 0x55ec4e578340, connecting_: 0, bind_error_: 0, state(): Open, read_buffer_limit_: 1048576
socket_: 
  ListenSocketImpl 0x55ec4dfebb80, transport_protocol_: tls
  connection_info_provider_: 
    ConnectionInfoSetterImpl 0x55ec4e55ca60, remote_address_: 10.129.3.147:53974, direct_remote_address_: 10.129.3.147:53974, local_address_: 10.128.2.146:9080, server_name_: productpage
2024-02-20T09:33:46.811366Z	info	ads	ADS: "@" productpage-v1-7c5c65566c-l54hv.bookinfo-2 terminated
2024-02-20T09:33:46.811461Z	info	ads	ADS: "@" productpage-v1-7c5c65566c-l54hv.bookinfo-1 terminated
2024-02-20T09:33:46.811764Z	error	Envoy exited with error: signal: segmentation fault (core dumped)
2024-02-20T09:33:46.811895Z	error	error serving tap http server: http: Server closed

      It doesn't happen in the SMCPversion: v2.4 or in the previous v2.5 versions ( with the old Proxy). See reproducer.

      SMCP config, istio-proxy log and proxyConfig (istioctl proxy-config all productpage-v1...) files are in the attachments.
      I can also provide a cluster with that setup.

        1. productpage-v1-5f5cdbf975-9vnb2-istio-proxy(1).log
          4 kB
        2. proxyconfig
          397 kB
        3. smcp.yaml
          0.7 kB

              rhn-support-twalsh Tim Walsh
              mkralik@redhat.com Matej Kralik
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: