Uploaded image for project: 'OpenShift Service Mesh'
  1. OpenShift Service Mesh
  2. OSSM-6434

Impending issue with init containers that require an external call

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • OSSM 2.6.0, OSSM 3.0-TP1
    • Maistra, Sail Operator
    • None
    • False
    • None
    • False
    • Hide
      1. Install sail operator (or OSSM 2.6 I bet)
      2. Create istio and cni resources (if using OSSM3)
      3. Deploy an ingress gateway
      4. Deploy bookinfo application
      5. Amend one of the deployments (e.g. ratings) to use an init container that does an external call e.g. [1]
      6. Set the sa's scc to nonroot
      7. The pod will now go error/crash as it can't make the external call anymore. it only works when setting the uid to match the proxy e.g. 1000739999 [2]

      [1] example init container within a deployment

       

          spec:
      initContainers:
      - command:
        - sh
        - -c
        - whoami; curl -Is http://www.google.com
        image: registry.access.redhat.com/ubi9/ubi:latest
        imagePullPolicy: Always
        name: init-myservice
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          runAsUser: 1337

      [2] iptables example

       

      sh-5.1# crictl ps | grep ratings
35ca1c98af685       0757c4bc24dc9f51f39c7f3f9ef9ff467ae6e9e2d2dee989abc7836b54b86dae                                                         56 minutes ago      Running             istio-proxy                                   1                   85ce3e46d44f5       ratings-v1-7595c88c4f-7x6fw

sh-5.1# crictl inspect 35ca1c98af685  | jq .info.pid
28726

sh-5.1# nsenter -t 28726 -n iptables -L -t nat -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
  924 55440 ISTIO_INBOUND  tcp  --  any    any     anywhere             anywhere            Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   14   840 ISTIO_OUTPUT  tcp  --  any    any     anywhere             anywhere            Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         Chain ISTIO_INBOUND (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15008
    0     0 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15020
  240 14400 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15021
    0     0 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15090
  684 41040 ISTIO_IN_REDIRECT  tcp  --  any    any     anywhere             anywhere            Chain ISTIO_IN_REDIRECT (3 references)
 pkts bytes target     prot opt in     out     source               destination         
  685 41100 REDIRECT   tcp  --  any    any     anywhere             anywhere             redir ports 15006Chain ISTIO_OUTPUT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   120 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15020
    3   180 RETURN     all  --  any    lo      127.0.0.6            anywhere            
    1    60 ISTIO_IN_REDIRECT  tcp  --  any    lo      anywhere            !localhost            tcp dpt:!15008 owner UID match 1000739999
    0     0 RETURN     all  --  any    lo      anywhere             anywhere             ! owner UID match 1000739999
    8   480 RETURN     all  --  any    any     anywhere             anywhere             owner UID match 1000739999
    0     0 ISTIO_IN_REDIRECT  tcp  --  any    lo      anywhere            !localhost            tcp dpt:!15008 owner GID match 1000739999
    0     0 RETURN     all  --  any    lo      anywhere             anywhere             ! owner GID match 1000739999
    0     0 RETURN     all  --  any    any     anywhere             anywhere             owner GID match 1000739999
    0     0 RETURN     all  --  any    any     anywhere             localhost           
    0     0 ISTIO_REDIRECT  all  --  any    any     anywhere             anywhere            Chain ISTIO_REDIRECT (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 REDIRECT   tcp  --  any    any     anywhere             anywhere             redir ports 15001

      If you'd like all the relevant manifests for an easier reproducer let me know. 

       

       

      Show
      Install sail operator (or OSSM 2.6 I bet) Create istio and cni resources (if using OSSM3) Deploy an ingress gateway Deploy bookinfo application Amend one of the deployments (e.g. ratings) to use an init container that does an external call e.g. [1] Set the sa's scc to nonroot The pod will now go error/crash as it can't make the external call anymore. it only works when setting the uid to match the proxy e.g. 1000739999 [2] [1] example init container within a deployment     spec:       initContainers:       - command:         - sh         - -c         - whoami; curl -Is http: //www.google.com         image: registry.access.redhat.com/ubi9/ubi:latest         imagePullPolicy: Always         name: init-myservice         securityContext:           allowPrivilegeEscalation: false           capabilities:             drop:             - ALL           runAsUser: 1337 [2] iptables example   sh-5.1# crictl ps | grep ratings 35ca1c98af685       0757c4bc24dc9f51f39c7f3f9ef9ff467ae6e9e2d2dee989abc7836b54b86dae                                                         56 minutes ago      Running             istio-proxy                                   1                   85ce3e46d44f5       ratings-v1-7595c88c4f-7x6fw sh-5.1# crictl inspect 35ca1c98af685  | jq .info.pid 28726 sh-5.1# nsenter -t 28726 -n iptables -L -t nat -v Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)  pkts bytes target     prot opt in     out     source               destination            924 55440 ISTIO_INBOUND  tcp  --  any    any     anywhere             anywhere            Chain INPUT (policy ACCEPT 0 packets, 0 bytes)  pkts bytes target     prot opt in     out     source               destination         Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)  pkts bytes target     prot opt in     out     source               destination             14   840 ISTIO_OUTPUT  tcp  --  any    any     anywhere             anywhere            Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)  pkts bytes target     prot opt in     out     source               destination         Chain ISTIO_INBOUND (1 references)  pkts bytes target     prot opt in     out     source               destination              0     0 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15008     0     0 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15020   240 14400 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15021     0     0 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15090   684 41040 ISTIO_IN_REDIRECT  tcp  --  any    any     anywhere             anywhere            Chain ISTIO_IN_REDIRECT (3 references)  pkts bytes target     prot opt in     out     source               destination            685 41100 REDIRECT   tcp  --  any    any     anywhere             anywhere             redir ports 15006Chain ISTIO_OUTPUT (1 references)  pkts bytes target     prot opt in     out     source               destination              2   120 RETURN     tcp  --  any    any     anywhere             anywhere             tcp dpt:15020     3   180 RETURN     all  --  any    lo      127.0.0.6            anywhere                 1    60 ISTIO_IN_REDIRECT  tcp  --  any    lo      anywhere            !localhost            tcp dpt:!15008 owner UID match 1000739999     0     0 RETURN     all  --  any    lo      anywhere             anywhere             ! owner UID match 1000739999     8   480 RETURN     all  --  any    any     anywhere             anywhere             owner UID match 1000739999     0     0 ISTIO_IN_REDIRECT  tcp  --  any    lo      anywhere            !localhost            tcp dpt:!15008 owner GID match 1000739999     0     0 RETURN     all  --  any    lo      anywhere             anywhere             ! owner GID match 1000739999     0     0 RETURN     all  --  any    any     anywhere             anywhere             owner GID match 1000739999     0     0 RETURN     all  --  any    any     anywhere             localhost                0     0 ISTIO_REDIRECT  all  --  any    any     anywhere             anywhere            Chain ISTIO_REDIRECT (1 references)  pkts bytes target     prot opt in     out     source               destination              0     0 REDIRECT   tcp  --  any    any     anywhere             anywhere             redir ports 15001 If you'd like all the relevant manifests for an easier reproducer let me know.     

      Issue:

      For application pods that have a sidecar and use an init container to interact with an external service via a FQDN (a probable common one being Vault) this has always been blocked by default (expected behaviour) and has required a workaround, see [1].

      The workaround required the applications service account having the 'nonroot' scc and then defining 'runAsUser: 1337' like illustrated here [2] as by default all init containers are blocked to external services until the Istio sidecar is started. Configuring the uid to 1337 (the default for the sidecar proxy) allowed these external calls to work due to the pod iptables firewall rules configured by the CNI Plugin on initialisation which permit access to external services and DNS.

      However, after conducting tests with an early build of OSSM 3.0 (istio 1.21) this workaround no longer functions. Consequently, deployments with init containers configured using this workaround will break. This change seems attributable to fixes incorporated into Istio [3] and [4], which appear to have been included in Istio 1.20 (so will affect 2.6 as well). The workaround logic/mechanism remains operational but now requires setting the uid to match the random proxy's uid (e.g. 1000739999).

      Impact:

      The impact of this change is that the initcontainer must now know which UID the istio sidecar will run under as there is no consistent uid option as like before. This is not predictable and depends on many factors and can change when the application deployments change.

      References:

      1. https://istio.io/latest/docs/setup/additional-setup/cni/#compatibility-with-application-init-containers
      2. https://access.redhat.com/solutions/6653601#example-1337
      3. https://github.com/istio/istio/pull/47898
      4. https://github.com/istio/istio/pull/45394

              jsantana@redhat.com Jonh Wendell
              jforce1 James Force
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: