Application pods running in Istio Ambient Mesh mode on a worker node lose network connectivity within the mesh after the following sequence of events:

• A worker node is restarted.
• Application pods and the istio-cni DaemonSet pod restart on the same node.

This results in a service disruption for ambient-enabled workloads. This behaviour has been internally validated and is reproducible. Tested initially by rhn-gps-tbox https://gist.github.com/trevorbox/0bb8ec02f84dbf98e0a8c5d70c7e0947 and validated by me on OCP 4.20.1

The application pods lose connectivity because the necessary in-pod iptables rules required for Ambient Mesh redirection are missing or lost during the restart.This occurs because the Istio CNI agent feature designed to repair this state is disabled by default with the flag ambient.reconcileIptablesOnStartup, the istio-cni agent, upon startup, does not inspect or re-reconcile the iptables configuration of existing ambient-enabled pods on the node. This leaves the application pod in a state where it is enrolled in the mesh but lacks the required networking setup.

The solution or workaround for this is to enable the flag when the customer create the IstioCNI resource, I tested this solution here: https://gist.github.com/fjglira/bfa78a9019e222a87da4999a1c2f503a

We need update the relevant Red Hat documentation sections pertaining to Istio Ambient Mesh deployment and configuration to include a mandatory instruction:

• Customers must explicitly set the configuration flag ambient.reconcileIptablesOnStartup to true when deploying the Istio CNI in Ambient Mesh mode.

slack thread about the issue: https://redhat-internal.slack.com/archives/C019EPZ233P/p1764613967607979