Pods do not trust the certs signed by previous CA, after automatic rotation of Service serving CA

4.11

100%

The cluster where this issue occured is younger than 26 months, but as per the official DOC, service-CA automatic rotation occurs after 13 months.

1. Stage a: before CA rotation

Pod A (old CA) -> Pod B (old CA) - OK

2. Stage b: after CA rotation, before any the pods have been restarted, they still use the old CA:

 Pod A (old CA) -> Pod B (old CA) - OK 

3. Stage c: after CA rotation and after pod B restarted

 Pod A (old CA) -> Pod B (new CA) - not OK!  

At this point, Pod A tries to call Pod B, but since Pod B have been restarted, it has a new CA cert that Pod A doesn't know of, so this request will fail

Call from Pod A to Pod B should happen without any TLS cert trust issue, as certificates signed by previous CA are still valid.