-
Bug
-
Resolution: Won't Do
-
Normal
-
None
-
4.12
-
None
-
Moderate
-
No
-
False
-
Problem Statement:
According to the documentation, The service CA certificate, which issues the service certificates, is valid for 26 months and is automatically rotated when there is less than 13 months of validity left. After rotation, the previous service CA configuration is still trusted until its expiration. This allows a grace period for all affected services to refresh their key material before the expiration.
In the customer environment, service-ca automatically renewed after 13 Months and clusters still have old service-ca within old pods.Customer running java application with three replicas, one pod running with new CA and other two with old CA.
After renewing the service-ca found an issue in application pods.
~~~
PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors; nested exception is javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors","level":"ERROR"}]}
~~~
Over a call we found that pods were having difficulty communicating with each other.
We tested this with the help of curl command and from that we can see it was working intermittently.
CURL COMMAND:
~~~
sh-4.4$ curl --cacert /srvssl/tls.crt https://js-5-to-6.dot-kustosz-5-to-6.svc:8443
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
sh-4.4$ curl --cacert /srvssl/tls.crt https://js-5-to-6.dot-kustosz-5-to-6.svc:8443
<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
</body>
</html>
~~~
Pods with Java during the start read a serivce-ca file. But if in the meantime openshift-service-serving-signer generate a new CA cert, they are not able to read it (only during a boot it's read) . That’s why they can not trust a new CA.
In our opinion, they should be some kind of trust relation between the CA.
~~~
Here you have a code from our entrypoint
if [[ -f /srvssl/tls.crt && -f /srvssl/tls.key ]]; then
show_msg "Populating keystore"
export KEYSTORE_P12=$HOME/keystore.p12
export KEYSTORE=$HOME/keystore.jks
export KEYSTORE_PASSWORD=${KEYSTORE_PASSWORD:-aaa}
export TRUSTSTORE=$HOME/truststore.jks
export TRUSTSTORE_PASSWORD=${TRUSTSTORE_PASSWORD:-aaa}
openssl pkcs12 -export -in /srvssl/tls.crt -inkey /srvssl/tls.key -certfile /srvssl/tls.crt -name "tls" -out $KEYSTORE_P12 -password pass:$KEYSTORE_PASSWORD
keytool -importkeystore -noprompt -srckeystore $KEYSTORE_P12 -destkeystore $KEYSTORE -storepass $KEYSTORE_PASSWORD -srcstorepass $KEYSTORE_PASSWORD -deststoretype JKS
fi
if [[ -f /etc/pki/ca-trust/extracted/java/cacerts ]]; then
show_msg "Copying system certs to truststore"
keytool -importkeystore -srckeystore /etc/pki/ca-trust/extracted/java/cacerts -srcstorepass $KEYSTORE_PASSWORD -destkeystore $TRUSTSTORE -deststorepass $KEYSTORE_PASSWORD
fi
if [[ -f /run/secrets/kubernetes.io/serviceaccount/service-ca.crt ]]; then
show_msg "Copying service certs to truststore"
CERTS=$(grep 'END CERTIFICATE' /run/secrets/kubernetes.io/serviceaccount/service-ca.crt | wc -l)
for N in $(seq 0 $(($CERTS - 1))); do
ALIAS="osh-$N"
cat /run/secrets/kubernetes.io/serviceaccount/service-ca.crt |
awk "n==$N
{ print }; /END CERTIFICATE/
{ n++ }" |
keytool -noprompt -import -trustcacerts -alias $ALIAS -keystore $TRUSTSTORE -storepass $TRUSTSTORE_PASSWORD
done
fi
~~~
Restarting the pod and accepting the CA in the pods is not acceptable from the customer side due to a major business impact.
We do have an article that explains the same situation.
https://access.redhat.com/solutions/7053504
I believe the certificate explanation in the documentation creates confusion here.
Need input for further investigation.
As per the customer, this statement should work with all the applications, there should be no manual injecting the certificates in the trust store.
