Loading...

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: 2.1.0.GA
Affects Version/s: 1.2.0.GA, 1.5.0.GA
Component/s: None
Labels:
None

Blocked:
False
Ready:
False
GSS Priority:
Release Note Text:
Undefined
Target Release:

2.1.0.GA
Workaround:

Workaround Exists
Workaround Description:

Hide

a workaround is to delete the <cluster>-entity-operator-certs, <cluster>-kafka-brokers, and <cluster>-zookeeper-nodes certificates and then restart their pods

Show
a workaround is to delete the <cluster>-entity-operator-certs, <cluster>-kafka-brokers, and <cluster>-zookeeper-nodes certificates and then restart their pods
Market:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Zookeeper, Kafka, and EntityOperaotor certs are not renewed when to use an own cluster ca cert
- Eventually, these Zookeeper, Kafka, and EntityOperaotor certificates will expire and fail to communicate between these Kafka components[1].
- From reading our official AMQ Streams and Strimzi documentation, it appears that we do not provide a way to update these certificates when to use an own cluster ca cert.
- By the way, "Installing your own CA certificates[2]" tells us to set clusterCa.generateCertificateAuthority to false, I'm always testing with clusterCa.generateCertificateAuthority: false. And I tested with AMQ Streams 1.2 and 1.5.

FYI, the following steps did NOT renew Zookeeper, Kafka, and EntityOperaotor certificates.
- Following "Renewing your own CA certificates" procedure [3]
  - Zookeeper, Kafka, and EntityOperaotor certificates will not be updated.
- Using "strimzi.io/force-renew=true" annotation [4]
  - This annotation does not seem to work with clusterCa.generateCertificateAuthority: false.

There are a few other things that I noticed during testing that are potential bugs.
- "clusterCa.validityDays" option is used for Zookeeper, Kafka, and EntityOperaotor certificates even if the users use their own cluster ca cert.
  - There is a gap between the validity days of the cluster ca and the validity days of Zookeeper, Kafka, and EntityOperaotor certificates.
  - Zookeeper, Kafka, and EntityOperaotor certificates may expire before the cluster ca certificate without the user being aware of it.
- Using "strimzi.io/force-renew=true" annotation[4] does not seem to work with clusterCa.generateCertificateAuthority: false.
  - As far as I read the documentation, strimzi.io/force-renew=true annotation must work even with clusterCa.generateCertificateAuthority: false. This behavior may also be a bug.

[1] fail to communicate between these Kafka components

my-cluster-zookeeper-2 zookeeper 2020-10-12 10:47:27,802 WARN Exception caught (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-1]
my-cluster-zookeeper-2 zookeeper io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
my-cluster-zookeeper-2 zookeeper        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
my-cluster-zookeeper-2 zookeeper        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
my-cluster-zookeeper-2 zookeeper        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
my-cluster-zookeeper-2 zookeeper        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
my-cluster-zookeeper-2 zookeeper        at java.lang.Thread.run(Thread.java:748)
my-cluster-zookeeper-2 zookeeper Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1566)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:545)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:819)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:783)
my-cluster-zookeeper-2 zookeeper        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:626)
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281)
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1340)
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1235)
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1282)
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:498)
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:437)
my-cluster-zookeeper-2 zookeeper        ... 17 more
my-cluster-zookeeper-2 zookeeper Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.Alerts.getSSLException(Alerts.java:198)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1729)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:333)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:325)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:2055)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:233)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1082)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.Handshaker$1.run(Handshaker.java:1015)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.Handshaker$1.run(Handshaker.java:1012)
my-cluster-zookeeper-2 zookeeper        at java.security.AccessController.doPrivileged(Native Method)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1504)
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1510)
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1524)
my-cluster-zookeeper-2 zookeeper        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1408)
my-cluster-zookeeper-2 zookeeper        ... 21 more
my-cluster-zookeeper-2 zookeeper Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed
my-cluster-zookeeper-2 zookeeper        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:380)
my-cluster-zookeeper-2 zookeeper        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:273)
my-cluster-zookeeper-2 zookeeper        at sun.security.validator.Validator.validate(Validator.java:262)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:287)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(X509TrustManagerImpl.java:138)
my-cluster-zookeeper-2 zookeeper        at org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:88)
my-cluster-zookeeper-2 zookeeper        at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:2042)
my-cluster-zookeeper-2 zookeeper        ... 30 more
my-cluster-zookeeper-2 zookeeper Caused by: java.security.cert.CertPathValidatorException: validity check failed
my-cluster-zookeeper-2 zookeeper        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:135)
my-cluster-zookeeper-2 zookeeper        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:233)
my-cluster-zookeeper-2 zookeeper        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:141)
my-cluster-zookeeper-2 zookeeper        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:80)
my-cluster-zookeeper-2 zookeeper        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:292)
my-cluster-zookeeper-2 zookeeper        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:375)
my-cluster-zookeeper-2 zookeeper        ... 37 more
my-cluster-zookeeper-2 zookeeper Caused by: java.security.cert.CertificateNotYetValidException: NotBefore: Tue Oct 13 10:12:39 GMT 2020
my-cluster-zookeeper-2 zookeeper        at sun.security.x509.CertificateValidity.valid(CertificateValidity.java:273)
my-cluster-zookeeper-2 zookeeper        at sun.security.x509.X509CertImpl.checkValidity(X509CertImpl.java:677)
my-cluster-zookeeper-2 zookeeper        at sun.security.provider.certpath.BasicChecker.verifyValidity(BasicChecker.java:190)
my-cluster-zookeeper-2 zookeeper        at sun.security.provider.certpath.BasicChecker.check(BasicChecker.java:144)
my-cluster-zookeeper-2 zookeeper        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:125)
my-cluster-zookeeper-2 zookeeper        ... 42 more


my-cluster-kafka-1 kafka 2020-10-12 10:48:34,285 INFO Opening socket connection to server localhost/127.0.0.1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn) [main-SendThread(localhost:2181)]
my-cluster-kafka-1 tls-sidecar 2020.10.12 10:48:34 LOG5[1:139688013022976]: Service [zookeeper-2181] accepted connection from 127.0.0.1:57418
my-cluster-kafka-1 kafka 2020-10-12 10:48:34,286 INFO Socket connection established, initiating session, client: /127.0.0.1:57418, server: localhost/127.0.0.1:2181 (org.apache.zookeeper.ClientCnxn) [main-SendThread(localhost:2181)]
my-cluster-kafka-1 tls-sidecar 2020.10.12 10:48:34 LOG5[1:139688013022976]: connect_blocking: connected 172.25.42.19:2181
my-cluster-kafka-1 tls-sidecar 2020.10.12 10:48:34 LOG5[1:139688013022976]: Service [zookeeper-2181] connected remote server from 10.116.0.23:46852
my-cluster-kafka-1 tls-sidecar 2020.10.12 10:48:34 LOG5[1:139688013022976]: Certificate accepted: depth=1, /C=JP/ST=Tokyo/L=Shibuya-ku/O=redhatexample/OU=cee/CN=amqstreams.redhatexample.com
my-cluster-kafka-1 tls-sidecar 2020.10.12 10:48:34 LOG5[1:139688013022976]: Certificate accepted: depth=0, /O=io.strimzi/CN=my-cluster-zookeeper
my-cluster-zookeeper-1 zookeeper 2020-10-12 10:48:34,340 ERROR Unsuccessful handshake with session 0x0 (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-7-2]
my-cluster-kafka-1 tls-sidecar 2020.10.12 10:48:34 LOG3[1:139688013022976]: SSL_connect: 14094416: error:14094416:SSL routines:ssl3_read_bytes:sslv3 alert certificate unknown
my-cluster-kafka-1 tls-sidecar 2020.10.12 10:48:34 LOG5[1:139688013022976]: Connection reset: 0 byte(s) sent to SSL, 0 byte(s) sent to socket
my-cluster-kafka-1 kafka 2020-10-12 10:48:34,347 WARN Session 0x0 for server localhost/127.0.0.1:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn) [main-SendThread(localhost:2181)]
my-cluster-kafka-1 kafka java.io.IOException: Connection reset by peer
my-cluster-kafka-1 kafka        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
my-cluster-kafka-1 kafka        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
my-cluster-kafka-1 kafka        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
my-cluster-kafka-1 kafka        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
my-cluster-kafka-1 kafka        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:377)
my-cluster-kafka-1 kafka        at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:75)
my-cluster-kafka-1 kafka        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:363)
my-cluster-kafka-1 kafka        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)

[2] Installing your own CA certificates
https://access.redhat.com/documentation/en-us/red_hat_amq/7.7/html-single/using_amq_streams_on_openshift/index#installing-your-own-ca-certificates-str

[3] Renewing your own CA certificates
https://access.redhat.com/documentation/en-us/red_hat_amq/7.7/html-single/using_amq_streams_on_openshift/index#renewing-your-own-ca-certificates-str

[4] using "strimzi.io/force-renew=true" annotation
https://strimzi.io/docs/operators/master/using.html#proc-renewing-ca-certs-manually-deployment-configuration-kafka

is documented by

ENTMQST-2597 [DOC OCP] Make it clear that CA annotations do not work with custom CAs

Closed

is related to

ENTMQST-3702 [QE] - System test for fixed renewal of user provided CAs

Closed

Details

Description

Attachments

Issue Links

Easy Agile Planning Poker

Activity

People

Dates