Details
-
Bug
-
Resolution: Not a Bug
-
Minor
-
None
-
Logging 5.6.0
-
None
-
False
-
None
-
False
-
NEW
-
NEW
Description
Description of problem:
When connecting to LokiStack services endpoint using openssl, the connection fails due to an invalid CN in the certificate chain.
sh-4.4# openssl s_client -tls1_2 -cipher "ECDHE-RSA-AES128-GCM-SHA256" -CAfile /run/secrets/kubernetes.io/serviceaccount/service-ca.crt -connect lokistack-instance-distributor-http:3100 CONNECTED(00000003) Can't use SSL_get_servername depth=1 CN = openshift-logging_lokistack-instance-signing-ca@1667972287 verify error:num=19:self signed certificate in certificate chain verify return:1 depth=1 CN = openshift-logging_lokistack-instance-signing-ca@1667972287 verify return:1 depth=0 O = system:logging, CN = system:lokistacks verify return:1 140095779673920:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42 --- Certificate chain 0 s:O = system:logging, CN = system:lokistacks i:CN = openshift-logging_lokistack-instance-signing-ca@1667972287 1 s:CN = openshift-logging_lokistack-instance-signing-ca@1667972287 i:CN = openshift-logging_lokistack-instance-signing-ca@1667972287 --- Server certificate -----BEGIN CERTIFICATE----- MIIEETCCAvmgAwIBAgIIK2BWq3qfdG8wDQYJKoZIhvcNAQELBQAwRTFDMEEGA1UE Aww6b3BlbnNoaWZ0LWxvZ2dpbmdfbG9raXN0YWNrLWluc3RhbmNlLXNpZ25pbmct Y2FAMTY2Nzk3MjI4NzAeFw0yMjExMDkwNTM4MDlaFw0yMzAyMDcwNTM4MTBaMDUx FzAVBgNVBAoTDnN5c3RlbTpsb2dnaW5nMRowGAYDVQQDExFzeXN0ZW06bG9raXN0 YWNrczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPpR7jSV4hmGG81q TCfnTXNhXtJhWhj4KL7JMxUjbkfvoJAofyALIlq3chfws8DeJZtWXSmK+jW1RsIX ttOMO/udwM4TQ1L/B08PekXrwCLQP9STSNrjNUlu99B0pUYm4xzUM/ZSL3uyorv5 aIH2kuzqbP7TBiO+T+IlhwlFnDKANmY4yrM7wF6KrrUBFJKWTTM3qS8CvJwX/qGC 5+GrzWjlbChOTNWmS9dBW6wC5hIkPWSCVWOvlEQrokijo2KoP5p7VjZdZH+ZW3p2 ev5Q2rR6PtXrglnu/S+wIanjYTUQfz7BxlS+c1Cbl0LOakuDuH7y5vLQRxf3rvis /lezSWUCAwEAAaOCARMwggEPMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggr BgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUvNdkZoUm fBm65NjyuvVGAavfucswHwYDVR0jBBgwFoAUppboxDwgzWq4nGRfjbvGCQIfyCIw gY8GA1UdEQSBhzCBhII5bG9raXN0YWNrLWluc3RhbmNlLWRpc3RyaWJ1dG9yLWh0 dHAub3BlbnNoaWZ0LWxvZ2dpbmcuc3Zjgkdsb2tpc3RhY2staW5zdGFuY2UtZGlz dHJpYnV0b3ItaHR0cC5vcGVuc2hpZnQtbG9nZ2luZy5zdmMuY2x1c3Rlci5sb2Nh bDANBgkqhkiG9w0BAQsFAAOCAQEAmfIMIYS+ZjZZkDpb1xt9/lXK/wnBtH1Pu/D/ M8GfkzhUJd8g3wTTXlmHPJHaTC8Tll2JIWq4F7jy3R4tzQhxtF1IFrtwM8Zf2e9Q fYZHjxdwcUtjulVQihdO5SXokdzKhcJl0TJ0wZxMfRGCyC9FNCiQP89dqV5eE06M aoN3Vgsf8aC3IELqAXnCQ0s/vkzALU/aHwl54GvnASvdFe+7BNb6mJp7AZQG7QVX Zzo6uVlbH63qizaV7iJrW3HM3ldKVD0U5bxLtyW2X4/XES9GI2A7Z5IwYGYQzxXx Iw1n48BQj7VqgZOFLQVIrEOBpefkrortRI5BKVWlZ3VBFAzqqw== -----END CERTIFICATE----- subject=O = system:logging, CN = system:lokistacks issuer=CN = openshift-logging_lokistack-instance-signing-ca@1667972287 --- Acceptable client certificate CA names CN = openshift-logging_lokistack-instance-signing-ca@1667972287 Client Certificate Types: RSA sign, ECDSA sign Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1 Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1 Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 2444 bytes and written 237 bytes Verification error: self signed certificate in certificate chain --- New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES128-GCM-SHA256 Session-ID: Session-ID-ctx: Master-Key: C7B8294185D16FE5DB42B00B8F5C347478ECCA27B51793908D1204632908333A095F58995DE9D17EB0E93D6335C44478 PSK identity: None PSK identity hint: None SRP username: None Start Time: 1667974452 Timeout : 7200 (sec) Verify return code: 19 (self signed certificate in certificate chain) Extended master secret: no ---
Version-Release number of selected component (if applicable):
Server Version: 4.11.0-0.nightly-2022-11-08-222031
cluster-logging.v5.6.0
loki-operator.v5.6.0
How reproducible:
Always.
Checked on a OCP 4.12 and 4.11 cluster.
Steps to Reproduce:
*Deploy Cluster Logging and LokiStack 5.6.0 operators.
*Create a LokiStack instance.
apiVersion: loki.grafana.com/v1 kind: LokiStack metadata: name: lokistack-instance namespace: openshift-logging spec: managementState: Managed replicationFactor: 1 size: 1x.extra-small storage: secret: name: s3-secret type: s3 storageClassName: gp2 tenants: mode: openshift-logging
*Create a ClusterLogging instance to forward logs to LokiStack instance.
apiVersion: "logging.openshift.io/v1" kind: "ClusterLogging" metadata: name: "instance" namespace: openshift-logging spec: managementState: "Managed" logStore: type: "lokistack" lokistack: name: lokistack-instance collection: type: "vector"
*Check connection to different LokiStack components. For example the connection works fine for lokistack gateway which has the right CN but fails for lokistack distributor.
For lokistack gateway:
oc rsh cluster-logging-operator-67bd95ddd9-tztt2 sh-4.4# openssl s_client -tls1_2 -cipher "ECDHE-RSA-AES128-GCM-SHA256" -CAfile /run/secrets/kubernetes.io/serviceaccount/service-ca.crt -connect lokistack-instance-gateway-http:8081 CONNECTED(00000003) Can't use SSL_get_servername depth=1 CN = openshift-service-serving-signer@1667970706 verify return:1 depth=0 CN = lokistack-instance-gateway-http.openshift-logging.svc verify return:1 --- Certificate chain 0 s:CN = lokistack-instance-gateway-http.openshift-logging.svc i:CN = openshift-service-serving-signer@1667970706 1 s:CN = openshift-service-serving-signer@1667970706 i:CN = openshift-service-serving-signer@1667970706 --- Server certificate -----BEGIN CERTIFICATE----- MIIEMDCCAxigAwIBAgIITNKtfRdekf0wDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTY2Nzk3MDcwNjAe Fw0yMjExMDkwNTM4MTBaFw0yNDExMDgwNTM4MTFaMEAxPjA8BgNVBAMTNWxva2lz dGFjay1pbnN0YW5jZS1nYXRld2F5LWh0dHAub3BlbnNoaWZ0LWxvZ2dpbmcuc3Zj MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAs4f2PFVORDxb0joExStA WNE/hN/bh9TSOq7wqfcpYZ9OKpwp7qk63GnaX1srOMMOLiv8aH8dgkGl7U6BU0ss nwofZrJBH5+4dv5dZdAVaxgwShHzYL9Plcquz/hwpy8JOrU4KcnTtplv0jldyXYW yjfJtWRTA4OXkQ0KqHzhcmkUAjLlnHvdLOKoPZZuYUeK879J7cTLqQ97TxkKZgG2 wIruYCe82D/MCoEjMjR9oQdzJ6CYHBmftKN6tv9MTxJwAbcwHdOK4OjtMgXupNla pGX78KvmrLSbbrwoTb1QiEYN/rC2cOHLiavHhzKQwpgyG7/AMjYjPeIHNg7My0Te gwIDAQABo4IBNjCCATIwDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUF BwMBMAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFJaYGZ1HmBgbIZ0cqHkpmEvuA1br MB8GA1UdIwQYMBaAFN1+wWsB7RjMX3IyrNLyiyUuW9uPMIGFBgNVHREEfjB8gjVs b2tpc3RhY2staW5zdGFuY2UtZ2F0ZXdheS1odHRwLm9wZW5zaGlmdC1sb2dnaW5n LnN2Y4JDbG9raXN0YWNrLWluc3RhbmNlLWdhdGV3YXktaHR0cC5vcGVuc2hpZnQt bG9nZ2luZy5zdmMuY2x1c3Rlci5sb2NhbDA1BgsrBgEEAZIIEWQCAQQmEyRiODM5 OTVlYS04YjdjLTQ4NGMtYTdlNy0yNDMzMjQ0MTQ5ZGMwDQYJKoZIhvcNAQELBQAD ggEBAGhtVo/HMjDP4DYZDoJ6m1l/NhY0pwRNaBG7o/y0eATHFz5R1E99YoXYuJx2 M7AKrYs3UgSKst2wbMtUQ+2uArjrdvD2WuO5dSXOrwkfIw6Wd47foVhoZV0/i0lM V0XVeh7xZHMMCKoF4ysOgG1e/qY88oYhgV0CZlfZxWIxV+MFoVtlZmvGaiD9B+WR GKe4Tsm01M1Z8elltaBdzlCU68/WHxKRp9EryCydfUaZocEorp2SuOCJoKWz1hPT +h3nAdiaqTCRVCLpMTyVaMTxtH5os+E/tBbufOFR+cnr/7NWtWt8Q9hlYRxLVDsa rTOm0p0S1uFPegY/0zOqKBJrnKQ= -----END CERTIFICATE----- subject=CN = lokistack-instance-gateway-http.openshift-logging.svc issuer=CN = openshift-service-serving-signer@1667970706 --- No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 2520 bytes and written 225 bytes Verification: OK --- New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES128-GCM-SHA256 Session-ID: EC4C7E273B6173306C07C00E9AC6610074849C9955996EF0B4F42FFFA2B8D7B8 Session-ID-ctx: Master-Key: A387DC5BBED5033BD1E53760010AF042F35DC193479A0D9A0BD48A71E787CAF906D5CFDA4C256FF8E0C4C01A06900204 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket: 0000 - d9 b9 15 b1 a7 2d 69 5a-94 51 a7 ec 00 28 fc 8b .....-iZ.Q...(.. 0010 - 57 46 1e e7 e8 12 78 d7-81 af 55 6d 4d 59 10 bf WF....x...UmMY.. 0020 - 9a 1c 7c 0c 05 fa 64 eb-65 fa 6d 70 fd d5 bf 9f ..|...d.e.mp.... 0030 - 1e 77 67 66 85 91 83 36-8d 92 dc 47 c1 4f 04 07 .wgf...6...G.O.. 0040 - b0 5f d4 2f 65 fc 0b 13-1c e1 b7 0d 5d 32 51 88 ._./e.......]2Q. 0050 - 44 59 e9 59 3e 5d e4 f4-02 06 4a 9f c3 b7 a9 56 DY.Y>]....J....V 0060 - 65 ba fe 79 5a c1 52 0e-56 d9 ae 24 00 9b 2b 8c e..yZ.R.V..$..+. 0070 - 47 55 83 a3 48 6c 67 1c-90 aa 00 9c 44 35 77 52 GU..Hlg.....D5wR 0080 - f4 . Start Time: 1667974976 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: no ---
For lokistack distributor:
sh-4.4# openssl s_client -tls1_2 -cipher "ECDHE-RSA-AES128-GCM-SHA256" -CAfile /run/secrets/kubernetes.io/serviceaccount/service-ca.crt -connect lokistack-instance-distributor-http:3100 CONNECTED(00000003) Can't use SSL_get_servername depth=1 CN = openshift-logging_lokistack-instance-signing-ca@1667972287 verify error:num=19:self signed certificate in certificate chain verify return:1 depth=1 CN = openshift-logging_lokistack-instance-signing-ca@1667972287 verify return:1 depth=0 O = system:logging, CN = system:lokistacks verify return:1 140095779673920:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate:ssl/record/rec_layer_s3.c:1544:SSL alert number 42 --- Certificate chain 0 s:O = system:logging, CN = system:lokistacks i:CN = openshift-logging_lokistack-instance-signing-ca@1667972287 1 s:CN = openshift-logging_lokistack-instance-signing-ca@1667972287 i:CN = openshift-logging_lokistack-instance-signing-ca@1667972287 --- Server certificate -----BEGIN CERTIFICATE----- MIIEETCCAvmgAwIBAgIIK2BWq3qfdG8wDQYJKoZIhvcNAQELBQAwRTFDMEEGA1UE Aww6b3BlbnNoaWZ0LWxvZ2dpbmdfbG9raXN0YWNrLWluc3RhbmNlLXNpZ25pbmct Y2FAMTY2Nzk3MjI4NzAeFw0yMjExMDkwNTM4MDlaFw0yMzAyMDcwNTM4MTBaMDUx FzAVBgNVBAoTDnN5c3RlbTpsb2dnaW5nMRowGAYDVQQDExFzeXN0ZW06bG9raXN0 YWNrczCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAPpR7jSV4hmGG81q TCfnTXNhXtJhWhj4KL7JMxUjbkfvoJAofyALIlq3chfws8DeJZtWXSmK+jW1RsIX ttOMO/udwM4TQ1L/B08PekXrwCLQP9STSNrjNUlu99B0pUYm4xzUM/ZSL3uyorv5 aIH2kuzqbP7TBiO+T+IlhwlFnDKANmY4yrM7wF6KrrUBFJKWTTM3qS8CvJwX/qGC 5+GrzWjlbChOTNWmS9dBW6wC5hIkPWSCVWOvlEQrokijo2KoP5p7VjZdZH+ZW3p2 ev5Q2rR6PtXrglnu/S+wIanjYTUQfz7BxlS+c1Cbl0LOakuDuH7y5vLQRxf3rvis /lezSWUCAwEAAaOCARMwggEPMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggr BgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUvNdkZoUm fBm65NjyuvVGAavfucswHwYDVR0jBBgwFoAUppboxDwgzWq4nGRfjbvGCQIfyCIw gY8GA1UdEQSBhzCBhII5bG9raXN0YWNrLWluc3RhbmNlLWRpc3RyaWJ1dG9yLWh0 dHAub3BlbnNoaWZ0LWxvZ2dpbmcuc3Zjgkdsb2tpc3RhY2staW5zdGFuY2UtZGlz dHJpYnV0b3ItaHR0cC5vcGVuc2hpZnQtbG9nZ2luZy5zdmMuY2x1c3Rlci5sb2Nh bDANBgkqhkiG9w0BAQsFAAOCAQEAmfIMIYS+ZjZZkDpb1xt9/lXK/wnBtH1Pu/D/ M8GfkzhUJd8g3wTTXlmHPJHaTC8Tll2JIWq4F7jy3R4tzQhxtF1IFrtwM8Zf2e9Q fYZHjxdwcUtjulVQihdO5SXokdzKhcJl0TJ0wZxMfRGCyC9FNCiQP89dqV5eE06M aoN3Vgsf8aC3IELqAXnCQ0s/vkzALU/aHwl54GvnASvdFe+7BNb6mJp7AZQG7QVX Zzo6uVlbH63qizaV7iJrW3HM3ldKVD0U5bxLtyW2X4/XES9GI2A7Z5IwYGYQzxXx Iw1n48BQj7VqgZOFLQVIrEOBpefkrortRI5BKVWlZ3VBFAzqqw== -----END CERTIFICATE----- subject=O = system:logging, CN = system:lokistacks issuer=CN = openshift-logging_lokistack-instance-signing-ca@1667972287 --- Acceptable client certificate CA names CN = openshift-logging_lokistack-instance-signing-ca@1667972287 Client Certificate Types: RSA sign, ECDSA sign Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1 Shared Requested Signature Algorithms: RSA-PSS+SHA256:ECDSA+SHA256:Ed25519:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA1:ECDSA+SHA1 Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits --- SSL handshake has read 2444 bytes and written 237 bytes Verification error: self signed certificate in certificate chain --- New, TLSv1.2, Cipher is ECDHE-RSA-AES128-GCM-SHA256 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES128-GCM-SHA256 Session-ID: Session-ID-ctx: Master-Key: C7B8294185D16FE5DB42B00B8F5C347478ECCA27B51793908D1204632908333A095F58995DE9D17EB0E93D6335C44478 PSK identity: None PSK identity hint: None SRP username: None Start Time: 1667974452 Timeout : 7200 (sec) Verify return code: 19 (self signed certificate in certificate chain) Extended master secret: no ---
Check that for gateway the CN shows as below in certificate chain.
Certificate chain 0 s:CN = lokistack-instance-gateway-http.openshift-logging.svc i:CN = openshift-service-serving-signer@1667970706 1 s:CN = openshift-service-serving-signer@1667970706 i:CN = openshift-service-serving-signer@1667970706
And for distributor the CN shows as below.
Certificate chain 0 s:O = system:logging, CN = system:lokistacks i:CN = openshift-logging_lokistack-instance-signing-ca@1667972287 1 s:CN = openshift-logging_lokistack-instance-signing-ca@1667972287 i:CN = openshift-logging_lokistack-instance-signing-ca@1667972287
Additional details:
The issue is affecting query frontend, querier, ingester, index gateway, compactor components as well.
