-
Bug
-
Resolution: Done
-
Normal
-
Logging 5.7.0
-
False
-
None
-
False
-
NEW
-
OBSDA-160 - Comply with OCP cluster-wide cryptographic policies
-
VERIFIED
-
Prior to this update, vector log collector did not honor the tlsSecurityProfile settings for outgoing TLS connections. After this update, vector does honor the tlsSecurityProfile settings for outgoing TLS connections.
-
Bug Fix
-
-
-
Log Collection - Sprint 236, Log Collection - Sprint 237
Description of problem:
When a custom tlsSecurityProfile is set with a ciphersuite not supported by remote server, the connection to the remote output should fail. For example connecting to an externally hosted loki, the remote Loki output only supports TLS v1.2 and above.
openssl s_client -tls1_1 --connect logs-prod3.grafana.net:443
CONNECTED(00000003)
808BA8CDF77F0000:error:0A00042E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1605:SSL alert number 70
—
no peer certificate available
—
No client certificate CA names sent
—
SSL handshake has read 7 bytes and written 135 bytes
Verification: OK
—
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.1
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1682431553
Timeout : 7200 (sec)
Verify return code: 0 (ok)
Extended master secret: no
—
$ openssl s_client -tls1_2 --connect logs-prod3.grafana.net:443 CONNECTED(00000003) depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA verify return:1 depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 verify return:1 depth=0 C = US, ST = New York, L = New York, O = Raintank Inc., CN = grafana.com verify return:1 — Certificate chain 0 s:C = US, ST = New York, L = New York, O = Raintank Inc., CN = grafana.com i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Jan 31 00:00:00 2023 GMT; NotAfter: Mar 2 23:59:59 2024 GMT 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256 v:NotBefore: Apr 14 00:00:00 2021 GMT; NotAfter: Apr 13 23:59:59 2031 GMT — Server certificate ----BEGIN CERTIFICATE---- MIIHbDCCBlSgAwIBAgIQCLtVHY4/+djh9O3AfHzjjDANBgkqhkiG9w0BAQsFADBP MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSkwJwYDVQQDEyBE aWdpQ2VydCBUTFMgUlNBIFNIQTI1NiAyMDIwIENBMTAeFw0yMzAxMzEwMDAwMDBa Fw0yNDAzMDIyMzU5NTlaMGExCzAJBgNVBAYTAlVTMREwDwYDVQQIEwhOZXcgWW9y azERMA8GA1UEBxMITmV3IFlvcmsxFjAUBgNVBAoTDVJhaW50YW5rIEluYy4xFDAS BgNVBAMTC2dyYWZhbmEuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC AQEAoj+bmhK4AUvv9/dK5aDxBn5l01BrkATCPIR43uj5zxGCyGBoX7n5pFWsEAMb rkdSfqRluoFTi8f/u9ECcMwF1M4ISyoQ6iPh7E6pkQULoAgITLptiANVzyw30pxl RRKmNqxC4fzEXwxc3qHgaWB+xMwC/i3vExnqHoAUYnvejXT8W/Szmbm8eX5WLTbe r21hDfGRMztlUur3UfQ9e2UVe8dsLKMLXjmu6uHsBJ4I1tOT81RohL0+/OiPi9ct JL/TpbLRdLFCpJd7aLT186R4e50qTwIQVBXLG91wdvoxd9oaxSp9xTHfkYW0OqMM ci9ZqUbJ9FahtCjDhDBaQeym9wIDAQABo4IEMDCCBCwwHwYDVR0jBBgwFoAUt2ui 6qiqhIx56rTaD5iyxZV2ufQwHQYDVR0OBBYEFEFCsIHtSmR60zTDE6D0yaNtX4RG MIHbBgNVHREEgdMwgdCCC2dyYWZhbmEuY29tgg0qLmdyYWZhbmEuY29tgg0qLmdy YWZhbmEubmV0gg0qLmdyYWZhbmEub3JnghwqLmhvc3RlZC1tZXRyaWNzLmdyYWZh bmEubmV0gg0qLnJhaW50YW5rLmlvggtncmFmYW5hLm5ldIILZ3JhZmFuYS5vcmeC C3JhaW50YW5rLmlvghEqLmdyYWZhbmFsYWJzLmNvbYIPZ3JhZmFuYWxhYnMuY29t gg4qLnJhaW50YW5rLmNvbYIMcmFpbnRhbmsuY29tMA4GA1UdDwEB/wQEAwIFoDAd BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwgY8GA1UdHwSBhzCBhDBAoD6g PIY6aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0VExTUlNBU0hBMjU2 MjAyMENBMS00LmNybDBAoD6gPIY6aHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0Rp Z2lDZXJ0VExTUlNBU0hBMjU2MjAyMENBMS00LmNybDA+BgNVHSAENzA1MDMGBmeB DAECAjApMCcGCCsGAQUFBwIBFhtodHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMw fwYIKwYBBQUHAQEEczBxMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2Vy dC5jb20wSQYIKwYBBQUHMAKGPWh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9E aWdpQ2VydFRMU1JTQVNIQTI1NjIwMjBDQTEtMS5jcnQwCQYDVR0TBAIwADCCAX0G CisGAQQB1nkCBAIEggFtBIIBaQFnAHUA7s3QZNXbGs7FXLedtM0TojKHRny87N7D UUhZRnEftZsAAAGGCBdyVQAABAMARjBEAh8ueDr4NtVhJSQ6wkBCZQNiWPviyAx+ L95Bclm2OPm8AiEA8OnhpIHH1vMZgAwlZR2szZMMJcfhjL0jNodTlw1YH84AdgBz 2Z6JG0yWeKAgfUed5rLGHNBRXnEZKoxrgBB6wXdytQAAAYYIF3JFAAAEAwBHMEUC IQCmj8ZWqspndtvQBFcA/zSkIsDt2k5u+cMsEqEgSQpNCAIgB2rfvSP1fDK0eGmN nJ+46bp3IBPNDa9jcxR0j6T4h14AdgBIsONr2qZHNA/lagL6nTDrHFIBy1bdLIHZ u7+rOdiEcwAAAYYIF3IVAAAEAwBHMEUCIFoSQTsvCBbeTuzhqApnd8rFbcR+aQpI jOeuO54sGiMqAiEAlLIWxwG0QWLj+Qw7vBQ7EcTw8GeGjRfEw5uxE+gyT6gwDQYJ KoZIhvcNAQELBQADggEBAEIm5uKQm/7/qJpIPGg1fKE5K46ZLSSL65MLbqTKgggO /bPJZ4W4heCx1fS4Fnyq83+bg1W/fEzuz6BY9RfYmIroz6MIdB0zDUzu/Ntbf0CR Z4D83TsS/UM+rTtznugLArSXR4ERomIm195YKJKB1vbemiSxSi55CsDk+9Fc11Tm dw5TRa9lwTiw8cO6+L+zYg9vgm2pl3oT/DYpPcf8Ncy+fJOdghwa37HGvrVD5VoQ vD+OABv6Gjp3EbDFOvN3LZhY28XW/SEeleFTbs2DY1Wumk3qbCUms3Qq2251i3QT FcAHfINhretx7ZFqxXH4VeoRTNtHEa+IcZZrxI9/HHA= ----END CERTIFICATE---- subject=C = US, ST = New York, L = New York, O = Raintank Inc., CN = grafana.com issuer=C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1 — No client certificate CA names sent Peer signing digest: SHA256 Peer signature type: RSA-PSS Server Temp Key: X25519, 253 bits — SSL handshake has read 3809 bytes and written 304 bytes Verification: OK — New, TLSv1.2, Cipher is ECDHE-RSA-CHACHA20-POLY1305 Server public key is 2048 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-CHACHA20-POLY1305 Session-ID: E81F8A38184A9462DBE281A9C7534AEADDDFF4A9F071E7E396D354133C923FF0 Session-ID-ctx: Master-Key: 2FA60364F87854A8E8BC9D6B0EC6759A37E8136E3C6B7C27F613913E31ED38C19795464835921DF86B532788AAB0AD56 PSK identity: None PSK identity hint: None SRP username: None TLS session ticket lifetime hint: 100800 (seconds) TLS session ticket: 0000 - 02 97 1b 8d 1a e5 b4 1b-32 17 b1 70 ca 40 22 23 ........2..p.@"# 0010 - d3 a7 04 20 7e 83 61 08-31 92 2b 81 14 1c 3f a2 ... ~.a.1.+...?. 0020 - 97 70 c4 09 49 2f 89 89-a2 55 0b 8f 56 77 a0 18 .p..I/...U..Vw.. 0030 - 0c 2d 47 87 ab 00 8b 1d-44 58 07 0d 36 ce 89 40 .-G.....DX..6..@ 0040 - f0 b5 7b e7 bd 37 5e 67-d1 3f 16 17 86 56 02 41 ..{..7^g.?...V.A 0050 - f8 17 82 07 a8 9f 7a fb-ab 07 ef 46 fe 5c dc dc ......z....F.\.. 0060 - 54 3b 79 69 9b 9b dc 50-a9 85 3b 9b fe a6 f9 55 T;yi...P..;....U 0070 - 31 8f 6d d6 be 6b 88 c7-22 53 5e f9 c4 57 31 f8 1.m..k.."S^..W1. 0080 - f6 bd e5 67 e4 c2 92 eb-da 50 85 d7 99 17 42 fe ...g.....P....B. 0090 - 14 a5 0e 59 d3 e3 f8 a5-16 80 a2 5b 72 86 05 a8 ...Y.......[r... 00a0 - 05 72 12 9f 33 79 8a a0-b5 db c7 c7 4a 0c cc 71 .r..3y......J..q 00b0 - 77 b2 b3 e0 20 0f eb 54-bb 8a 34 cd fb 1a a0 95 w... ..T..4..... 00c0 - cd 8b 6a e4 2f 24 d8 a3-a0 fe 55 6b 8b ba b1 b5 ..j./$....Uk.... 00d0 - 3c 7f a0 e5 cd a7 74 bb-8b 2c 42 4f 4d 04 39 67 <.....t..,BOM.9g 00e0 - b8 95 b5 ed 51 ....Q Start Time: 1682431845 Timeout : 7200 (sec) Verify return code: 0 (ok) Extended master secret: yes —
However if we use a ciphersuite from TLSv1_1 which is unsupported by remote output, vector is still able to make connection to the output and forward logs to it.
tlsSecurityProfile: custom: ciphers: - ECDHE-ECDSA-AES128-SHA minTLSVersion: VersionTLS10 type: Custom
This is not just for Loki output but observed for other outputs as well.
Version-Release number of selected component (if applicable):
cluster-logging.v5.7.0
CPaaS index image used: quay.io/openshift-qe-optional-operators/aosqe-index:openshift-logging-5.7.0-20230424.46
Server Version: 4.13.0-0.nightly-2023-04-21-084440
How reproducible:
Always
Steps to Reproduce:
*Create secret in openshift-logging namespace to create HTTPS connection to external Loki
oc create secret generic loki-client -n openshift-logging --from-literal=username=<Loki Grafana username> --from-literal=password=<loki Grafana API token>
*Create a CLF with an unsupported ciphersuite set in custom profile.
apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
annotations:
logging.openshift.io/preview-tls-security-profile: enabled
name: instance
namespace: openshift-logging
spec:
outputs:
- name: loki-server
secret:
name: loki-client
type: loki
url: https://logs-prod3.grafana.net
pipelines:
- inputRefs:
- application
name: to-loki
outputRefs:
- loki-server
tlsSecurityProfile:
custom:
ciphers:
- ECDHE-ECDSA-AES128-SHA
minTLSVersion: VersionTLS10
type: Custom
*Create a ClusterLogging instance.
apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: annotations: logging.openshift.io/preview-vector-collector: enabled name: instance namespace: openshift-logging spec: collection: logs: type: vector type: vector managementState: Managed
*Check the collector pods logs, there are no errors observed in the logs and the logs are forwarded to the external output.
$ oc logs --selector=component=collector | grpe ERROR
bash: grpe: command not found...
Similar command is: 'grep'
Defaulted container "collector" out of: collector, logfilesmetricexporter
Defaulted container "collector" out of: collector, logfilesmetricexporter
Defaulted container "collector" out of: collector, logfilesmetricexporter
Defaulted container "collector" out of: collector, logfilesmetricexporter
Defaulted container "collector" out of: collector, logfilesmetricexporter
Defaulted container "collector" out of: collector, logfilesmetricexporter
Defaulted container "collector" out of: collector, logfilesmetricexporter
| 2023-04-25 19:58:48.335 | {"@timestamp":"2023-04-25T14:28:48.023042867Z","file":"/var/log/pods/test_loggen-qa-json-t4jkm_b81cc1da-ad03-4c29-870b-3ab75ac6168e/loggen-qa-json/0.log","hostname":"ip-10-0-74-158.us-east-2.compute.internal","kubernetes":{"annotations":{"k8s.v1.cni.cncf.io/network-status":"[{\n \"name\": \"openshift-sdn\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.131.0.20\"\n ],\n \"default\": true,\n \"dns\": {}\n}]","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_id":"cri-o://315a34391852727d544fc5bc5a0e0894f79dd1d94eb4be9712e0ca527b816c51","container_image":"quay.io/openshifttest/ocp-logtest@sha256:16232868ba1143721b786dbabb3f7384645acb663fadb4af48e9ea1228a67635","container_name":"loggen-qa-json","labels":{"run":"centos-logtest","test":"loggen-qa-json"},"namespace_labels":{"kubernetes_io_metadata_name":"test","pod-security_kubernetes_io_audit":"restricted","pod-security_kubernetes_io_audit-version":"v1.24","pod-security_kubernetes_io_warn":"restricted","pod-security_kubernetes_io_warn-version":"v1.24"},"namespace_name":"test","pod_id":"b81cc1da-ad03-4c29-870b-3ab75ac6168e","pod_ip":"10.131.0.20","pod_name":"loggen-qa-json-t4jkm","pod_owner":"ReplicationController/loggen-qa-json"},"level":"default","log_type":"application","message":"{\"message\": \"MERGE_JSON_LOG=true\", \"level\": \"debug\",\"Layer1\": \"layer1 0\", \"layer2\":
{\"name\":\"Layer2 1\", \"tips\":\"Decide by PRESERVE_JSON_LOG\"}
, \"StringNumber\":\"10\", \"Number\": 10,\"foo.bar\":\"Dot Item\",\"{foobar}\":\"Brace Item\",\"[foobar]\":\"Bracket Item\", \"foo:bar\":\"Colon Item\",\"foo bar\":\"Space Item\" }","openshift":{"cluster_id":"e574e644-a047-45e4-b007-9bbf9c29cdab","sequence":1439}} |
Expected results:
Vector uses the ciphersuite defined in the custom profile to connect to the remote output.
Additional info:
After setting the ClusterLogging to Unmanaged and setting VECTOR_LOG env var to TRACE in collector daemonset, we do not see any TLS protocol or ciphersuite being used in the collector logs.
Vector loki sink TLS config.
[sinks.loki_server.tls] enabled = true min_tls_version = "VersionTLS10" ciphersuites = "ECDHE-ECDSA-AES128-SHA"
- is cloned by
-
-
- Closed
-
- links to
- mentioned on
(1 mentioned on)
