Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4047

[Vector] Collector not complying with the custom tlsSecurityProfile configuration

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • OBSDA-160 - Comply with OCP cluster-wide cryptographic policies
    • VERIFIED
    • Prior to this update, vector log collector did not honor the tlsSecurityProfile settings for outgoing TLS connections. After this update, vector does honor the tlsSecurityProfile settings for outgoing TLS connections.
    • Log Collection - Sprint 236, Log Collection - Sprint 237

      Description of problem:

      When a custom tlsSecurityProfile is set with a ciphersuite not supported by remote server, the connection to the remote output should fail. For example connecting to an externally hosted loki, the remote Loki output only supports TLS v1.2 and above.

      openssl s_client -tls1_1 --connect logs-prod3.grafana.net:443
CONNECTED(00000003)
808BA8CDF77F0000:error:0A00042E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1605:SSL alert number 70
—
no peer certificate available
—
No client certificate CA names sent
—
SSL handshake has read 7 bytes and written 135 bytes
Verification: OK
—
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.1
    Cipher    : 0000
    Session-ID: 
    Session-ID-ctx: 
    Master-Key: 
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1682431553
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
—
      $ openssl s_client -tls1_2 --connect logs-prod3.grafana.net:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
verify return:1
depth=0 C = US, ST = New York, L = New York, O = Raintank Inc., CN = grafana.com
verify return:1
—
Certificate chain
 0 s:C = US, ST = New York, L = New York, O = Raintank Inc., CN = grafana.com
   i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 31 00:00:00 2023 GMT; NotAfter: Mar  2 23:59:59 2024 GMT
 1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
   i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 14 00:00:00 2021 GMT; NotAfter: Apr 13 23:59:59 2031 GMT
—
Server certificate
----BEGIN CERTIFICATE----
MIIHbDCCBlSgAwIBAgIQCLtVHY4/+djh9O3AfHzjjDANBgkqhkiG9w0BAQsFADBP
MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSkwJwYDVQQDEyBE
aWdpQ2VydCBUTFMgUlNBIFNIQTI1NiAyMDIwIENBMTAeFw0yMzAxMzEwMDAwMDBa
Fw0yNDAzMDIyMzU5NTlaMGExCzAJBgNVBAYTAlVTMREwDwYDVQQIEwhOZXcgWW9y
azERMA8GA1UEBxMITmV3IFlvcmsxFjAUBgNVBAoTDVJhaW50YW5rIEluYy4xFDAS
BgNVBAMTC2dyYWZhbmEuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAoj+bmhK4AUvv9/dK5aDxBn5l01BrkATCPIR43uj5zxGCyGBoX7n5pFWsEAMb
rkdSfqRluoFTi8f/u9ECcMwF1M4ISyoQ6iPh7E6pkQULoAgITLptiANVzyw30pxl
RRKmNqxC4fzEXwxc3qHgaWB+xMwC/i3vExnqHoAUYnvejXT8W/Szmbm8eX5WLTbe
r21hDfGRMztlUur3UfQ9e2UVe8dsLKMLXjmu6uHsBJ4I1tOT81RohL0+/OiPi9ct
JL/TpbLRdLFCpJd7aLT186R4e50qTwIQVBXLG91wdvoxd9oaxSp9xTHfkYW0OqMM
ci9ZqUbJ9FahtCjDhDBaQeym9wIDAQABo4IEMDCCBCwwHwYDVR0jBBgwFoAUt2ui
6qiqhIx56rTaD5iyxZV2ufQwHQYDVR0OBBYEFEFCsIHtSmR60zTDE6D0yaNtX4RG
MIHbBgNVHREEgdMwgdCCC2dyYWZhbmEuY29tgg0qLmdyYWZhbmEuY29tgg0qLmdy
YWZhbmEubmV0gg0qLmdyYWZhbmEub3JnghwqLmhvc3RlZC1tZXRyaWNzLmdyYWZh
bmEubmV0gg0qLnJhaW50YW5rLmlvggtncmFmYW5hLm5ldIILZ3JhZmFuYS5vcmeC
C3JhaW50YW5rLmlvghEqLmdyYWZhbmFsYWJzLmNvbYIPZ3JhZmFuYWxhYnMuY29t
gg4qLnJhaW50YW5rLmNvbYIMcmFpbnRhbmsuY29tMA4GA1UdDwEB/wQEAwIFoDAd
BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwgY8GA1UdHwSBhzCBhDBAoD6g
PIY6aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0VExTUlNBU0hBMjU2
MjAyMENBMS00LmNybDBAoD6gPIY6aHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0Rp
Z2lDZXJ0VExTUlNBU0hBMjU2MjAyMENBMS00LmNybDA+BgNVHSAENzA1MDMGBmeB
DAECAjApMCcGCCsGAQUFBwIBFhtodHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMw
fwYIKwYBBQUHAQEEczBxMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2Vy
dC5jb20wSQYIKwYBBQUHMAKGPWh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9E
aWdpQ2VydFRMU1JTQVNIQTI1NjIwMjBDQTEtMS5jcnQwCQYDVR0TBAIwADCCAX0G
CisGAQQB1nkCBAIEggFtBIIBaQFnAHUA7s3QZNXbGs7FXLedtM0TojKHRny87N7D
UUhZRnEftZsAAAGGCBdyVQAABAMARjBEAh8ueDr4NtVhJSQ6wkBCZQNiWPviyAx+
L95Bclm2OPm8AiEA8OnhpIHH1vMZgAwlZR2szZMMJcfhjL0jNodTlw1YH84AdgBz
2Z6JG0yWeKAgfUed5rLGHNBRXnEZKoxrgBB6wXdytQAAAYYIF3JFAAAEAwBHMEUC
IQCmj8ZWqspndtvQBFcA/zSkIsDt2k5u+cMsEqEgSQpNCAIgB2rfvSP1fDK0eGmN
nJ+46bp3IBPNDa9jcxR0j6T4h14AdgBIsONr2qZHNA/lagL6nTDrHFIBy1bdLIHZ
u7+rOdiEcwAAAYYIF3IVAAAEAwBHMEUCIFoSQTsvCBbeTuzhqApnd8rFbcR+aQpI
jOeuO54sGiMqAiEAlLIWxwG0QWLj+Qw7vBQ7EcTw8GeGjRfEw5uxE+gyT6gwDQYJ
KoZIhvcNAQELBQADggEBAEIm5uKQm/7/qJpIPGg1fKE5K46ZLSSL65MLbqTKgggO
/bPJZ4W4heCx1fS4Fnyq83+bg1W/fEzuz6BY9RfYmIroz6MIdB0zDUzu/Ntbf0CR
Z4D83TsS/UM+rTtznugLArSXR4ERomIm195YKJKB1vbemiSxSi55CsDk+9Fc11Tm
dw5TRa9lwTiw8cO6+L+zYg9vgm2pl3oT/DYpPcf8Ncy+fJOdghwa37HGvrVD5VoQ
vD+OABv6Gjp3EbDFOvN3LZhY28XW/SEeleFTbs2DY1Wumk3qbCUms3Qq2251i3QT
FcAHfINhretx7ZFqxXH4VeoRTNtHEa+IcZZrxI9/HHA=
----END CERTIFICATE----
subject=C = US, ST = New York, L = New York, O = Raintank Inc., CN = grafana.com
issuer=C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
—
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
—
SSL handshake has read 3809 bytes and written 304 bytes
Verification: OK
—
New, TLSv1.2, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-CHACHA20-POLY1305
    Session-ID: E81F8A38184A9462DBE281A9C7534AEADDDFF4A9F071E7E396D354133C923FF0
    Session-ID-ctx: 
    Master-Key: 2FA60364F87854A8E8BC9D6B0EC6759A37E8136E3C6B7C27F613913E31ED38C19795464835921DF86B532788AAB0AD56
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 100800 (seconds)
    TLS session ticket:
    0000 - 02 97 1b 8d 1a e5 b4 1b-32 17 b1 70 ca 40 22 23   ........2..p.@"#
    0010 - d3 a7 04 20 7e 83 61 08-31 92 2b 81 14 1c 3f a2   ... ~.a.1.+...?.
    0020 - 97 70 c4 09 49 2f 89 89-a2 55 0b 8f 56 77 a0 18   .p..I/...U..Vw..
    0030 - 0c 2d 47 87 ab 00 8b 1d-44 58 07 0d 36 ce 89 40   .-G.....DX..6..@
    0040 - f0 b5 7b e7 bd 37 5e 67-d1 3f 16 17 86 56 02 41   ..{..7^g.?...V.A
    0050 - f8 17 82 07 a8 9f 7a fb-ab 07 ef 46 fe 5c dc dc   ......z....F.\..
    0060 - 54 3b 79 69 9b 9b dc 50-a9 85 3b 9b fe a6 f9 55   T;yi...P..;....U
    0070 - 31 8f 6d d6 be 6b 88 c7-22 53 5e f9 c4 57 31 f8   1.m..k.."S^..W1.
    0080 - f6 bd e5 67 e4 c2 92 eb-da 50 85 d7 99 17 42 fe   ...g.....P....B.
    0090 - 14 a5 0e 59 d3 e3 f8 a5-16 80 a2 5b 72 86 05 a8   ...Y.......[r...
    00a0 - 05 72 12 9f 33 79 8a a0-b5 db c7 c7 4a 0c cc 71   .r..3y......J..q
    00b0 - 77 b2 b3 e0 20 0f eb 54-bb 8a 34 cd fb 1a a0 95   w... ..T..4.....
    00c0 - cd 8b 6a e4 2f 24 d8 a3-a0 fe 55 6b 8b ba b1 b5   ..j./$....Uk....
    00d0 - 3c 7f a0 e5 cd a7 74 bb-8b 2c 42 4f 4d 04 39 67   <.....t..,BOM.9g
    00e0 - b8 95 b5 ed 51                                    ....Q
    Start Time: 1682431845
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: yes
—

      However if we use a ciphersuite from TLSv1_1 which is unsupported by remote output, vector is still able to make connection to the output and forward logs to it.

        tlsSecurityProfile:
    custom:
      ciphers:
      - ECDHE-ECDSA-AES128-SHA
      minTLSVersion: VersionTLS10
    type: Custom

      This is not just for Loki output but observed for other outputs as well. 

      Version-Release number of selected component (if applicable):

      cluster-logging.v5.7.0

      CPaaS index image used: quay.io/openshift-qe-optional-operators/aosqe-index:openshift-logging-5.7.0-20230424.46

      Server Version: 4.13.0-0.nightly-2023-04-21-084440

      How reproducible:

      Always

      Steps to Reproduce:

      *Create secret in openshift-logging namespace to create HTTPS connection to external Loki

      oc create secret generic loki-client -n openshift-logging --from-literal=username=<Loki Grafana username> --from-literal=password=<loki Grafana API token>

      *Create a CLF with an unsupported ciphersuite set in custom profile.

      apiVersion: logging.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  annotations:
    logging.openshift.io/preview-tls-security-profile: enabled
  name: instance
  namespace: openshift-logging
spec:
  outputs:
  - name: loki-server
    secret:
      name: loki-client
    type: loki
    url: https://logs-prod3.grafana.net
  pipelines:
  - inputRefs:
    - application
    name: to-loki
    outputRefs:
    - loki-server
  tlsSecurityProfile:
    custom:
      ciphers:
      - ECDHE-ECDSA-AES128-SHA
      minTLSVersion: VersionTLS10
    type: Custom

      *Create a ClusterLogging instance.

      apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  annotations:
    logging.openshift.io/preview-vector-collector: enabled
  name: instance
  namespace: openshift-logging
spec:
  collection:
    logs:
      type: vector
    type: vector
  managementState: Managed

      *Check the collector pods logs, there are no errors observed in the logs and the logs are forwarded to the external output.

      $ oc logs --selector=component=collector | grpe ERROR
      bash: grpe: command not found...
      Similar command is: 'grep'
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter

      2023-04-25 19:58:48.335 {"@timestamp":"2023-04-25T14:28:48.023042867Z","file":"/var/log/pods/test_loggen-qa-json-t4jkm_b81cc1da-ad03-4c29-870b-3ab75ac6168e/loggen-qa-json/0.log","hostname":"ip-10-0-74-158.us-east-2.compute.internal","kubernetes":{"annotations":{"k8s.v1.cni.cncf.io/network-status":"[{\n \"name\": \"openshift-sdn\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.131.0.20\"\n ],\n \"default\": true,\n \"dns\": {}\n}]","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_id":"cri-o://315a34391852727d544fc5bc5a0e0894f79dd1d94eb4be9712e0ca527b816c51","container_image":"quay.io/openshifttest/ocp-logtest@sha256:16232868ba1143721b786dbabb3f7384645acb663fadb4af48e9ea1228a67635","container_name":"loggen-qa-json","labels":{"run":"centos-logtest","test":"loggen-qa-json"},"namespace_labels":{"kubernetes_io_metadata_name":"test","pod-security_kubernetes_io_audit":"restricted","pod-security_kubernetes_io_audit-version":"v1.24","pod-security_kubernetes_io_warn":"restricted","pod-security_kubernetes_io_warn-version":"v1.24"},"namespace_name":"test","pod_id":"b81cc1da-ad03-4c29-870b-3ab75ac6168e","pod_ip":"10.131.0.20","pod_name":"loggen-qa-json-t4jkm","pod_owner":"ReplicationController/loggen-qa-json"},"level":"default","log_type":"application","message":"{\"message\": \"MERGE_JSON_LOG=true\", \"level\": \"debug\",\"Layer1\": \"layer1 0\", \"layer2\": {\"name\":\"Layer2 1\", \"tips\":\"Decide by PRESERVE_JSON_LOG\"}

      , \"StringNumber\":\"10\", \"Number\": 10,\"foo.bar\":\"Dot Item\",\"{foobar}\":\"Brace Item\",\"[foobar]\":\"Bracket Item\", \"foo:bar\":\"Colon Item\",\"foo bar\":\"Space Item\" }","openshift":{"cluster_id":"e574e644-a047-45e4-b007-9bbf9c29cdab","sequence":1439}}

      Expected results:

      Vector uses the ciphersuite defined in the custom profile  to connect to the remote output. 

      Additional info:

      After setting the ClusterLogging to Unmanaged and setting VECTOR_LOG env var to TRACE in collector daemonset, we do not see any TLS protocol or ciphersuite being used in the collector logs. 

      Vector loki sink TLS config. 

      [sinks.loki_server.tls]
enabled = true
min_tls_version = "VersionTLS10"
ciphersuites = "ECDHE-ECDSA-AES128-SHA"

              syedriko_sub@redhat.com Sergey Yedrikov
              rhn-support-ikanse Ishwar Kanse
              Qiaoling Tang Qiaoling Tang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated:
                Resolved: