Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-4011

[release-5.7] [Vector] Collector not complying with the custom tlsSecurityProfile configuration.

XMLWordPrintable

    • False
    • None
    • False
    • NEW
    • OBSDA-160 - Comply with OCP cluster-wide cryptographic policies
    • VERIFIED
    • Prior to this update, vector log collector did not honor the tlsSecurityProfile settings for outgoing TLS connections. After this update, vector does honor the tlsSecurityProfile settings for outgoing TLS connections.
    • Bug Fix
    • Log Collection - Sprint 236, Log Collection - Sprint 237

      Description of problem:

      When a custom tlsSecurityProfile is set with a ciphersuite not supported by remote server, the connection to the remote output should fail. For example connecting to an externally hosted loki, the remote Loki output only supports TLS v1.2 and above.

      openssl s_client -tls1_1 --connect logs-prod3.grafana.net:443
      CONNECTED(00000003)
      808BA8CDF77F0000:error:0A00042E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1605:SSL alert number 70
      —
      no peer certificate available
      —
      No client certificate CA names sent
      —
      SSL handshake has read 7 bytes and written 135 bytes
      Verification: OK
      —
      New, (NONE), Cipher is (NONE)
      Secure Renegotiation IS NOT supported
      Compression: NONE
      Expansion: NONE
      No ALPN negotiated
      SSL-Session:
          Protocol  : TLSv1.1
          Cipher    : 0000
          Session-ID: 
          Session-ID-ctx: 
          Master-Key: 
          PSK identity: None
          PSK identity hint: None
          SRP username: None
          Start Time: 1682431553
          Timeout   : 7200 (sec)
          Verify return code: 0 (ok)
          Extended master secret: no
      —
      $ openssl s_client -tls1_2 --connect logs-prod3.grafana.net:443
      CONNECTED(00000003)
      depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
      verify return:1
      depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
      verify return:1
      depth=0 C = US, ST = New York, L = New York, O = Raintank Inc., CN = grafana.com
      verify return:1
      —
      Certificate chain
       0 s:C = US, ST = New York, L = New York, O = Raintank Inc., CN = grafana.com
         i:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
         a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
         v:NotBefore: Jan 31 00:00:00 2023 GMT; NotAfter: Mar  2 23:59:59 2024 GMT
       1 s:C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
         i:C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
         a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
         v:NotBefore: Apr 14 00:00:00 2021 GMT; NotAfter: Apr 13 23:59:59 2031 GMT
      —
      Server certificate
      ----BEGIN CERTIFICATE----
      MIIHbDCCBlSgAwIBAgIQCLtVHY4/+djh9O3AfHzjjDANBgkqhkiG9w0BAQsFADBP
      MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMSkwJwYDVQQDEyBE
      aWdpQ2VydCBUTFMgUlNBIFNIQTI1NiAyMDIwIENBMTAeFw0yMzAxMzEwMDAwMDBa
      Fw0yNDAzMDIyMzU5NTlaMGExCzAJBgNVBAYTAlVTMREwDwYDVQQIEwhOZXcgWW9y
      azERMA8GA1UEBxMITmV3IFlvcmsxFjAUBgNVBAoTDVJhaW50YW5rIEluYy4xFDAS
      BgNVBAMTC2dyYWZhbmEuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
      AQEAoj+bmhK4AUvv9/dK5aDxBn5l01BrkATCPIR43uj5zxGCyGBoX7n5pFWsEAMb
      rkdSfqRluoFTi8f/u9ECcMwF1M4ISyoQ6iPh7E6pkQULoAgITLptiANVzyw30pxl
      RRKmNqxC4fzEXwxc3qHgaWB+xMwC/i3vExnqHoAUYnvejXT8W/Szmbm8eX5WLTbe
      r21hDfGRMztlUur3UfQ9e2UVe8dsLKMLXjmu6uHsBJ4I1tOT81RohL0+/OiPi9ct
      JL/TpbLRdLFCpJd7aLT186R4e50qTwIQVBXLG91wdvoxd9oaxSp9xTHfkYW0OqMM
      ci9ZqUbJ9FahtCjDhDBaQeym9wIDAQABo4IEMDCCBCwwHwYDVR0jBBgwFoAUt2ui
      6qiqhIx56rTaD5iyxZV2ufQwHQYDVR0OBBYEFEFCsIHtSmR60zTDE6D0yaNtX4RG
      MIHbBgNVHREEgdMwgdCCC2dyYWZhbmEuY29tgg0qLmdyYWZhbmEuY29tgg0qLmdy
      YWZhbmEubmV0gg0qLmdyYWZhbmEub3JnghwqLmhvc3RlZC1tZXRyaWNzLmdyYWZh
      bmEubmV0gg0qLnJhaW50YW5rLmlvggtncmFmYW5hLm5ldIILZ3JhZmFuYS5vcmeC
      C3JhaW50YW5rLmlvghEqLmdyYWZhbmFsYWJzLmNvbYIPZ3JhZmFuYWxhYnMuY29t
      gg4qLnJhaW50YW5rLmNvbYIMcmFpbnRhbmsuY29tMA4GA1UdDwEB/wQEAwIFoDAd
      BgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwgY8GA1UdHwSBhzCBhDBAoD6g
      PIY6aHR0cDovL2NybDMuZGlnaWNlcnQuY29tL0RpZ2lDZXJ0VExTUlNBU0hBMjU2
      MjAyMENBMS00LmNybDBAoD6gPIY6aHR0cDovL2NybDQuZGlnaWNlcnQuY29tL0Rp
      Z2lDZXJ0VExTUlNBU0hBMjU2MjAyMENBMS00LmNybDA+BgNVHSAENzA1MDMGBmeB
      DAECAjApMCcGCCsGAQUFBwIBFhtodHRwOi8vd3d3LmRpZ2ljZXJ0LmNvbS9DUFMw
      fwYIKwYBBQUHAQEEczBxMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5kaWdpY2Vy
      dC5jb20wSQYIKwYBBQUHMAKGPWh0dHA6Ly9jYWNlcnRzLmRpZ2ljZXJ0LmNvbS9E
      aWdpQ2VydFRMU1JTQVNIQTI1NjIwMjBDQTEtMS5jcnQwCQYDVR0TBAIwADCCAX0G
      CisGAQQB1nkCBAIEggFtBIIBaQFnAHUA7s3QZNXbGs7FXLedtM0TojKHRny87N7D
      UUhZRnEftZsAAAGGCBdyVQAABAMARjBEAh8ueDr4NtVhJSQ6wkBCZQNiWPviyAx+
      L95Bclm2OPm8AiEA8OnhpIHH1vMZgAwlZR2szZMMJcfhjL0jNodTlw1YH84AdgBz
      2Z6JG0yWeKAgfUed5rLGHNBRXnEZKoxrgBB6wXdytQAAAYYIF3JFAAAEAwBHMEUC
      IQCmj8ZWqspndtvQBFcA/zSkIsDt2k5u+cMsEqEgSQpNCAIgB2rfvSP1fDK0eGmN
      nJ+46bp3IBPNDa9jcxR0j6T4h14AdgBIsONr2qZHNA/lagL6nTDrHFIBy1bdLIHZ
      u7+rOdiEcwAAAYYIF3IVAAAEAwBHMEUCIFoSQTsvCBbeTuzhqApnd8rFbcR+aQpI
      jOeuO54sGiMqAiEAlLIWxwG0QWLj+Qw7vBQ7EcTw8GeGjRfEw5uxE+gyT6gwDQYJ
      KoZIhvcNAQELBQADggEBAEIm5uKQm/7/qJpIPGg1fKE5K46ZLSSL65MLbqTKgggO
      /bPJZ4W4heCx1fS4Fnyq83+bg1W/fEzuz6BY9RfYmIroz6MIdB0zDUzu/Ntbf0CR
      Z4D83TsS/UM+rTtznugLArSXR4ERomIm195YKJKB1vbemiSxSi55CsDk+9Fc11Tm
      dw5TRa9lwTiw8cO6+L+zYg9vgm2pl3oT/DYpPcf8Ncy+fJOdghwa37HGvrVD5VoQ
      vD+OABv6Gjp3EbDFOvN3LZhY28XW/SEeleFTbs2DY1Wumk3qbCUms3Qq2251i3QT
      FcAHfINhretx7ZFqxXH4VeoRTNtHEa+IcZZrxI9/HHA=
      ----END CERTIFICATE----
      subject=C = US, ST = New York, L = New York, O = Raintank Inc., CN = grafana.com
      issuer=C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
      —
      No client certificate CA names sent
      Peer signing digest: SHA256
      Peer signature type: RSA-PSS
      Server Temp Key: X25519, 253 bits
      —
      SSL handshake has read 3809 bytes and written 304 bytes
      Verification: OK
      —
      New, TLSv1.2, Cipher is ECDHE-RSA-CHACHA20-POLY1305
      Server public key is 2048 bit
      Secure Renegotiation IS supported
      Compression: NONE
      Expansion: NONE
      No ALPN negotiated
      SSL-Session:
          Protocol  : TLSv1.2
          Cipher    : ECDHE-RSA-CHACHA20-POLY1305
          Session-ID: E81F8A38184A9462DBE281A9C7534AEADDDFF4A9F071E7E396D354133C923FF0
          Session-ID-ctx: 
          Master-Key: 2FA60364F87854A8E8BC9D6B0EC6759A37E8136E3C6B7C27F613913E31ED38C19795464835921DF86B532788AAB0AD56
          PSK identity: None
          PSK identity hint: None
          SRP username: None
          TLS session ticket lifetime hint: 100800 (seconds)
          TLS session ticket:
          0000 - 02 97 1b 8d 1a e5 b4 1b-32 17 b1 70 ca 40 22 23   ........2..p.@"#
          0010 - d3 a7 04 20 7e 83 61 08-31 92 2b 81 14 1c 3f a2   ... ~.a.1.+...?.
          0020 - 97 70 c4 09 49 2f 89 89-a2 55 0b 8f 56 77 a0 18   .p..I/...U..Vw..
          0030 - 0c 2d 47 87 ab 00 8b 1d-44 58 07 0d 36 ce 89 40   .-G.....DX..6..@
          0040 - f0 b5 7b e7 bd 37 5e 67-d1 3f 16 17 86 56 02 41   ..{..7^g.?...V.A
          0050 - f8 17 82 07 a8 9f 7a fb-ab 07 ef 46 fe 5c dc dc   ......z....F.\..
          0060 - 54 3b 79 69 9b 9b dc 50-a9 85 3b 9b fe a6 f9 55   T;yi...P..;....U
          0070 - 31 8f 6d d6 be 6b 88 c7-22 53 5e f9 c4 57 31 f8   1.m..k.."S^..W1.
          0080 - f6 bd e5 67 e4 c2 92 eb-da 50 85 d7 99 17 42 fe   ...g.....P....B.
          0090 - 14 a5 0e 59 d3 e3 f8 a5-16 80 a2 5b 72 86 05 a8   ...Y.......[r...
          00a0 - 05 72 12 9f 33 79 8a a0-b5 db c7 c7 4a 0c cc 71   .r..3y......J..q
          00b0 - 77 b2 b3 e0 20 0f eb 54-bb 8a 34 cd fb 1a a0 95   w... ..T..4.....
          00c0 - cd 8b 6a e4 2f 24 d8 a3-a0 fe 55 6b 8b ba b1 b5   ..j./$....Uk....
          00d0 - 3c 7f a0 e5 cd a7 74 bb-8b 2c 42 4f 4d 04 39 67   <.....t..,BOM.9g
          00e0 - b8 95 b5 ed 51                                    ....Q
          Start Time: 1682431845
          Timeout   : 7200 (sec)
          Verify return code: 0 (ok)
          Extended master secret: yes
      —
      

      However if we use a ciphersuite from TLSv1_1 which is unsupported by remote output, vector is still able to make connection to the output and forward logs to it.

        tlsSecurityProfile:
          custom:
            ciphers:
            - ECDHE-ECDSA-AES128-SHA
            minTLSVersion: VersionTLS10
          type: Custom

      This is not just for Loki output but observed for other outputs as well. 

      Version-Release number of selected component (if applicable):

      cluster-logging.v5.7.0

      CPaaS index image used: quay.io/openshift-qe-optional-operators/aosqe-index:openshift-logging-5.7.0-20230424.46

      Server Version: 4.13.0-0.nightly-2023-04-21-084440

      How reproducible:

      Always

      Steps to Reproduce:

      *Create secret in openshift-logging namespace to create HTTPS connection to external Loki

      oc create secret generic loki-client -n openshift-logging --from-literal=username=<Loki Grafana username> --from-literal=password=<loki Grafana API token>

      *Create a CLF with an unsupported ciphersuite set in custom profile.

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogForwarder
      metadata:
        annotations:
          logging.openshift.io/preview-tls-security-profile: enabled
        name: instance
        namespace: openshift-logging
      spec:
        outputs:
        - name: loki-server
          secret:
            name: loki-client
          type: loki
          url: https://logs-prod3.grafana.net
        pipelines:
        - inputRefs:
          - application
          name: to-loki
          outputRefs:
          - loki-server
        tlsSecurityProfile:
          custom:
            ciphers:
            - ECDHE-ECDSA-AES128-SHA
            minTLSVersion: VersionTLS10
          type: Custom

      *Create a ClusterLogging instance.

      apiVersion: logging.openshift.io/v1
      kind: ClusterLogging
      metadata:
        annotations:
          logging.openshift.io/preview-vector-collector: enabled
        name: instance
        namespace: openshift-logging
      spec:
        collection:
          logs:
            type: vector
          type: vector
        managementState: Managed

      *Check the collector pods logs, there are no errors observed in the logs and the logs are forwarded to the external output.

      $ oc logs --selector=component=collector | grpe ERROR
      bash: grpe: command not found...
      Similar command is: 'grep'
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter
      Defaulted container "collector" out of: collector, logfilesmetricexporter

      2023-04-25 19:58:48.335 {"@timestamp":"2023-04-25T14:28:48.023042867Z","file":"/var/log/pods/test_loggen-qa-json-t4jkm_b81cc1da-ad03-4c29-870b-3ab75ac6168e/loggen-qa-json/0.log","hostname":"ip-10-0-74-158.us-east-2.compute.internal","kubernetes":{"annotations":{"k8s.v1.cni.cncf.io/network-status":"[{\n \"name\": \"openshift-sdn\",\n \"interface\": \"eth0\",\n \"ips\": [\n \"10.131.0.20\"\n ],\n \"default\": true,\n \"dns\": {}\n}]","openshift.io/scc":"restricted-v2","seccomp.security.alpha.kubernetes.io/pod":"runtime/default"},"container_id":"cri-o://315a34391852727d544fc5bc5a0e0894f79dd1d94eb4be9712e0ca527b816c51","container_image":"quay.io/openshifttest/ocp-logtest@sha256:16232868ba1143721b786dbabb3f7384645acb663fadb4af48e9ea1228a67635","container_name":"loggen-qa-json","labels":{"run":"centos-logtest","test":"loggen-qa-json"},"namespace_labels":{"kubernetes_io_metadata_name":"test","pod-security_kubernetes_io_audit":"restricted","pod-security_kubernetes_io_audit-version":"v1.24","pod-security_kubernetes_io_warn":"restricted","pod-security_kubernetes_io_warn-version":"v1.24"},"namespace_name":"test","pod_id":"b81cc1da-ad03-4c29-870b-3ab75ac6168e","pod_ip":"10.131.0.20","pod_name":"loggen-qa-json-t4jkm","pod_owner":"ReplicationController/loggen-qa-json"},"level":"default","log_type":"application","message":"{\"message\": \"MERGE_JSON_LOG=true\", \"level\": \"debug\",\"Layer1\": \"layer1 0\", \"layer2\": {\"name\":\"Layer2 1\", \"tips\":\"Decide by PRESERVE_JSON_LOG\"}

      , \"StringNumber\":\"10\", \"Number\": 10,\"foo.bar\":\"Dot Item\",\"{foobar}\":\"Brace Item\",\"[foobar]\":\"Bracket Item\", \"foo:bar\":\"Colon Item\",\"foo bar\":\"Space Item\" }","openshift":{"cluster_id":"e574e644-a047-45e4-b007-9bbf9c29cdab","sequence":1439}}

      Expected results:

      Vector uses the ciphersuite defined in the custom profile  to connect to the remote output. 

      Additional info:

      After setting the ClusterLogging to Unmanaged and setting VECTOR_LOG env var to TRACE in collector daemonset, we do not see any TLS protocol or ciphersuite being used in the collector logs. 

      Vector loki sink TLS config. 

      [sinks.loki_server.tls]
      enabled = true
      min_tls_version = "VersionTLS10"
      ciphersuites = "ECDHE-ECDSA-AES128-SHA"

            syedriko_sub@redhat.com Sergey Yedrikov
            rhn-support-ikanse Ishwar Kanse
            Qiaoling Tang Qiaoling Tang
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Created:
              Updated:
              Resolved: