Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-17886

Unable to install Dynatrace operator in FIPS enabled 4.10 and 4.11 OCP clusters

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Not a Bug
    • Icon: Normal Normal
    • None
    • 4.11, 4.10
    • Node / Kubelet
    • None
    • Quality / Stability / Reliability
    • False
    • Hide

      None

      Show
      None
    • None
    • None
    • No
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      Unable to install Dynatrace operator in FIPS enabled 4.10 and 4.11 OCP clusters. The webhook containers are not become ready with below error
      
      From pod logs:
      
      ~~~
      2023-08-14T13:31:33.582301605+03:00 2023/08/14 10:31:33 http: TLS handshake error from 10.17.26.2:54422: tls: client offered only unsupported versions: [303]
      2023-08-14T13:31:41.767209430+03:00 2023/08/14 10:31:41 http: TLS handshake error from 10.17.26.2:59544: tls: client offered only unsupported versions: [303]
      2023-08-14T13:31:51.739990057+03:00 2023/08/14 10:31:51 http: TLS handshake error from 10.17.26.2:38118: tls: client offered only unsupported versions: [303]
      ~~~
      
      From namespace event:
      
      ~~~
      3m18s      Warning  Unhealthy           pod/dynatrace-webhook-559cb59845-5z7q8            Readiness probe failed: Get "https://10.17.27.233:8443/livez": dial tcp 10.17.27.233:8443: connect: connection refused
      36s        Warning  Unhealthy           pod/dynatrace-webhook-559cb59845-5z7q8            Readiness probe failed: Get "https://10.17.27.233:8443/livez": remote error: tls: protocol version not supported
      ~~~
      
      The same dynatrace operator (version 0.12.1) is able to install in OCP 4.12 and 4.13 FIPS enabled cluster.
      
      Also, the operator is able to install in 4.10 and 4.11 if FIPS is not enabled

       

      Version-Release number of selected component (if applicable):

      Dynatrace operator 0.12.1 and OCP 4.10 and 4.11 FIPS enabled cluster

      How reproducible:

      Try to install Dynatrace operator (0.12.1) version in FIPS enabled 4.10 or 4.11 cluster

      Steps to Reproduce:

      1. Create a 4.10 FIPS enabled cluster
      2. Try to install Dynatrace operator (0.12.1)
      3. Observer the dynatrace webhook contaiers are failing with below error
      
      ~~~
      2023-08-14T13:31:33.582301605+03:00 2023/08/14 10:31:33 http: TLS handshake error from 10.17.26.2:54422: tls: client offered only unsupported versions: [303]
      2023-08-14T13:31:41.767209430+03:00 2023/08/14 10:31:41 http: TLS handshake error from 10.17.26.2:59544: tls: client offered only unsupported versions: [303]
      2023-08-14T13:31:51.739990057+03:00 2023/08/14 10:31:51 http: TLS handshake error from 10.17.26.2:38118: tls: client offered only unsupported versions: [303]
      2023-08-14T13:32:01.740363826+03:00 2023/08/14 10:32:01 http: TLS handshake error from 10.17.26.2:51422: tls: client offered only unsupported versions: [303]
      ~~~
      ~~~
      dynatrace-webhook-559cb59845-5z7q8   0/1    Running  0         3m28s
      dynatrace-webhook-559cb59845-rqmgn   0/1    Running  0         3m28s
      ~~~

      Actual results:

      Webhook containers are not ready

      Expected results:

      The pod should be up

      Additional info:

      As per the dynatrace release 0.11.0 release notes Increased the minimum TLS version for the Dynatrace webhook to version 1.3
      
      I am not seeing any changes ciphersuits or tlsminversion from both the clusters (working 4.12 and non working 4.11) from kubelet conf
      
      While testing on 4.11 FIPS, it fails to communicate with TLSv13 and fall back on tlsv1
      
      ~~~
      $ echo Q | openssl s_client -tls1_3 -connect ip-10-0-130-41:10250
      CONNECTED(00000003)
      140380892469056:error:1409442E:SSL routines:ssl3_read_bytes:tlsv1 alert protocol version:ssl/record/rec_layer_s3.c:1544:SSL alert number 70
      
      SSL alert number 70: The protocol version the client attempted to negotiate is recognized, but not supported.
      
      sh-4.4# kubelet --version
      Kubernetes v1.24.15+a9da4a8
      
      sh-4.4# openssl ciphers -v | grep 1.3
      TLS_AES_256_GCM_SHA384  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(256) Mac=AEAD
      TLS_AES_128_GCM_SHA256  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(128) Mac=AEAD
      TLS_AES_128_CCM_SHA256  TLSv1.3 Kx=any      Au=any  Enc=AESCCM(128) Mac=AEAD
      
      The required cipher exist, still it fails to connect.
      ~~~
      
      From a 4.12 FIPS clusters, it can use one of the TLSv1.3 ciphers and does not trigger SSL alert.
      
      ~~~
      sh-4.4# echo Q | openssl s_client -tls1_3 -connect ip-10-0-128-77:10250
      CONNECTED(00000003)
      ...
      New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256 <---
      
      sh-4.4# kubelet --version 
      Kubernetes v1.25.11+1485cc9
      
      sh-4.4# openssl ciphers -v | grep 1.3
      TLS_AES_256_GCM_SHA384  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(256) Mac=AEAD
      TLS_AES_128_GCM_SHA256  TLSv1.3 Kx=any      Au=any  Enc=AESGCM(128) Mac=AEAD
      TLS_AES_128_CCM_SHA256  TLSv1.3 Kx=any      Au=any  Enc=AESCCM(128) Mac=AEAD
      ~~~

              aos-node@redhat.com Node Team Bot Account
              rhn-support-amuhamme MUHAMMED ASLAM V K
              None
              None
              Sunil Choudhary Sunil Choudhary
              None
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: