Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-25454

Problems with Unsetting minTLSVersion in Custom tlsSecurityProfile Leading to Failures

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • None
    • 4.15.z
    • apiserver-auth
    • No
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      Unsetting the `minTLSVersion` in `apiserver.spec.tlsSecurityProfile` with a custom type leads to authentication CO unavailability and pods entering a CrashLoopBackOff state. There's a concern that setting the wrong version could cause similar issues. Implementing an API hook could help avoid such problems by enforcing correct configuration.

      Version-Release number of selected component (if applicable):

      4.15.0-0.nightly-2023-12-14-115151    

      How reproducible:

      Always    

      Steps to Reproduce:

      1. Unset `minTLSVersion` in `apiserver.spec.tlsSecurityProfile` when the type is set to Custom.
      $ oc get apiserver cluster -o yaml 
      apiVersion: config.openshift.io/v1
      kind: APIServer
      ...
        tlsSecurityProfile:
          custom:
            ciphers:
            - ECDHE-ECDSA-CHACHA20-POLY1305
            - ECDHE-ECDSA-AES128-GCM-SHA256
          type: Custom
      
      2. Observe the status of the authentication CO and pods in `openshift-oauth-apiserver`.
      # oc get co --no-headers | grep -v '.True.*False.*False'
      authentication                             4.15.0-0.nightly-2023-12-14-115151   False   False   True    40m   APIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node....
      console                                    4.15.0-0.nightly-2023-12-14-115151   True    False   True    74m   OAuthClientSyncDegraded: the server is currently unable to handle the request (get oauthclients.oauth.openshift.io console)...
      storage                                    4.15.0-0.nightly-2023-12-14-115151   True    True    False   92m   AWSEBSCSIDriverOperatorCRProgressing: AWSEBSDriverControllerServiceControllerProgressing: Waiting for Deployment to deploy pods# oc get co authentication
      NAME             VERSION                              AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
      authentication   4.15.0-0.nightly-2023-12-14-115151   False       False         True       40m     APIServerDeploymentAvailable: no apiserver.openshift-oauth-apiserver pods available on any node....
      # oc -n openshift-oauth-apiserver get pod
      NAME                         READY   STATUS             RESTARTS         AGE
      apiserver-868885f66f-6l2nf   0/1     CrashLoopBackOff   13 (3m17s ago)   45m
      apiserver-868885f66f-twsbp   0/1     CrashLoopBackOff   13 (3m48s ago)   43m
      apiserver-868885f66f-xftcg   0/1     CrashLoopBackOff   13 (3m26s ago)   44m
      
      
      3. Check the logs for `--tls-min-version` argument and connection errors.
      # oc -n openshift-oauth-apiserver logs apiserver-868885f66f-6l2nf
      Defaulted container "oauth-apiserver" out of: oauth-apiserver, fix-audit-permissions (init)...
      I1215 11:25:28.102311       1 flags.go:64] FLAG: --tls-cipher-suites="[TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256]"
      I1215 11:25:28.102314       1 flags.go:64] FLAG: --tls-min-version=""
      ......
      I1215 11:25:28.406657       1 apf_controller.go:846] Introducing queues for priority level "catch-all": config={"type":"Limited","limited":{"nominalConcurrencyShares":5,"limitResponse":{"type":"Reject"},"lendablePercent":0}}, nominalCL=600, lendableCL=0, borrowingCL=600, currentCL=600, quiescing=false (shares=5, shareSum=5)
      W1215 11:25:28.412109       1 logging.go:59] [core] [Channel #1 SubChannel #3] grpc: addrConn.createTransport failed to connect to {
        "Addr": "10.0.81.181:2379",
        "ServerName": "10.0.81.181",
        "Attributes": null,
        "BalancerAttributes": null,
        "Type": 0,
        "Metadata": null
      }. Err: connection error: desc = "transport: authentication handshake failed: remote error: tls: handshake failure"
      

      Actual results:

      - Authentication CO becomes unavailable.
      - Pods in `openshift-oauth-apiserver` exhibit CrashLoopBackOff or Error states.
      - Logs display `--tls-min-version=""` and connection errors due to TLS handshake failures.    

      Expected results:

      - Authentication CO should remain available.
      - Pods should be stable, without CrashLoopBackOff or Error.
      - No connection errors related to TLS in logs.    

      Additional info:

      - Suggestion to introduce an API hook to validate and reject configurations without `minTLSVersion`.
      - Aim to remind users to set the version correctly and avoid widespread component issues.
      - The issue has been tested and confirmed on multiple clusters, indicating a broader impact.    

            rh-ee-irinis Ilias Rinis
            rhn-support-dpunia Deepak Punia
            Xingxing Xia Xingxing Xia
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: