Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-16621

CI fails on TestClientTLS

XMLWordPrintable

    • Important
    • No
    • 1
    • Sprint 239, Sprint 240
    • 2
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • N/A
    • Release Note Not Required

      This is a clone of issue OCPBUGS-10846. The following is the description of the original issue:

      Description of problem

      CI is flaky because the TestClientTLS test fails.

      Version-Release number of selected component (if applicable)

      I have seen these failures in 4.13 and 4.14 CI jobs.

      How reproducible

      Presently, search.ci reports the following stats for the past 14 days:

      Found in 16.07% of runs (20.93% of failures) across 56 total runs and 13 jobs (76.79% failed) in 185ms
      

      Steps to Reproduce

      1. Post a PR and have bad luck.
      2. Check https://search.ci.openshift.org/?search=FAIL%3A+TestAll%2Fparallel%2FTestClientTLS&maxAge=336h&context=1&type=all&name=cluster-ingress-operator&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job.

      Actual results

      The test fails:

      === RUN   TestAll/parallel/TestClientTLS
      === PAUSE TestAll/parallel/TestClientTLS
      === CONT  TestAll/parallel/TestClientTLS
      === CONT  TestAll/parallel/TestClientTLS
              stdout:
              Healthcheck requested
              200
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [8 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [313 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [313 bytes data]
              * TLSv1.3 (IN), TLS app data, [no content] (0):
              { [1 bytes data]
              < HTTP/1.1 200 OK
              < x-request-port: 8080
              < date: Wed, 22 Mar 2023 18:56:24 GMT
              < content-length: 22
              < content-type: text/plain; charset=utf-8
              < set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=683c60a6110214134bed475edc895cb9; path=/; HttpOnly; Secure; SameSite=None
              < cache-control: private
              <
              { [22 bytes data]
      
              * Connection #0 to host canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com left intact
      
              stdout:
              Healthcheck requested
              200
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [799 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS app data, [no content] (0):
              { [1 bytes data]
              < HTTP/1.1 200 OK
              < x-request-port: 8080
              < date: Wed, 22 Mar 2023 18:56:24 GMT
              < content-length: 22
              < content-type: text/plain; charset=utf-8
              < set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=eb40064e54af58007f579a6c82f2bcd7; path=/; HttpOnly; Secure; SameSite=None
              < cache-control: private
              <
              { [22 bytes data]
      
              * Connection #0 to host canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com left intact
      
              stdout:
              Healthcheck requested
              200
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [802 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS app data, [no content] (0):
              { [1 bytes data]
              < HTTP/1.1 200 OK
              < x-request-port: 8080
              < date: Wed, 22 Mar 2023 18:56:25 GMT
              < content-length: 22
              < content-type: text/plain; charset=utf-8
              < set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=104beed63d6a19782a5559400bd972b6; path=/; HttpOnly; Secure; SameSite=None
              < cache-control: private
              <
              { [22 bytes data]
      
              * Connection #0 to host canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com left intact
      
              stdout:
              000
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [799 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS alert, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS alert, unknown CA (560):
              { [2 bytes data]
              * OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0
      
              * Closing connection 0
              curl: (56) OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0
      
      === CONT  TestAll/parallel/TestClientTLS
              stdout:
              000
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [8 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS alert, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS alert, unknown (628):
              { [2 bytes data]
              * OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0
      
              * Closing connection 0
              curl: (56) OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0
      
      === CONT  TestAll/parallel/TestClientTLS
              stdout:
              Healthcheck requested
              200
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [799 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS app data, [no content] (0):
              { [1 bytes data]
              < HTTP/1.1 200 OK
              < x-request-port: 8080
              < date: Wed, 22 Mar 2023 18:57:00 GMT
              < content-length: 22
              < content-type: text/plain; charset=utf-8
              < set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=683c60a6110214134bed475edc895cb9; path=/; HttpOnly; Secure; SameSite=None
              < cache-control: private
              <
              { [22 bytes data]
      
              * Connection #0 to host canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com left intact
      
      === CONT  TestAll/parallel/TestClientTLS
              stdout:
              Healthcheck requested
              200
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [802 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS app data, [no content] (0):
              { [1 bytes data]
              < HTTP/1.1 200 OK
              < x-request-port: 8080
              < date: Wed, 22 Mar 2023 18:57:00 GMT
              < content-length: 22
              < content-type: text/plain; charset=utf-8
              < set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=eb40064e54af58007f579a6c82f2bcd7; path=/; HttpOnly; Secure; SameSite=None
              < cache-control: private
              <
              { [22 bytes data]
      
              * Connection #0 to host canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com left intact
      
      === CONT  TestAll/parallel/TestClientTLS
              stdout:
              000
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [799 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS alert, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS alert, unknown CA (560):
              { [2 bytes data]
              * OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0
      
              * Closing connection 0
              curl: (56) OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0
      
      === CONT  TestAll/parallel/TestClientTLS
      --- FAIL: TestAll (1538.53s)
          --- FAIL: TestAll/parallel (0.00s)
              --- FAIL: TestAll/parallel/TestClientTLS (123.10s)
      

      Expected results

      CI passes, or it fails on a different test.

      Additional info

      I saw that TestClientTLS failed on the test case with no client certificate and ClientCertificatePolicy set to "Required". My best guess is that the test is racy and is hitting a terminating router pod. The test uses waitForDeploymentComplete to wait until all new pods are available, but perhaps waitForDeploymentComplete should also wait until all old pods are terminated.

            mmasters1@redhat.com Miciah Masters
            openshift-crt-jira-prow OpenShift Prow Bot
            Shudi Li Shudi Li
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: