Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-10846

CI fails on TestClientTLS

    XMLWordPrintable

Details

    • Important
    • 3
    • Sprint 234, Sprint 235
    • 2
    • No
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • N/A
    • Release Note Not Required

    Description

      Description of problem

      CI is flaky because the TestClientTLS test fails.

      Version-Release number of selected component (if applicable)

      I have seen these failures in 4.13 and 4.14 CI jobs.

      How reproducible

      Presently, search.ci reports the following stats for the past 14 days:

      Found in 16.07% of runs (20.93% of failures) across 56 total runs and 13 jobs (76.79% failed) in 185ms
      

      Steps to Reproduce

      1. Post a PR and have bad luck.
      2. Check https://search.ci.openshift.org/?search=FAIL%3A+TestAll%2Fparallel%2FTestClientTLS&maxAge=336h&context=1&type=all&name=cluster-ingress-operator&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job.

      Actual results

      The test fails:

      === RUN   TestAll/parallel/TestClientTLS
      === PAUSE TestAll/parallel/TestClientTLS
      === CONT  TestAll/parallel/TestClientTLS
      === CONT  TestAll/parallel/TestClientTLS
              stdout:
              Healthcheck requested
              200
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [8 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [313 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [313 bytes data]
              * TLSv1.3 (IN), TLS app data, [no content] (0):
              { [1 bytes data]
              < HTTP/1.1 200 OK
              < x-request-port: 8080
              < date: Wed, 22 Mar 2023 18:56:24 GMT
              < content-length: 22
              < content-type: text/plain; charset=utf-8
              < set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=683c60a6110214134bed475edc895cb9; path=/; HttpOnly; Secure; SameSite=None
              < cache-control: private
              <
              { [22 bytes data]
      
              * Connection #0 to host canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com left intact
      
              stdout:
              Healthcheck requested
              200
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [799 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS app data, [no content] (0):
              { [1 bytes data]
              < HTTP/1.1 200 OK
              < x-request-port: 8080
              < date: Wed, 22 Mar 2023 18:56:24 GMT
              < content-length: 22
              < content-type: text/plain; charset=utf-8
              < set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=eb40064e54af58007f579a6c82f2bcd7; path=/; HttpOnly; Secure; SameSite=None
              < cache-control: private
              <
              { [22 bytes data]
      
              * Connection #0 to host canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com left intact
      
              stdout:
              Healthcheck requested
              200
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [802 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS app data, [no content] (0):
              { [1 bytes data]
              < HTTP/1.1 200 OK
              < x-request-port: 8080
              < date: Wed, 22 Mar 2023 18:56:25 GMT
              < content-length: 22
              < content-type: text/plain; charset=utf-8
              < set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=104beed63d6a19782a5559400bd972b6; path=/; HttpOnly; Secure; SameSite=None
              < cache-control: private
              <
              { [22 bytes data]
      
              * Connection #0 to host canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com left intact
      
              stdout:
              000
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [799 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS alert, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS alert, unknown CA (560):
              { [2 bytes data]
              * OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0
      
              * Closing connection 0
              curl: (56) OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0
      
      === CONT  TestAll/parallel/TestClientTLS
              stdout:
              000
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [8 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS alert, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS alert, unknown (628):
              { [2 bytes data]
              * OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0
      
              * Closing connection 0
              curl: (56) OpenSSL SSL_read: error:1409445C:SSL routines:ssl3_read_bytes:tlsv13 alert certificate required, errno 0
      
      === CONT  TestAll/parallel/TestClientTLS
              stdout:
              Healthcheck requested
              200
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [799 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS app data, [no content] (0):
              { [1 bytes data]
              < HTTP/1.1 200 OK
              < x-request-port: 8080
              < date: Wed, 22 Mar 2023 18:57:00 GMT
              < content-length: 22
              < content-type: text/plain; charset=utf-8
              < set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=683c60a6110214134bed475edc895cb9; path=/; HttpOnly; Secure; SameSite=None
              < cache-control: private
              <
              { [22 bytes data]
      
              * Connection #0 to host canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com left intact
      
      === CONT  TestAll/parallel/TestClientTLS
              stdout:
              Healthcheck requested
              200
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [802 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
              { [1097 bytes data]
              * TLSv1.3 (IN), TLS app data, [no content] (0):
              { [1 bytes data]
              < HTTP/1.1 200 OK
              < x-request-port: 8080
              < date: Wed, 22 Mar 2023 18:57:00 GMT
              < content-length: 22
              < content-type: text/plain; charset=utf-8
              < set-cookie: c6e529a6ab19a530fd4f1cceb91c08a9=eb40064e54af58007f579a6c82f2bcd7; path=/; HttpOnly; Secure; SameSite=None
              < cache-control: private
              <
              { [22 bytes data]
      
              * Connection #0 to host canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com left intact
      
      === CONT  TestAll/parallel/TestClientTLS
              stdout:
              000
      
              stderr:
              * Added canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com:443:172.30.53.236 to DNS cache
              * Rebuilt URL to: https://canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com/
              * Hostname canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com was found in DNS cache
              *   Trying 172.30.53.236...
              * TCP_NODELAY set
                % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                               Dload  Upload   Total   Spent    Left  Speed
      
              * ALPN, offering h2
              * ALPN, offering http/1.1
              * successfully set certificate verify locations:
              *   CAfile: /etc/pki/tls/certs/ca-bundle.crt
                CApath: none
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Client hello (1):
              } [512 bytes data]
              * TLSv1.3 (IN), TLS handshake, Server hello (2):
              { [122 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
              { [10 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Request CERT (13):
              { [82 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Certificate (11):
              { [1763 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, CERT verify (15):
              { [264 bytes data]
              * TLSv1.3 (IN), TLS handshake, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS handshake, Finished (20):
              { [36 bytes data]
              * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Certificate (11):
              } [799 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, CERT verify (15):
              } [264 bytes data]
              * TLSv1.3 (OUT), TLS handshake, [no content] (0):
              } [1 bytes data]
              * TLSv1.3 (OUT), TLS handshake, Finished (20):
              } [36 bytes data]
              * SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
              * ALPN, server did not agree to a protocol
              * Server certificate:
              *  subject: CN=*.client-tls.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              *  start date: Mar 22 18:55:46 2023 GMT
              *  expire date: Mar 21 18:55:47 2025 GMT
              *  issuer: CN=ingress-operator@1679509964
              *  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
              } [5 bytes data]
              * TLSv1.3 (OUT), TLS app data, [no content] (0):
              } [1 bytes data]
              > GET / HTTP/1.1
              > Host: canary-openshift-ingress-canary.apps.ci-op-21xplx9n-43abb.origin-ci-int-aws.dev.rhcloud.com
              > User-Agent: curl/7.61.1
              > Accept: */*
              >
              { [5 bytes data]
              * TLSv1.3 (IN), TLS alert, [no content] (0):
              { [1 bytes data]
              * TLSv1.3 (IN), TLS alert, unknown CA (560):
              { [2 bytes data]
              * OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0
      
              * Closing connection 0
              curl: (56) OpenSSL SSL_read: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca, errno 0
      
      === CONT  TestAll/parallel/TestClientTLS
      --- FAIL: TestAll (1538.53s)
          --- FAIL: TestAll/parallel (0.00s)
              --- FAIL: TestAll/parallel/TestClientTLS (123.10s)
      

      Expected results

      CI passes, or it fails on a different test.

      Additional info

      I saw that TestClientTLS failed on the test case with no client certificate and ClientCertificatePolicy set to "Required". My best guess is that the test is racy and is hitting a terminating router pod. The test uses waitForDeploymentComplete to wait until all new pods are available, but perhaps waitForDeploymentComplete should also wait until all old pods are terminated.

      Attachments

        Issue Links

          Activity

            People

              rfredett@redhat.com Ryan Fredette
              mmasters1@redhat.com Miciah Masters
              Shudi Li Shudi Li
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: