Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-5073

Connection reset by peer issue with SSL OAuth Proxy when route objects are created more than 80.

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Normal
    • None
    • 4.11
    • Networking / router
    • None
    • Moderate
    • Sprint 229, Sprint 230, Sprint 231
    • 3
    • Rejected
    • False
    • Hide

      None

      Show
      None

    Description

      Description of problem:

      When routes are created more than 80, SSL connections between OAuth Proxy container and HAProxy are disconnected with the following error messages
      2022/12/15 21:37:01 server.go:3120: http: TLS handshake error from 10.128.18.27:47142: write tcp 10.128.10.57:8443->10.128.18.27:47142: write: connection reset by peer 
      With Model serving with 100 connections made OAuth proxy container failed so model serving pod failed too.

      Version-Release number of selected component (if applicable):

       

      How reproducible:

      Always

      Steps to Reproduce:

      git clone https://github.com/Jooho/jhouse_openshift.git
      cd jhouse_openshift/OAuthProxy/https-reencrypt 
      
      oc new-project oauth-proxy
      oc new-app -S php -n oauth-proxy
      oc new-app --template=cakephp-mysql-example -n oauth-proxy
      oc apply -f ./
      oc replace -f ./svc-cakephp-mysql-example.yaml
      oc scale dc/cakephp-mysql-example --replicas=2
      
      # Wait until all pods are running. 
      export Token=$(oc sa new-token user-one)
      export URL=$(oc get route cakephp-mysql-example -ojsonpath='{.spec.host}')
      curl -o /dev/null -I -w "%{http_code}"  --silent --location --fail --show-error --insecure https://${URL}/ -H "Authorization: Bearer ${Token}"
      
      # Start reproducing the error
      cat <<EOF> /tmp/cakephp.yaml
      apiVersion: route.openshift.io/v1
      kind: Route
      metadata:
        labels:
          app: cakephp-mysql-example
          template: cakephp-mysql-example
        name: cakephp-mysql-example
        namespace: oauth-proxy
      spec:
        port:
          targetPort: oauth-https
        tls:
          insecureEdgeTerminationPolicy: Redirect
          termination: reencrypt
        to:
          kind: Service
          name: cakephp-mysql-example
          weight: 100
        wildcardPolicy: None
      EOF
      
      for i in {1..100} ; do sed "7s/name:.*/name: cakephp-mysql-example-$i/g" /tmp/cakephp.yaml |oc apply -f - ; done
      
      # Check the error
      oc logs dc/cakephp-mysql-example  -c oauth-proxy

      Actual results:

      Disconnected connections between OAuth Proxy and HAProxy

      Expected results:

      No errors happen

      Additional info:

      When I set replica of Router to 1, the issue was gone. However, if I increased it to 3, the issue was still around. So I don't think it is resource limitation issue. 

       

      Attachments

        1. dc.yaml
          4 kB
        2. image-2023-01-09-13-02-03-062.png
          image-2023-01-09-13-02-03-062.png
          29 kB
        3. oauth.tar.xz
          63 kB
        4. screenshot-2.png
          screenshot-2.png
          144 kB
        5. screenshot-3.png
          screenshot-3.png
          42 kB
        6. screenshot-4.png
          screenshot-4.png
          46 kB
        7. screenshot-5.png
          screenshot-5.png
          43 kB
        8. screenshot-6.png
          screenshot-6.png
          20 kB

        Issue Links

          Activity

            People

              gspence@redhat.com Grant Spence
              rhn-support-jlee JOOHO LEE
              Melvin Joseph Melvin Joseph
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: