Uploaded image for project: 'WildFly WIP'
  1. WildFly WIP
  2. WFWIP-248

Avoid Route unavailability caused by load balancer recreation on CRD change

    XMLWordPrintable

Details

    Description

      Tracked by https://github.com/wildfly/wildfly-operator/issues/117

      Reproduce:

      ### REPRODUCER TERMINAL
      # deploy quickstart project
      oc new-project wildfly-test
      wildfly-operator]$ ./build/run-openshift.sh
      oc apply -f deploy/crds/quickstart-cr.yaml
      wildfly-operator]$ oc get pods
      NAME                                READY   STATUS    RESTARTS   AGE
      quickstart-0                        1/1     Running   0          34s
      quickstart-1                        1/1     Running   0          34s
      wildfly-operator-7f555b86d5-jkctc   1/1     Running   0          99s
      
      # verify the app is responsive
      ROUTE=quickstart-route-wildfly-test.apps-crc.testing
      curl $ROUTE
      {"ip":"10.128.1.43"}
      
      # watch pods
      oc get pods -w &
      # verify route readiness
      ROUTE=quickstart-route-wildfly-test.apps-crc.testing
      # I've seen some weird bash optimization using `while true` in past which I'd like to avoid here...
      for i in {0..100000} ; do 
      	echo '----------------------------------------' 
      	date 
      	curl -sI $ROUTE | grep HTTP
      	curl -s $ROUTE | head -n 1
      	echo '' 
      	oc describe service/quickstart-loadbalancer | grep Endpoints 
      	sleep 0.5
      done
      
      ### ADMIN TERMINAL - scale the application
      oc patch wildflyserver quickstart -p '[{"op":"replace", "path":"/spec/replicas", "value":3}]' --type json
      

      output:

      ----------------------------------------
      Mon 14 Oct 2019 01:12:09 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      quickstart-2                        0/1     Pending   0          0s
      quickstart-2                        0/1     Pending       0          0s
      quickstart-2                        0/1     Pending       0          0s
      quickstart-2                        0/1     Pending   0          0s
      quickstart-2                        0/1     ContainerCreating   0          0s
      quickstart-2                        0/1     ContainerCreating   0          0s
      Error from server (NotFound): services "quickstart-loadbalancer" not found
      ----------------------------------------
      Mon 14 Oct 2019 01:12:09 PM CEST
      HTTP/1.0 503 Service Unavailable
      <html>
      
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:10 PM CEST
      HTTP/1.0 503 Service Unavailable
      <html>
      
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:11 PM CEST
      HTTP/1.0 503 Service Unavailable
      <html>
      
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:12 PM CEST
      HTTP/1.0 503 Service Unavailable
      <html>
      
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:13 PM CEST
      HTTP/1.0 503 Service Unavailable
      <html>
      
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:13 PM CEST
      HTTP/1.0 503 Service Unavailable
      <html>
      
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:14 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:15 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:16 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080
      quickstart-2quickstart-2                                                0/10/1          ContainerCreatingContainerCreating      00                    8s8s
      
      ----------------------------------------
      Mon 14 Oct 2019 01:12:17 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080
      quickstart-2                        1/1     Running             0          8s
      quickstart-2                        1/1     Running             0          8s
      ----------------------------------------
      Mon 14 Oct 2019 01:12:17 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:18 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:19 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:20 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:24 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:24 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:25 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:26 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:27 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:28 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:28 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:29 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:30 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:31 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:32 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:32 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:33 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:34 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.44"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:35 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.43"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      ----------------------------------------
      Mon 14 Oct 2019 01:12:36 PM CEST
      HTTP/1.1 200 OK
      {"ip":"10.128.1.47"}
      Endpoints:                10.128.1.43:8080,10.128.1.44:8080,10.128.1.47:8080
      

      Attachments

        Activity

          People

            jmesnil1@redhat.com Jeff Mesnil
            pkremens@redhat.com Petr Kremensky (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: