Uploaded image for project: 'JBoss Enterprise Application Platform'
  1. JBoss Enterprise Application Platform
  2. JBEAP-24448

Operator TX recovery facility does not work with KitchenSink quickstart

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Major Major
    • None
    • OP-2.3.4.GA
    • Operator
    • False
    • None
    • False
    • Hide

      The reproducer uses the following application: https://github.com/deewhyweb/eap-quickstarts.git

      Which is the kitchensink application adapted to be configured with the MySql Drivers during the S2I process.

      — build the image by using heml charts:

      cat <<EOF > /tmp/kitchensink-build.yaml
      image:
        tag: latest
      build:
        enabled: true
        mode: s2i
        uri: 'https://github.com/deewhyweb/eap-quickstarts.git'
        ref: 7.4.x
        contextDir: kitchensink
        output:
          kind: ImageStreamTag
        env:
          - name: MAVEN_ARGS_APPEND
            value: '-Dcom.redhat.xpaas.repo.jbossorg'
          - name: CUSTOM_INSTALL_DIRECTORIES
            value: extensions
        triggers: {}
        s2i:
          version: latest
          arch: amd64
          jdk: '11'
          amd64:
            jdk8:
              builderImage: registry.redhat.io/jboss-eap-7/eap74-openjdk8-openshift-rhel7
              runtimeImage: registry.redhat.io/jboss-eap-7/eap74-openjdk8-runtime-openshift-rhel7
            jdk11:
              builderImage: registry.redhat.io/jboss-eap-7/eap74-openjdk11-openshift-rhel8
              runtimeImage: registry.redhat.io/jboss-eap-7/eap74-openjdk11-runtime-openshift-rhel8
      deploy:
        enabled: false
      EOF
      
      
      helm install kitchensink-app \
          -f /tmp/kitchensink-build.yaml \
          jboss-eap/eap74
      

      — deploy a mysql server:

       oc new-app \
          -e MYSQL_USER=eap \
          -e MYSQL_PASSWORD=demo \
          -e MYSQL_DATABASE=eap \
          -e MYSQL_ROOT_PASSWORD=root \
          mysql:8.0-el9
      

      – create a config map with the database environment configurations:

      cat <<EOF > /tmp/kitchensink-configmap.yaml
      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: eap-config
      data: 
        DATASOURCES: "TEST"
        TEST_DATABASE: "eap"
        TEST_NAME: "mysql"
        TEST_DRIVER: "mysql"
        TEST_JNDI: "java:/jdbc/mysql"
        TEST_USERNAME: "eap"
        TEST_PASSWORD: "demo"
        TEST_URL: "jdbc:mysql://mysql:3306/eap"
        TEST_NONXA: "true"
      EOF
      
      oc apply -f /tmp/kitchensink-configmap.yaml
      

      – Deploy the Application with the Operator

      cat <<EOF > /tmp/eap-operator.yaml
      apiVersion: wildfly.org/v1alpha1
      kind: WildFlyServer
      metadata:
        name: kitchensink-operator-app
      spec:
        applicationImage: "kitchensink-app:latest"
        replicas: 3
        storage:
          volumeClaimTemplate:
            spec:
              resources:
                requests:
                  storage: 10Mi
        envFrom:
          - configMapRef:
              name: eap-config
      EOF
      
      oc apply -f /tmp/eap-operator.yaml
      
      Show
      The reproducer uses the following application: https://github.com/deewhyweb/eap-quickstarts.git Which is the kitchensink application adapted to be configured with the MySql Drivers during the S2I process. — build the image by using heml charts: cat <<EOF > /tmp/kitchensink-build.yaml image: tag: latest build: enabled: true mode: s2i uri: 'https://github.com/deewhyweb/eap-quickstarts.git' ref: 7.4.x contextDir: kitchensink output: kind: ImageStreamTag env: - name: MAVEN_ARGS_APPEND value: '-Dcom.redhat.xpaas.repo.jbossorg' - name: CUSTOM_INSTALL_DIRECTORIES value: extensions triggers: {} s2i: version: latest arch: amd64 jdk: '11' amd64: jdk8: builderImage: registry.redhat.io/jboss-eap-7/eap74-openjdk8-openshift-rhel7 runtimeImage: registry.redhat.io/jboss-eap-7/eap74-openjdk8-runtime-openshift-rhel7 jdk11: builderImage: registry.redhat.io/jboss-eap-7/eap74-openjdk11-openshift-rhel8 runtimeImage: registry.redhat.io/jboss-eap-7/eap74-openjdk11-runtime-openshift-rhel8 deploy: enabled: false EOF helm install kitchensink-app \ -f /tmp/kitchensink-build.yaml \ jboss-eap/eap74 — deploy a mysql server: oc new-app \ -e MYSQL_USER=eap \ -e MYSQL_PASSWORD=demo \ -e MYSQL_DATABASE=eap \ -e MYSQL_ROOT_PASSWORD=root \ mysql:8.0-el9 – create a config map with the database environment configurations: cat <<EOF > /tmp/kitchensink-configmap.yaml kind: ConfigMap apiVersion: v1 metadata: name: eap-config data: DATASOURCES: "TEST" TEST_DATABASE: "eap" TEST_NAME: "mysql" TEST_DRIVER: "mysql" TEST_JNDI: "java:/jdbc/mysql" TEST_USERNAME: "eap" TEST_PASSWORD: "demo" TEST_URL: "jdbc:mysql://mysql:3306/eap" TEST_NONXA: "true" EOF oc apply -f /tmp/kitchensink-configmap.yaml – Deploy the Application with the Operator cat <<EOF > /tmp/eap-operator.yaml apiVersion: wildfly.org/v1alpha1 kind: WildFlyServer metadata: name: kitchensink-operator-app spec: applicationImage: "kitchensink-app:latest" replicas: 3 storage: volumeClaimTemplate: spec: resources: requests: storage: 10Mi envFrom: - configMapRef: name: eap-config EOF oc apply -f /tmp/eap-operator.yaml

      The EAP Operator is unable to properly scale down a pod created from the KitchenSink quickstart.

      The TX recovery restarts the server after configuring it to recover transactions but the restarting process fails and the resource is deleted:

      {"level":"info","ts":1673959267.072433,"logger":"wildflyserver_controller","msg":"Restarting application server to apply the env properties","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959268.93733,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 1/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959270.311968,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 2/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959271.6165996,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 3/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959272.9772751,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 4/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959274.2855117,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 5/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959276.2841911,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 6/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959277.9949346,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 7/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959280.5484192,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 8/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959282.0426357,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 9/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959283.347628,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 10/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959291.461513,"logger":"wildflyserver_resources","msg":"Deleting Resource","WildFlyServer.Namespace":"eap-operator-demo","WildFlyServer.Name":"kitchensink-operator-app","Resource.Name":"kitchensink-operator-app-2","Resource.Type":"*v1.Pod"}
      
      {"level":"error","ts":1673959291.4804368,"logger":"controller","msg":"Reconciler error","controller":"wildflyserver-controller","name":"kitchensink-operator-app","namespace":"eap-operator-demo","error":"Found 1 errors:\n [[Cannot restart application server at pod kitchensink-operator-app-2 after setting up the periodic recovery properties. The pod was deleted to be restarted for new recovery attempt. Error: Application server was not reinitialized successfully in time. Operation ':shutdown(restart=true)' at pod kitchensink-operator-app-2, JSON management operation result: map[outcome:success result:<nil>], error: Cannot decode JBoss CLI '${JBOSS_HOME}/bin/jboss-cli.sh --output-json -c --commands=':read-attribute(name=server-state)'' executed on pod kitchensink-operator-app-2 return data 'Failed to connect to the controller: Timeout waiting for the system to boot.\n' to JSON. Cause: Fail to parse reader data to JSON, error: invalid character 'F' looking for beginning of value]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/src/github.com/wildfly/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:237\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}
      

      After increasing the number of tries from 10 to 20, the pod is scaled down, but it doesn't work always:

      {"level":"info","ts":1673962200.2645516,"logger":"wildflyserver_controller","msg":"Restarting application server to apply the env properties","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962202.2448435,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 1/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962203.7255769,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 2/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962205.0437005,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 3/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962206.3449464,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 4/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962208.2623868,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 5/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962210.413443,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 6/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962212.7975974,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 7/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962214.1961286,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 8/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962215.654061,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 9/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962223.564317,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 10/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.3337781,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 11/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.3508961,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 12/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.3673959,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 13/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.3876672,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 14/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.4055603,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 15/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.4260724,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 16/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.4430568,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 17/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.4599392,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 18/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.475704,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 19/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.4916313,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 20/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.5080538,"logger":"wildflyserver_resources","msg":"Deleting Resource","WildFlyServer.Namespace":"eap-operator-demo","WildFlyServer.Name":"kitchensink-operator-app","Resource.Name":"kitchensink-operator-app-1","Resource.Type":"*v1.Pod"}
      
      {"level":"error","ts":1673962255.5162292,"logger":"controller","msg":"Reconciler error","controller":"wildflyserver-controller","name":"kitchensink-operator-app","namespace":"eap-operator-demo","error":"Found 1 errors:\n [[Cannot restart application server at pod kitchensink-operator-app-1 after setting up the periodic recovery properties. The pod was deleted to be restarted for new recovery attempt. Error: Application server was not reinitialized successfully in time. Operation ':shutdown(restart=true)' at pod kitchensink-operator-app-1, JSON management operation result: map[outcome:success result:<nil>], error: Cannot execute JBoss CLI command ${JBOSS_HOME}/bin/jboss-cli.sh --output-json -c --commands=':read-attribute(name=server-state)' at pod kitchensink-operator-app-1. Cause: error on execution '${JBOSS_HOME}/bin/jboss-cli.sh --output-json -c --commands=':read-attribute(name=server-state)'': unable to upgrade connection: container not found (\"kitchensink-operator-app\")]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/src/github.com/wildfly/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:237\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}
      

      The number of tries can be increased by adding the following SERVER_RESTART_RETRY_COUNTER variable on the Operator subscription:

      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        labels:
          operators.coreos.com/eap.eap-operator-demo: ""
        name: eap
        namespace: eap-operator-demo
      spec:
        channel: stable
        config:
          env:
          - name: SERVER_RESTART_RETRY_COUNTER
            value: "20"
        installPlanApproval: Automatic
        name: eap
        source: redhat-operators
        sourceNamespace: openshift-marketplace
        startingCSV: eap-operator.v2.3.8
      

            yborgess1@redhat.com Yeray Borges Santana
            yborgess1@redhat.com Yeray Borges Santana
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: