-
Bug
-
Resolution: Cannot Reproduce
-
Major
-
None
-
OP-2.3.4.GA
-
False
-
None
-
False
-
-
-
-
-
-
-
-
The EAP Operator is unable to properly scale down a pod created from the KitchenSink quickstart.
The TX recovery restarts the server after configuring it to recover transactions but the restarting process fails and the resource is deleted:
{"level":"info","ts":1673959267.072433,"logger":"wildflyserver_controller","msg":"Restarting application server to apply the env properties","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959268.93733,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 1/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959270.311968,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 2/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959271.6165996,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 3/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959272.9772751,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 4/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959274.2855117,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 5/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959276.2841911,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 6/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959277.9949346,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 7/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959280.5484192,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 8/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959282.0426357,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 9/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959283.347628,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 10/10","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-2"} \{"level":"info","ts":1673959291.461513,"logger":"wildflyserver_resources","msg":"Deleting Resource","WildFlyServer.Namespace":"eap-operator-demo","WildFlyServer.Name":"kitchensink-operator-app","Resource.Name":"kitchensink-operator-app-2","Resource.Type":"*v1.Pod"} {"level":"error","ts":1673959291.4804368,"logger":"controller","msg":"Reconciler error","controller":"wildflyserver-controller","name":"kitchensink-operator-app","namespace":"eap-operator-demo","error":"Found 1 errors:\n [[Cannot restart application server at pod kitchensink-operator-app-2 after setting up the periodic recovery properties. The pod was deleted to be restarted for new recovery attempt. Error: Application server was not reinitialized successfully in time. Operation ':shutdown(restart=true)' at pod kitchensink-operator-app-2, JSON management operation result: map[outcome:success result:<nil>], error: Cannot decode JBoss CLI '${JBOSS_HOME}/bin/jboss-cli.sh --output-json -c --commands=':read-attribute(name=server-state)'' executed on pod kitchensink-operator-app-2 return data 'Failed to connect to the controller: Timeout waiting for the system to boot.\n' to JSON. Cause: Fail to parse reader data to JSON, error: invalid character 'F' looking for beginning of value]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/src/github.com/wildfly/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:237\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}
After increasing the number of tries from 10 to 20, the pod is scaled down, but it doesn't work always:
{"level":"info","ts":1673962200.2645516,"logger":"wildflyserver_controller","msg":"Restarting application server to apply the env properties","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962202.2448435,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 1/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962203.7255769,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 2/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962205.0437005,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 3/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962206.3449464,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 4/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962208.2623868,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 5/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962210.413443,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 6/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962212.7975974,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 7/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962214.1961286,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 8/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962215.654061,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 9/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962223.564317,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 10/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.3337781,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 11/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.3508961,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 12/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.3673959,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 13/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.3876672,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 14/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.4055603,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 15/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.4260724,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 16/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.4430568,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 17/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.4599392,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 18/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.475704,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 19/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.4916313,"logger":"wildflyserver_controller","msg":"Waiting for server to be reinitialized. Iteration 20/20","Request.Namespace":"eap-operator-demo","Request.Name":"kitchensink-operator-app","Pod Name":"kitchensink-operator-app-1"} \{"level":"info","ts":1673962255.5080538,"logger":"wildflyserver_resources","msg":"Deleting Resource","WildFlyServer.Namespace":"eap-operator-demo","WildFlyServer.Name":"kitchensink-operator-app","Resource.Name":"kitchensink-operator-app-1","Resource.Type":"*v1.Pod"} {"level":"error","ts":1673962255.5162292,"logger":"controller","msg":"Reconciler error","controller":"wildflyserver-controller","name":"kitchensink-operator-app","namespace":"eap-operator-demo","error":"Found 1 errors:\n [[Cannot restart application server at pod kitchensink-operator-app-1 after setting up the periodic recovery properties. The pod was deleted to be restarted for new recovery attempt. Error: Application server was not reinitialized successfully in time. Operation ':shutdown(restart=true)' at pod kitchensink-operator-app-1, JSON management operation result: map[outcome:success result:<nil>], error: Cannot execute JBoss CLI command ${JBOSS_HOME}/bin/jboss-cli.sh --output-json -c --commands=':read-attribute(name=server-state)' at pod kitchensink-operator-app-1. Cause: error on execution '${JBOSS_HOME}/bin/jboss-cli.sh --output-json -c --commands=':read-attribute(name=server-state)'': unable to upgrade connection: container not found (\"kitchensink-operator-app\")]],","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/src/github.com/wildfly/wildfly-operator/vendor/github.com/go-logr/zapr/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:237\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/src/github.com/wildfly/wildfly-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:188\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/src/github.com/wildfly/wildfly-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90"}
The number of tries can be increased by adding the following SERVER_RESTART_RETRY_COUNTER variable on the Operator subscription:
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: labels: operators.coreos.com/eap.eap-operator-demo: "" name: eap namespace: eap-operator-demo spec: channel: stable config: env: - name: SERVER_RESTART_RETRY_COUNTER value: "20" installPlanApproval: Automatic name: eap source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: eap-operator.v2.3.8
- relates to
-
JBEAP-24814 Can't scale down from 3 to 1 pods when Tx recovery is ongoing
- New