Details
-
Bug
-
Resolution: Done
-
Major
-
2.11.2 GA
-
False
-
None
-
False
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
Description
Issue:
- When a tenant is created declaratively, using the TenantCR, and afterwards it’s deleted from the 3scale admin portal UI, the operator starts creating tenants in an infinite loop.
- After the tenant is deleted (15 days) or via the rails console the operator pod logs shows the following:
{{{"level":"info","ts":1647368561.9186196,"logger":"controllers.Tenant","msg":"Creating a new tenant","tenant":"3scale211/xcorp-tenant","OrganizationName":"Xcorp","Username":"admin","Email":"admin@xcorp.com"}
\{"level":"error","ts":1647368571.236814,"logger":"controllers.Tenant","msg":"Error in tenant reconciliation","tenant":"3scale211/xcorp-tenant","error":"error calling 3scale system - reason: {\"status\":\"Not found\"} - code: 404","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/deps/gomod/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/3scale/3scale-operator/controllers/capabilities.(*TenantReconciler).Reconcile\n\t/remote-source/app/controllers/capabilities/tenant_controller.go:104\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"}
\{"level":"error","ts":1647368571.2372437,"logger":"controller","msg":"Reconciler error","reconcilerGroup":"capabilities.3scale.net","reconcilerKind":"Tenant","controller":"tenant","name":"xcorp-tenant","namespace":"3scale211","error":"error calling 3scale system - reason: {\"status\":\"Not found\"} - code: 404","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/remote-source/deps/gomod/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:246\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/remote-source/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/remote-source/deps/gomod/pkg/mod/k8s.io/apimachinery@v0.18.6/pkg/util/wait/wait.go:90"}
\{"level":"info","ts":1647368571.551711,"logger":"controllers.WebConsole","msg":"Reconciling ReconcileWebConsole","webconsole":"3scale211/zync-3scale-api-4mvm9","Operator version":"0.8.1"}}}
- The tenants are created with the admin and dev portal urls of the TenantCR, with the form:
xcorp-X-admin.apps.avila3scale.lab.rdu2.cee.redhat.com xcorp-X.apps.avila3scale.lab.rdu2.cee.redhat.com
where X is a number that increments for each tenant created.
Workaround
- To stop the loop the TenantCR should be patched with the ID of one of the created tenants using the command below:
apiserver=https://kubernetes.default.svc serviceaccount=/var/run/secrets/kubernetes.io/serviceaccount namespace=$(cat ${serviceaccount}/namespace) token=$(cat ${serviceaccount}/token) tenant_name=<TENANT-NAME> tenant_status='{"tenantId": <tenantID>, "adminId": <adminID>}'
curl -k -s -X PATCH -H "Accept: application/json" \ -H "Content-Type: application/merge-patch+json" \ "${apiserver}/apis/capabilities.3scale.net/v1alpha1/namespaces/${namespace}/tenants/${tenant_name}/status" \ --data "{\"status\":${tenant_status}}" \ --header "Authorization: Bearer ${token}" \ --fail -v
The tenantID is the ID of one of the tenants created, i.e. the first one that doesn't have the appended number in the admin and dev portal (not the value of the tenantId from the CR). The adminId is the ID of the admin of the Tenant.
Desired behaviour
It should not be possible to delete the operator owned resource from the UI and the operator should not create tenants if for some reason the cluster ends up in a similar state (CR still exists but the tenant object is deleted from the 3scale DB).
Attachments
Issue Links
- relates to
-
THREESCALE-8296 Display a warning or prevent actions being executed in the admin portal UI when the CRs are used
-
- Closed
-
-
-
- Closed
-
-
THREESCALE-6051 Support deletion of Tenant using Operator CRD
-
- Closed
-
- links to