Details
Minor Description
Backend "pjqmzden" is marked as "in use":
Checking backend details, there is no product using it:
When I try to delete, 403 Forbidden is returned with body saying: "cannot be deleted because it is used by at least one Product"
opening connection to 3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net:443... opened starting SSL for 3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net:443... SSL established, protocol: TLSv1.2, cipher: ECDHE-RSA-AES128-GCM-SHA256 <- "DELETE /admin/api/backend_apis/3323.json HTTP/1.1\r\nAccept: application/json\r\nContent-Type: application/json\r\nAuthorization: Basic OkNnR0k3RjNRdTNVTTd0RXM=\r\nAccept-Encoding: identity\r\nUser-Agent: Ruby\r\nConnection: close\r\nHost: 3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net\r\n\r\n" -> "HTTP/1.1 403 Forbidden\r\n" -> "Date: Wed, 01 Jul 2020 12:29:42 GMT\r\n" -> "Connection: close\r\n" -> "Content-Type: application/json; charset=utf-8\r\n" -> "Cache-Control: no-cache\r\n" -> "Set-Cookie: _system_session=Y1hvVXEzbHgxQmNPdmVVdEx5cnAxWXV5aXNmc3BVTmlmRVNjME9TeTJMaFRvVmdlRk1LWitBQUh1VTRPQUhRWk10NW8yRXZWWGJhczFOU3ZvYnhCRlE9PS0tMEpxYXpranlYU1JMclRiWi9zem84QT09--ad40e6b0cde06e2c7bf874781289c8059a55cb90; path=/; secure; HttpOnly; SameSite=Lax\r\n" -> "X-Request-Id: 72479e59-9aca-45e1-a827-ef6529e0baaf\r\n" -> "X-Runtime: 0.033578\r\n" -> "X-Served-By: system-app-3-dlx2z\r\n" -> "Vary: Accept-Encoding\r\n" -> "Strict-Transport-Security: max-age=15552000\r\n" -> "X-Frame-Options: DENY\r\n" -> "X-Content-Type-Options: nosniff\r\n" -> "X-XSS-Protection: 1; mode=block\r\n" -> "Transfer-Encoding: chunked\r\n" -> "Set-Cookie: 1a2f5865e23a34c8fa2970884bc06797=22829522e1a12d7e189d1552d6df2efb; path=/; HttpOnly; Secure\r\n" -> "\r\n" -> "54\r\n" reading 84 bytes... -> "{\"errors\":{\"base\":[\"cannot be deleted because it is used by at least one Product\"]}}" read 84 bytes reading 2 bytes... -> "\r\n" read 2 bytes -> "0\r\n" -> "\r\n" Conn close
Reading the backendapi:
<- "GET /admin/api/backend_apis/3323.json HTTP/1.1\r\nAccept: application/json\r\nContent-Type: application/json\r\nAuthorization: Basic OkNnR0k3RjNRdTNVTTd0RXM=\r\nAccept-Encoding: identity\r\nUser-Agent: Ruby\r\nConnection: close\r\nHost: 3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net\r\n\r\n" -> "HTTP/1.1 200 OK\r\n" -> "Date: Wed, 01 Jul 2020 13:40:51 GMT\r\n" -> "Connection: close\r\n" -> "ETag: W/\"b6208c39533b60a1552a1e06e96f0e6d\"\r\n" -> "Content-Type: application/json; charset=utf-8\r\n" -> "Cache-Control: max-age=0, private, must-revalidate\r\n" -> "X-Request-Id: 1d85b074-2154-499a-96f3-f7d3a689c860\r\n" -> "X-Runtime: 0.027064\r\n" -> "X-Served-By: system-app-3-dlx2z\r\n" -> "Vary: Accept-Encoding\r\n" -> "Strict-Transport-Security: max-age=15552000\r\n" -> "X-Frame-Options: DENY\r\n" -> "X-Content-Type-Options: nosniff\r\n" -> "X-XSS-Protection: 1; mode=block\r\n" -> "Transfer-Encoding: chunked\r\n" -> "Set-Cookie: 1a2f5865e23a34c8fa2970884bc06797=22829522e1a12d7e189d1552d6df2efb; path=/; HttpOnly; Secure\r\n" -> "\r\n" -> "175\r\n" reading 373 bytes... -> "{\"backend_api\":{\"id\":3323,\"name\":\"pjqmzden\",\"system_name\":\"pjqmzden\",\"private_endpoint\":\"http://echo-api-benchmarks.dev.3sca.net:8081\",\"account_id\":2,\"created_at\":\"2020-07-01T10:27:38Z\",\"updated_at\":\"2020-07-01T10:28:09Z\",\"links\":[{\"rel\":\"metrics\",\"href\":\"/admin/api/backend_apis/3323/metrics\"},{\"rel\":\"mapping_rules\",\"href\":\"/admin/api/backend_apis/3323/mapping_rules\"}]}}" read 373 bytes reading 2 bytes... -> "\r\n" read 2 bytes -> "0\r\n" -> "\r\n" Conn close
Do not worry about credentials shown in the logs. This is performance test environment and is will be deleted soon.
It is hard to reproduce, as it does not always happen. Basically I run script to remove first all the services (DELETE /admin/api/services/{id}.json). Then, delete all backends (DELETE /admin/api/backend_apis/{id}.json)
Since service removal happens in the background, some operation failed and the backendapi was left "orphan", when service using it was already removed.
Logs from system-provider container:
e862b87b-4bc3-471e-96f7-1e5488968a45] [3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net] [185.179.187.23] Started DELETE "/admin/api/backend_apis/3323.json" for 185.179.187.23 at 2020-07-01 10:57:03 +0000 [e862b87b-4bc3-471e-96f7-1e5488968a45] [3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net] [185.179.187.23] Processing by Admin::Api::BackendApisController#destroy as JSON [e862b87b-4bc3-471e-96f7-1e5488968a45] [3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net] [185.179.187.23] Parameters: {"id"=>"3323", "backend_api"=>{}} [e862b87b-4bc3-471e-96f7-1e5488968a45] [3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net] [185.179.187.23] Can't verify CSRF token authenticity. [e862b87b-4bc3-471e-96f7-1e5488968a45] [3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net] [185.179.187.23] Destroying user session [e862b87b-4bc3-471e-96f7-1e5488968a45] [3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net] [185.179.187.23] PermissionEnforcer: level = rw [e862b87b-4bc3-471e-96f7-1e5488968a45] [3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net] [185.179.187.23] PermissionEnforcer#ensure clear level [e862b87b-4bc3-471e-96f7-1e5488968a45] [3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net] [185.179.187.23] PermissionEnforcer: level = [e862b87b-4bc3-471e-96f7-1e5488968a45] [3scale-admin.3scale.apps.dev-eng-ocp4-3-benchmarks.dev.3sca.net] [185.179.187.23] Completed 403 Forbidden in 29ms (Views: 0.2ms | ActiveRecord: 4.4ms) 185.179.187.23 - - [01/Jul/2020:10:57:03 +0000] "DELETE /admin/api/backend_apis/3323.json HTTP/1.1" 403 - 0.0330
