-
Bug
-
Resolution: Unresolved
-
Major
-
None
-
2.16.1 GA
-
False
-
-
False
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
Not Started
-
Workaround Exists
-
-
-
-
Important
After upgrading 3scale from 2.15.x to 2.16.1, "Internal error" happens in Admin Portal while accessing product, backend , account etc.
In system-provider log, the following error happens:
Started GET "/p/admin/backend_apis/2" for xxx.xxx.xxx.xxx at 2025-12-26 03:35:02 +0000
Processing by Provider::Admin::BackendApisController#show as HTML
Parameters: {"id"=>"2"}
Rendered provider/admin/backend_apis/show.html.slim within layouts/provider (Duration: 10.8ms | Allocations: 11190)
Rendered layout layouts/provider.html.slim (Duration: 10.9ms | Allocations: 11240)
Completed 500 Internal Server Error in 47ms (ActiveRecord: 25.9ms | Allocations: 21382)
ActionView::Template::Error (Mysql2::Error: Table 'system.annotations' doesn't exist):
1: - if resource.respond_to?(:managed_by) && resource.managed_by.present?
2: - var = defined?(variant) ? variant : :info
3: - if defined?(plain)
4: = pf_inline_alert t('.managed_title', value: resource.managed_by), variant: var, plain: true
app/models/concerns/annotating/model.rb:34:in `annotation'
app/models/concerns/annotating/model.rb:38:in `value_of_annotation'
app/models/concerns/annotating/managed_by.rb:8:in `managed_by'
app/views/shared/_annotations.html.slim:1
app/views/provider/admin/backend_apis/show.html.slim:3
app/views/provider/admin/backend_apis/show.html.slim:2
lib/three_scale/middleware/multitenant.rb:106:in `_call'
lib/three_scale/middleware/multitenant.rb:101:in `call'
lib/three_scale/middleware/presigned_downloads.rb:62:in `call'
lib/three_scale/middleware/cors.rb:21:in `call'
In addition, when apicast-production gets restarted, it fails to start and gets into CrashLoopBackoff due to failure of configuration loading. In system-master log, the following error happens:
Started GET "/master/api/proxy/configs/production.json" for xxx.xxx.xxx.xxx at 2025-12-26 03:31:38 +0000
Processing by Master::Api::Proxy::ConfigsController#index as JSON
Parameters: {"environment"=>"production"}
Completed 500 Internal Server Error in 37ms (ActiveRecord: 13.4ms | Allocations: 15231)
NameError (undefined local variable or method `expires_at' for #<AccessToken id: 1, owner_id: 1, scopes: ["account_management"], value: "gs55cXIo", name: "APIcast", permission: "ro", tenant_id: nil, created_at: "2025-12-26 02:30:13.000000000 +0000", updated_at: "2025-12-26 02:30:13.000000000 +0000">
Did you mean? expires_at=):
app/models/access_token.rb:187:in `expired?'
app/lib/api_authentication/by_access_token.rb:114:in `authenticated_token'
app/lib/api_authentication/by_access_token.rb:8:in `current_user'
app/lib/api_authentication/by_provider_key.rb:15:in `current_account'
app/controllers/application_controller.rb:124:in `set_timezone'
lib/three_scale/middleware/multitenant.rb:106:in `_call'
lib/three_scale/middleware/multitenant.rb:101:in `call'
lib/three_scale/middleware/presigned_downloads.rb:62:in `call'
lib/three_scale/middleware/cors.rb:21:in `call'
Here is a summary of upgrade steps:
1. Take backup of 3scale 2.15
2. Externalize system-mysql/system-redis/backend redis by following the document:
3. Upgrade 3scale to 2.16 by following the document: