Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-12161

"Internal error" happens in Admin Portal while accessing product, backend , account etc after upgrading to 2.16.1

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 2.16.1 GA
    • 3scale Operator
    • False
    • Hide

      None

      Show
      None
    • False
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Workaround Exists
    • Hide

      1. delete system-app-pre job

        $ oc delete job system-app-pre

      2. restart system-app

        $ oc rollout restart deploy/system-app

      Show
      1. delete system-app-pre job   $ oc delete job system-app-pre 2. restart system-app   $ oc rollout restart deploy/system-app
    • Important

      After upgrading 3scale from 2.15.x to 2.16.1, "Internal error" happens in Admin Portal while accessing product, backend , account etc.

      In system-provider log, the following error happens:

       Started GET "/p/admin/backend_apis/2" for xxx.xxx.xxx.xxx at 2025-12-26 03:35:02 +0000
       Processing by Provider::Admin::BackendApisController#show as HTML
         Parameters: {"id"=>"2"}
         Rendered provider/admin/backend_apis/show.html.slim within layouts/provider (Duration: 10.8ms | Allocations: 11190)
         Rendered layout layouts/provider.html.slim (Duration: 10.9ms | Allocations: 11240)
       Completed 500 Internal Server Error in 47ms (ActiveRecord: 25.9ms | Allocations: 21382)
      
       ActionView::Template::Error (Mysql2::Error: Table 'system.annotations' doesn't exist):
           1: - if resource.respond_to?(:managed_by) && resource.managed_by.present?
           2:   - var = defined?(variant) ? variant : :info
           3:   - if defined?(plain)
           4:     = pf_inline_alert t('.managed_title', value: resource.managed_by), variant: var, plain: true    
      
       app/models/concerns/annotating/model.rb:34:in `annotation'
       app/models/concerns/annotating/model.rb:38:in `value_of_annotation'
       app/models/concerns/annotating/managed_by.rb:8:in `managed_by'
       app/views/shared/_annotations.html.slim:1
       app/views/provider/admin/backend_apis/show.html.slim:3
       app/views/provider/admin/backend_apis/show.html.slim:2
       lib/three_scale/middleware/multitenant.rb:106:in `_call'
       lib/three_scale/middleware/multitenant.rb:101:in `call'
       lib/three_scale/middleware/presigned_downloads.rb:62:in `call'
       lib/three_scale/middleware/cors.rb:21:in `call'
      

      In addition, when apicast-production gets restarted, it fails to start and gets into CrashLoopBackoff due to failure of configuration loading. In system-master log, the following error happens:

       Started GET "/master/api/proxy/configs/production.json" for xxx.xxx.xxx.xxx at 2025-12-26 03:31:38 +0000
       Processing by Master::Api::Proxy::ConfigsController#index as JSON
         Parameters: {"environment"=>"production"}
       Completed 500 Internal Server Error in 37ms (ActiveRecord: 13.4ms | Allocations: 15231)
      
       NameError (undefined local variable or method `expires_at' for #<AccessToken id: 1, owner_id: 1, scopes: ["account_management"], value: "gs55cXIo", name: "APIcast", permission: "ro", tenant_id: nil, created_at: "2025-12-26 02:30:13.000000000 +0000", updated_at: "2025-12-26 02:30:13.000000000 +0000">
      Did you mean?  expires_at=):
      
       app/models/access_token.rb:187:in `expired?'
       app/lib/api_authentication/by_access_token.rb:114:in `authenticated_token'
       app/lib/api_authentication/by_access_token.rb:8:in `current_user'
       app/lib/api_authentication/by_provider_key.rb:15:in `current_account'
       app/controllers/application_controller.rb:124:in `set_timezone'
       lib/three_scale/middleware/multitenant.rb:106:in `_call'
       lib/three_scale/middleware/multitenant.rb:101:in `call'
       lib/three_scale/middleware/presigned_downloads.rb:62:in `call'
       lib/three_scale/middleware/cors.rb:21:in `call'
      

      Here is a summary of upgrade steps:
      1. Take backup of 3scale 2.15
      2. Externalize system-mysql/system-redis/backend redis by following the document:

      3. Upgrade 3scale to 2.16 by following the document:

              Unassigned Unassigned
              rhn-support-hmiura Hiroko Miura
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: