Major +underlined text+In a log of baclend-cron pod, the following error happens at startup.
[pod/backend-cron-2-cxzcd/backend-cron] rake aborted! [pod/backend-cron-2-cxzcd/backend-cron] NameError: uninitialized constant #<Class:ThreeScale::Backend::Stats::Cleaner>::StatsParser [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:228:in `rescue in service_from_stats_key' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:226:in `service_from_stats_key' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:212:in `delete_key?' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:181:in `block (3 levels) in delete_keys' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:181:in `select' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:181:in `block (2 levels) in delete_keys' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:265:in `with_retries' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:178:in `block in delete_keys' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:177:in `loop' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:177:in `delete_keys' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:88:in `block in delete!' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:86:in `each' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:86:in `partition' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:86:in `delete!' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/Rakefile:270:in `block (2 levels) in <top (required)>' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:279:in `block in execute' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:279:in `each' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:279:in `execute' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `each' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:80:in `run' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/bin/rake:23:in `load' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/bin/rake:23:in `<main>' [pod/backend-cron-2-cxzcd/backend-cron] [pod/backend-cron-2-cxzcd/backend-cron] Caused by: [pod/backend-cron-2-cxzcd/backend-cron] NameError: uninitialized constant #<Class:ThreeScale::Backend::Stats::Cleaner>::StatsParser [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:227:in `service_from_stats_key' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:212:in `delete_key?' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:181:in `block (3 levels) in delete_keys' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:181:in `select' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:181:in `block (2 levels) in delete_keys' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:265:in `with_retries' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:178:in `block in delete_keys' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:177:in `loop' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:177:in `delete_keys' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:88:in `block in delete!' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:86:in `each' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:86:in `partition' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/lib/3scale/backend/stats/cleaner.rb:86:in `delete!' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/Rakefile:270:in `block (2 levels) in <top (required)>' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:279:in `block in execute' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:279:in `each' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:279:in `execute' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `each' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/lib/rake/application.rb:80:in `run' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/bin/rake:23:in `load' [pod/backend-cron-2-cxzcd/backend-cron] /opt/ruby/apisonator-3.4.3/vendor/bundle/ruby/2.7.0/bin/rake:23:in `<main>' [pod/backend-cron-2-cxzcd/backend-cron] Tasks: TOP => stats:cleanup
This error also happens repeatedly.
I run stats:cleanup job within backend-cron pod manually like the following and saw the same error.
sh-4.4$ bundle exec rake stats:cleanup["redis://backend-redis:6379/0",true]
