-
Bug
-
Resolution: Obsolete
-
Major
-
4.8.1
-
None
-
None
-
Incidents & Support
-
False
-
-
False
-
-
-
-
Rox Sprint 4.9G - Global, Rox Sprint 4.10B, Rox Sprint 4.10C, Rox Sprint 4.10D, Rox Sprint 4.10E
-
Important
While taking a look at issues in ACSCS around RHIT memory consumption I noticed an issue upon startup when injecting policies and finding the corresponding alerts. The query was timing out.
Is the query slow? Should we be using `WalkByQuery` instead of `GetByQueryFn`? We should investigate performance of those and see if we can solve this startup issue.
root logger: 2025/08/27 12:15:14.761633 logging.go:280: Error: Unexpected Error: timeout: context deadline exceededprocessing rowsgithub.com/stackrox/rox/pkg/search/postgres.RunQueryForSchemaFn[...] github.com/stackrox/rox/pkg/search/postgres/common.go:1059github.com/stackrox/rox/pkg/search/postgres.(*genericStore[...]).GetByQueryFn github.com/stackrox/rox/pkg/search/postgres/store.go:230github.com/stackrox/rox/central/alert/datastore/internal/search.(*searcherImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/internal/search/searcher_impl.go:66github.com/stackrox/rox/central/alert/datastore.(*datastoreImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/datastore_impl.go:75github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).mergeManyAlerts github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:321github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).AlertAndNotify github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:51github.com/stackrox/rox/central/detection/lifecycle.(*managerImpl).UpsertPolicy github.com/stackrox/rox/central/detection/lifecycle/manager_impl.go:432github.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:45sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700failed to search alertsgithub.com/stackrox/rox/central/alert/datastore/internal/search.(*searcherImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/internal/search/searcher_impl.go:71github.com/stackrox/rox/central/alert/datastore.(*datastoreImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/datastore_impl.go:75github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).mergeManyAlerts github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:321github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).AlertAndNotify github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:51github.com/stackrox/rox/central/detection/lifecycle.(*managerImpl).UpsertPolicy github.com/stackrox/rox/central/detection/lifecycle/manager_impl.go:432github.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:45sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700couldn't load previous alerts (query was Policy ID:"2361bb4c-4cf6-4997-bae6-825da6cf932e"+Violation State:"ACTIVE","ATTEMPTED")github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).mergeManyAlerts github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:323github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).AlertAndNotify github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:51github.com/stackrox/rox/central/detection/lifecycle.(*managerImpl).UpsertPolicy github.com/stackrox/rox/central/detection/lifecycle/manager_impl.go:432github.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:45sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700could not inject policygithub.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:46sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700root logger: 2025/08/27 12:16:14.876591 logging.go:280: Error: Unexpected Error: timeout: context deadline exceededprocessing rowsgithub.com/stackrox/rox/pkg/search/postgres.RunQueryForSchemaFn[...] github.com/stackrox/rox/pkg/search/postgres/common.go:1059github.com/stackrox/rox/pkg/search/postgres.(*genericStore[...]).GetByQueryFn github.com/stackrox/rox/pkg/search/postgres/store.go:230github.com/stackrox/rox/central/alert/datastore/internal/search.(*searcherImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/internal/search/searcher_impl.go:66github.com/stackrox/rox/central/alert/datastore.(*datastoreImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/datastore_impl.go:75github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).mergeManyAlerts github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:321github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).AlertAndNotify github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:51github.com/stackrox/rox/central/detection/lifecycle.(*managerImpl).UpsertPolicy github.com/stackrox/rox/central/detection/lifecycle/manager_impl.go:432github.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:45sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700failed to search alertsgithub.com/stackrox/rox/central/alert/datastore/internal/search.(*searcherImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/internal/search/searcher_impl.go:71github.com/stackrox/rox/central/alert/datastore.(*datastoreImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/datastore_impl.go:75github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).mergeManyAlerts github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:321github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).AlertAndNotify github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:51github.com/stackrox/rox/central/detection/lifecycle.(*managerImpl).UpsertPolicy github.com/stackrox/rox/central/detection/lifecycle/manager_impl.go:432github.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:45sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700couldn't load previous alerts (query was Policy ID:"f2183906-4577-47de-9bf4-270d09e0a93c"+Violation State:"ACTIVE","ATTEMPTED")github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).mergeManyAlerts github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:323github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).AlertAndNotify github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:51github.com/stackrox/rox/central/detection/lifecycle.(*managerImpl).UpsertPolicy github.com/stackrox/rox/central/detection/lifecycle/manager_impl.go:432github.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:45sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700could not inject policygithub.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:46sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700root logger: 2025/08/27 12:17:15.165037 logging.go:280: Error: Unexpected Error: timeout: context deadline exceededprocessing rowsgithub.com/stackrox/rox/pkg/search/postgres.RunQueryForSchemaFn[...] github.com/stackrox/rox/pkg/search/postgres/common.go:1059github.com/stackrox/rox/pkg/search/postgres.(*genericStore[...]).GetByQueryFn github.com/stackrox/rox/pkg/search/postgres/store.go:230github.com/stackrox/rox/central/alert/datastore/internal/search.(*searcherImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/internal/search/searcher_impl.go:66github.com/stackrox/rox/central/alert/datastore.(*datastoreImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/datastore_impl.go:75github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).mergeManyAlerts github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:321github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).AlertAndNotify github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:51github.com/stackrox/rox/central/detection/lifecycle.(*managerImpl).UpsertPolicy github.com/stackrox/rox/central/detection/lifecycle/manager_impl.go:432github.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:45sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700failed to search alertsgithub.com/stackrox/rox/central/alert/datastore/internal/search.(*searcherImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/internal/search/searcher_impl.go:71github.com/stackrox/rox/central/alert/datastore.(*datastoreImpl).SearchRawAlerts github.com/stackrox/rox/central/alert/datastore/datastore_impl.go:75github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).mergeManyAlerts github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:321github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).AlertAndNotify github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:51github.com/stackrox/rox/central/detection/lifecycle.(*managerImpl).UpsertPolicy github.com/stackrox/rox/central/detection/lifecycle/manager_impl.go:432github.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:45sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700couldn't load previous alerts (query was Policy ID:"ddb7af9c-5ec1-45e1-a0cf-c36e3ef2b2ce"+Violation State:"ACTIVE","ATTEMPTED")github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).mergeManyAlerts github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:323github.com/stackrox/rox/central/detection/alertmanager.(*alertManagerImpl).AlertAndNotify github.com/stackrox/rox/central/detection/alertmanager/alert_manager_impl.go:51github.com/stackrox/rox/central/detection/lifecycle.(*managerImpl).UpsertPolicy github.com/stackrox/rox/central/detection/lifecycle/manager_impl.go:432github.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:45sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700could not inject policygithub.com/stackrox/rox/central/detection/lifecycle.initialize github.com/stackrox/rox/central/detection/lifecycle/singleton.go:46sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/detection/lifecycle.SingletonManager github.com/stackrox/rox/central/detection/lifecycle/singleton.go:55github.com/stackrox/rox/central/policy/service.initialize github.com/stackrox/rox/central/policy/service/singleton.go:32sync.(*Once).doSlow sync/once.go:76sync.(*Once).Do sync/once.go:67github.com/stackrox/rox/central/policy/service.Singleton github.com/stackrox/rox/central/policy/service/singleton.go:40main.servicesToRegister github.com/stackrox/rox/central/main.go:439main.startGRPCServer github.com/stackrox/rox/central/main.go:625runtime.goexit runtime/asm_amd64.s:1700detection/lifecycle: 2025/08/27 12:17:19.019244 singleton.go:48: Info: Done injecting policies.
- relates to
-
-
- Closed
-
