While running the Warp workload (with concurrent 1000) on a BM),  the notification timeout is observed in the noobaa log.

./warp mixed --insecure --duration 60m --host ces-ip:6443 --access-key yyyyyyyyyyy --secret-key xxxxxxxxxxx --tls --obj.size 128M --disable-multipart --bucket newbucket-17apr-warp --concurrent 1000

config list

NOTIFICATION_FREQUENCY_IN_MINUTES : 2
 NOTIFICATION_LOG_DIR : /ibm/fvt_fs/notification-dir-17apr

ENDPOINT_FORKS : 2

UVTHREADPOOLSIZE : 16

noobaa.log (snippet)

Apr 21 00:06:10 node-gui0 node[3118357]: [noobaa-cli/3118357] [ERROR] core.util.notifications_util:: Notify timeout
Apr 21 00:06:10 node-gui0 node[3118357]: [noobaa-cli/3118357] [ERROR] core.util.notifications_util:: Notify timeout
Apr 21 00:06:10 node-gui0 node[3118357]: [noobaa-cli/3118357] [ERROR] core.util.notifications_util:: Notify timeout
Apr 21 00:06:10 node-gui0 node[3118357]: [noobaa-cli/3118357] [ERROR] core.util.notifications_util:: Notify timeout
Apr 21 00:06:10 node-gui0 node[3118357]: [noobaa-cli/3118357] [ERROR] core.util.notifications_util:: Notify timeout
Apr 21 00:06:10 node-gui0 node[3118357]: [noobaa-cli/3118357] [ERROR] core.util.notifications_util:: Notify timeout
Apr 21 00:06:10 node-gui0 node[3118357]: [noobaa-cli/3118357] [ERROR] core.util.notifications_util:: Notify timeout
Apr 21 00:06:10 node-gui0 node[3118357]: [noobaa-cli/3118357] [ERROR] core.util.notifications_util:: Notify timeout

cron log (when NOTIFICATION frequency ran via cron job for every  2 mins)

------------------------------------------------------------------------------
Apr 21 00:06:02 node-gui0 CROND[3118347]: (root) CMDOUT ((Use `node --trace-deprecation ...` to show where the warning was created))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT ((node:3118357) UnhandledPromiseRejectionWarning: TypeError: Cannot read properties of undefined (reading 'toString'))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Notificator.handle_failed_notification (/usr/local/noobaa-core/src/util/notifications_util.js:164:103))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at ClientRequest.<anonymous> (/usr/local/noobaa-core/src/util/notifications_util.js:242:17))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at ClientRequest.emit (node:events:518:28))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at ClientRequest.emit (node:domain:489:12))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Socket.emitRequestTimeout (node:_http_client:851:9))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Object.onceWrapper (node:events:632:28))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Socket.emit (node:events:530:35))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Socket.emit (node:domain:489:12))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Socket._onTimeout (node:net:595:8))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at listOnTimeout (node:internal/timers:594:17))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT ((node:3118357) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 5))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT ((node:3118357) UnhandledPromiseRejectionWarning: TypeError: Cannot read properties of undefined (reading 'toString'))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Notificator.handle_failed_notification (/usr/local/noobaa-core/src/util/notifications_util.js:164:103))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at ClientRequest.<anonymous> (/usr/local/noobaa-core/src/util/notifications_util.js:242:17))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at ClientRequest.emit (node:events:518:28))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at ClientRequest.emit (node:domain:489:12))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Socket.emitRequestTimeout (node:_http_client:851:9))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Object.onceWrapper (node:events:632:28))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Socket.emit (node:events:530:35))
Apr 21 00:06:10 node-gui0 CROND[3118347]: (root) CMDOUT (    at Socket.emit (node:domain:489:12))

Are the "Notify timeout" expected and the cron log messages for the stack has shown "RejectionWarning" . Could you confirm whether these are appropriate or not.

Note: Due to workload heavy (the IO timeout ) occurred for the warp and then killed itself.