While running Warp as part of verification of the https://issues.redhat.com/browse/DFBUGS-2292
on the BM cluster , the following noobaa endpoint restarts are observed
Warp:
./warp mixed --insecure --duration 30m --host gpfs-p10-s3-ces.rtp.raleigh.ibm.com:6443 --access-key uOmitMvnLt2SWVPwVWn1 --secret-key Fx37u//7SFYLdxt3SEQjZ/u0hYuXcfN+HemWWx26 --tls --obj.size 64M --disable-multipart --bucket warp-def-2292 --concurrent 1000 2>&1| tee /tmp/warp-def-2292.26may2026.log; date
Mon May 26 06:57:43 AM EDT 2025
Benchmarking: 30m00s / 30m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 100.00% 30m3s
warp: Benchmark data written to "warp-mixed-2025-05-26[070944]-urxh.csv.zst" Mixed operations.
Operation: DELETE, 0%, Concurrency: 0, Ran 0s.
Skipping DELETE too few samples. Longer benchmark run required for reliable results.
Operation: GET, 0%, Concurrency: 0, Ran 0s.
Skipping GET too few samples. Longer benchmark run required for reliable results.
Operation: PUT, 0%, Concurrency: 0, Ran 0s.
Skipping PUT too few samples. Longer benchmark run required for reliable results.
Operation: STAT, 0%, Concurrency: 0, Ran 0s.
Skipping STAT too few samples. Longer benchmark run required for reliable results.
Cluster Total: NaN obj/s over 0s.
warp: Cleanup Done.Mon May 26 11:11:20 AM EDT 2025
noobaa d/s noobaa-core-5.18.4-20250521
noobaa process when started on the node-gui1
ps -ef |grep noobaa
root 1809760 1 0 05:37 ? 00:00:20 noobaa
root 1810648 1809760 1 05:37 ? 00:07:24 noobaa
root 1810651 1809760 1 05:37 ? 00:06:07 noobaa
root 1810652 1809760 1 05:37 ? 00:07:01 noobaa
root 1810661 1809760 1 05:37 ? 00:06:54 noobaa
root 1913556 1809760 1 07:02 ? 00:04:27 noobaa --> this one and the 07:07 has restarted
root 1918485 1809760 0 07:07 ? 00:02:52 noobaa
noobaa.log
May 26 05:37:16 node-gui1 node[1809760]: [nsfs/1809760] [L0] core.endpoint.endpoint:: endpoint: replacing old umask: 22 with new umask: 0
May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760] [WARN] CONSOLE:: WORKER started { id: 1, pid: 1810648 }
May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760] [WARN] CONSOLE:: WORKER started { id: 2, pid: 1810651 }
May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760] [WARN] CONSOLE:: WORKER started { id: 3, pid: 1810652 }
May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760] [WARN] CONSOLE:: WORKER started { id: 4, pid: 1810659 }
May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760] [WARN] CONSOLE:: WORKER started { id: 5, pid: 1810661 }
May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760] [WARN] CONSOLE:: WORKER started { id: 6, pid: 1810673 }
...
May 26 07:02:11 node-gui1 [1810648]: 2025-05-26 07:02:11.649468 [PID-1810648/TID-1810775] [L0] FS::FSWorker::Execute: WARNING Fsync _path=/ibm/fvt_fs/new-
s3user25k2-dir/warp-def-2292/ou0PFCL4 took too long: 39095.4 ms
May 26 07:02:11 node-gui1 [1810659]: 2025-05-26 07:02:11.650888 [PID-1810659/TID-1810807] [L0] FS::FSWorker::Execute: WARNING FileOpen _path=/ibm/fvt_fs/n
ew-s3user25k2-dir/warp-def-2292/9FFL(Sni _flags=4210690 _mode=432 took too long: 437.14 ms
May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652] [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 elWHat4Y/1.9c0bePd0OQnsdlHE.rnd
May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652] [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 )XvxQPYK/1.yrwb0C5gwgOnX26E.rnd
May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652] [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 N3vglakP/1.f4HzyJcnutUe8pC2.rnd
May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652] [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 ctLMXuwe/1.AB7KRZ3Xt3Op(mUW.rnd
May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652] [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 8k(G0hvn/1.k7uh9VfGGfq4I65T.rnd
May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652] [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 36)thnRl/2.O(XqYnZYB09WDGxj.rnd
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [ERROR] CONSOLE:: PANIC: process uncaughtException Error: No such file or directory
May 26 07:02:11 node-gui1 [1810659]: 2025-05-26 07:02:11.676230 [PID-1810659/TID-1810809] [L0] FS::FSWorker::Execute: WARNING FileWritev _wrap->_path=/ibm/fvt_fs/new-s3user25k2-dir/warp-def-2292/ZRwlZfZF _total_len=8372224 buffers_len=511 _offset=-1 took too long: 159.112 ms
May 26 07:07:29 node-gui1 [1810673]: [nsfs/1810673] [ERROR] CONSOLE:: PANIC: process uncaughtException Error: FileWrite _wrap->_path=/ibm/fvt_fs/notification-dir-17apr/node-gui1_notification_logging.log _len=720 _offset=-1 : ERROR not opened /ibm/fvt_fs/notification-dir-17apr/<node-gui1>_notification_logging.log