Uploaded image for project: 'Data Foundation Bugs'
  1. Data Foundation Bugs
  2. DFBUGS-2624

PANIC: process uncaughtException Error: No such file or directory

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Undefined Undefined
    • None
    • odf-4.18.2
    • noobaa-nc
    • None
    • False
    • Hide

      None

      Show
      None
    • False
    • ?
    • ?
    • ppc64le
    • ?
    • ?
    • Hide

      will copy gpfs.snap to the /u/DUMPS on the server. Let me know if the access needs to be provided for the same.

      Show
      will copy gpfs.snap to the /u/DUMPS on the server. Let me know if the access needs to be provided for the same.
    • Important
    • None

      While running Warp as part of verification of the https://issues.redhat.com/browse/DFBUGS-2292
      on the BM cluster , the following noobaa endpoint restarts are observed

      Warp:

      ./warp mixed --insecure --duration 30m --host gpfs-p10-s3-ces.rtp.raleigh.ibm.com:6443 --access-key uOmitMvnLt2SWVPwVWn1 --secret-key Fx37u//7SFYLdxt3SEQjZ/u0hYuXcfN+HemWWx26 --tls --obj.size 64M --disable-multipart --bucket warp-def-2292 --concurrent 1000  2>&1| tee /tmp/warp-def-2292.26may2026.log; date
      Mon May 26 06:57:43 AM EDT 2025
      Benchmarking:                30m00s / 30m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 100.00% 30m3s

       

      warp: Benchmark data written to "warp-mixed-2025-05-26[070944]-urxh.csv.zst"                                                                                  Mixed operations.
      Operation: DELETE, 0%, Concurrency: 0, Ran 0s.
      Skipping DELETE too few samples. Longer benchmark run required for reliable results.

      Operation: GET, 0%, Concurrency: 0, Ran 0s.
      Skipping GET too few samples. Longer benchmark run required for reliable results.

      Operation: PUT, 0%, Concurrency: 0, Ran 0s.
      Skipping PUT too few samples. Longer benchmark run required for reliable results.

      Operation: STAT, 0%, Concurrency: 0, Ran 0s.
      Skipping STAT too few samples. Longer benchmark run required for reliable results.

      Cluster Total: NaN obj/s over 0s.
      warp: Cleanup Done.Mon May 26 11:11:20 AM EDT 2025

      noobaa d/s noobaa-core-5.18.4-20250521

      noobaa process when started on the node-gui1

      ps -ef |grep noobaa
      root     1809760       1  0 05:37 ?        00:00:20 noobaa
      root     1810648 1809760  1 05:37 ?        00:07:24 noobaa
      root     1810651 1809760  1 05:37 ?        00:06:07 noobaa
      root     1810652 1809760  1 05:37 ?        00:07:01 noobaa
      root     1810661 1809760  1 05:37 ?        00:06:54 noobaa
      root     1913556 1809760  1 07:02 ?        00:04:27 noobaa  --> this one and the 07:07 has restarted
      root     1918485 1809760  0 07:07 ?        00:02:52 noobaa

      noobaa.log

      May 26 05:37:16 node-gui1 node[1809760]: [nsfs/1809760]    [L0] core.endpoint.endpoint:: endpoint: replacing old umask:  22 with new umask:  0
      May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760]  [WARN] CONSOLE:: WORKER started { id: 1, pid: 1810648 }
      May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760]  [WARN] CONSOLE:: WORKER started { id: 2, pid: 1810651 }
      May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760]  [WARN] CONSOLE:: WORKER started { id: 3, pid: 1810652 }
      May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760]  [WARN] CONSOLE:: WORKER started { id: 4, pid: 1810659 }
      May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760]  [WARN] CONSOLE:: WORKER started { id: 5, pid: 1810661 }
      May 26 05:37:16 node-gui1 [1809760]: [nsfs/1809760]  [WARN] CONSOLE:: WORKER started { id: 6, pid: 1810673 }

      ...

       

      May 26 07:02:11 node-gui1 [1810648]: 2025-05-26 07:02:11.649468 [PID-1810648/TID-1810775] [L0] FS::FSWorker::Execute: WARNING Fsync _path=/ibm/fvt_fs/new-
      s3user25k2-dir/warp-def-2292/ou0PFCL4  took too long: 39095.4 ms
      May 26 07:02:11 node-gui1 [1810659]: 2025-05-26 07:02:11.650888 [PID-1810659/TID-1810807] [L0] FS::FSWorker::Execute: WARNING FileOpen _path=/ibm/fvt_fs/n
      ew-s3user25k2-dir/warp-def-2292/9FFL(Sni _flags=4210690 _mode=432  took too long: 437.14 ms
      May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 elWHat4Y/1.9c0bePd0OQnsdlHE.rnd
      May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 )XvxQPYK/1.yrwb0C5gwgOnX26E.rnd
      May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 N3vglakP/1.f4HzyJcnutUe8pC2.rnd
      May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 ctLMXuwe/1.AB7KRZ3Xt3Op(mUW.rnd
      May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 8k(G0hvn/1.k7uh9VfGGfq4I65T.rnd
      May 26 07:02:11 node-gui1 [1810652]: [nsfs/1810652]    [L0] core.endpoint.s3.ops.s3_put_object:: PUT OBJECT warp-def-2292 36)thnRl/2.O(XqYnZYB09WDGxj.rnd
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659]    [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name warp-def-2292
      May 26 07:02:11 node-gui1 [1810659]: [nsfs/1810659] [ERROR] CONSOLE:: PANIC: process uncaughtException Error: No such file or directory
      May 26 07:02:11 node-gui1 [1810659]: 2025-05-26 07:02:11.676230 [PID-1810659/TID-1810809] [L0] FS::FSWorker::Execute: WARNING FileWritev _wrap->_path=/ibm/fvt_fs/new-s3user25k2-dir/warp-def-2292/ZRwlZfZF _total_len=8372224 buffers_len=511 _offset=-1  took too long: 159.112 ms

      May 26 07:07:29 node-gui1 [1810673]: [nsfs/1810673] [ERROR] CONSOLE:: PANIC: process uncaughtException Error: FileWrite _wrap->_path=/ibm/fvt_fs/notification-dir-17apr/node-gui1_notification_logging.log _len=720 _offset=-1 : ERROR not opened /ibm/fvt_fs/notification-dir-17apr/<node-gui1>_notification_logging.log

       

       

       

              rh-ee-aprinzse Amit Prinz Setter
              rkomandu@in.ibm.com Ravi Kumar Komanduri (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

                Created:
                Updated:
                Resolved: