Description of problem - 
Scale implemented a crontab which will run the noobaacli lifecycle from there, this is not working and giving following error in the corresponding report file

 

[root@21may-11 lifecycle]# jq . lifecycle_run_1748239862213.json
{
  "running_host": "21may-11",
  "lifecycle_run_times":

{     "run_lifecycle_start_time": 1748239862213,     "list_buckets_start_time": 1748239862230,     "list_buckets_end_time": 1748239862230,     "list_buckets_took_ms": 0,     "process_buckets_start_time": 1748239862230,     "create_gpfs_candidates_files_start_time": 1748239862230,     "create_gpfs_candidates_files_end_time": 1748239862234,     "create_gpfs_candidates_files_took_ms": 4,     "process_buckets_end_time": 1748239862234,     "process_buckets_took_ms": 4,     "run_lifecycle_end_time": 1748239862234,     "run_lifecycle_took_ms": 21   }

,
  "total_stats":

{     "num_objects_deleted": 0,     "num_objects_delete_failed": 0,     "objects_delete_errors": [],     "num_mpu_aborted": 0,     "num_mpu_abort_failed": 0,     "mpu_abort_errors": []   }

,
  "state":

{     "is_finished": false   }

,
  "buckets_statuses": {},
  "errors": [
    "get_mount_points failed with error Error: Command failed: mmlsfs all -T -Y\n/bin/sh: line 1: mmlsfs: command not found\n",
    "get_mount_points failed with error Error: Command failed: mmlsfs all -T -Y\n/bin/sh: line 1: mmlsfs: command not found\n",
    "get_mount_points failed with error Error: Command failed: mmlsfs all -T -Y\n/bin/sh: line 1: mmlsfs: command not found\n"
  ]
}

 

 

The ODF deployment type (Internal, External, Internal-Attached (LSO), Multicluster, DR, Provider, etc):
Scale

 

[root@21may-11 ~]# mmdiag
Current GPFS build: "5.2.3.1".
Built on May 21 2025 at 18:06:23

[root@21may-11 ~]# rpm -qa | grep noobaa
noobaa-core-5.18.4-20250521.el9.x86_64

 

Does this issue impact your ability to continue to work with the product?
YRD

 

 

Is there any workaround available to the best of your knowledge?
No

 

 

Can this issue be reproduced? If so, please provide the hit rate

Yes it is reproducible

 

 

Steps to Reproduce:

1. Create a scale cluster. Create account and a bucket

2. Apply a Bucket lifecycle policy in the bucket 

3. List the buhcketlifecycle policy configuration
[root@fa-akarattu-mgmt-host ~]# aws-test s3api get-bucket-lifecycle-configuration --bucket bucket123
urllib3/connectionpool.py:1064: InsecureRequestWarning: Unverified HTTPS request is being made to host '21may-11'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
{
    "Rules": [
        {
            "Expiration":

{                 "Date": "2022-07-12T00:00:00+00:00"             }

,
            "ID": "Expiration Rule",
            "Filter":

{                 "Prefix": "file"             }

,
            "Status": "Enabled"
        }
    ]
}

 

4. List the crontab and change it to customised time. Wait for the cronjob to run

[root@21may-11 ~]# crontab -l
*/10 * * * * /usr/lpp/mmfs/bin/mmcess3backup /mnt/cesSharedRoot/ces /mnt/cesSharedRoot/ces/mms3-config.json 600
44 10 * * * /usr/local/bin/noobaa-cli lifecycle

Getting below error in noobaa.log 

May 26 10:44:02 21may-11 node[20245]: [noobaa-cli/20245] [ERROR] core.manage_nsfs.nc_lifecycle:: run_lifecycle_under_lock failed with error Error: get_mount_points failed with error Error: Command failed: mmlsfs all -T -Y/bin/sh: line 1: mmlsfs: command not found    at NCLifecycle.get_mount_points_map (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:1150:19)    at async NCLifecycle.create_gpfs_candidates_files (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:1185:43)    at async NCLifecycle._call_op_and_update_status (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:793:21)    at async NCLifecycle.process_buckets (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:166:13)    at async NCLifecycle._call_op_and_update_status (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:793:21)    at async NCLifecycle.run_lifecycle (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:148:9) undefined get_mount_points failed with error Error: Command failed: mmlsfs all -T -Y/bin/sh: line 1: mmlsfs: command not found
May 26 10:44:02 21may-11 node[20245]: [noobaa-cli/20245]    [L0] core.manage_nsfs.nc_lifecycle:: run_lifecycle_under_lock done lifecycle - released lock
May 26 10:44:02 21may-11 node[20245]: [noobaa-cli/20245] [ERROR] core.cmd.manage_nsfs:: manage_nsfs.lifecycle_management: Error while running run_lifecycle_under_lock /mnt/cesSharedRoot/ces/s3-config/config.json Error: get_mount_points failed with error Error: Command failed: mmlsfs all -T -Y/bin/sh: line 1: mmlsfs: command not found    at NCLifecycle.get_mount_points_map (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:1150:19)    at async NCLifecycle.create_gpfs_candidates_files (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:1185:43)    at async NCLifecycle._call_op_and_update_status (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:793:21)    at async NCLifecycle.process_buckets (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:166:13)    at async NCLifecycle._call_op_and_update_status (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:793:21)    at async NCLifecycle.run_lifecycle (/usr/local/noobaa-core/src/manage_nsfs/nc_lifecycle.js:148:9)

 

 

The exact date and time when the issue was observed, including timezone details:

 

Actual results:

Note: If I disable the crontab with "crontab -e" , I can run with "noobaa-cli lifecycle".
However with crontab, it is not running.

 

 

Logs collected and log location:

Attaching noobaa.log

Additional info:

 Please let me know if any other info can be provided