Description of problem -

Bucket Notifications for BLC deletion is not getting generated.
Notifications are generated for object delete via s3 protocol. However no notifications are generated for deletions via BLC worker

 

The OCP platform infrastructure and deployment type (AWS, Bare Metal, VMware, etc. Please clarify if it is platform agnostic deployment), (IPI/UPI):
Scale 
noobaa-core-5.18.4-20250527.el9.x86_64

 

The ODF deployment type (Internal, External, Internal-Attached (LSO), Multicluster, DR, Provider, etc):
Scale 

 

 

The version of all relevant components (OCP, ODF, RHCS, ACM whichever is applicable):

 

 

Does this issue impact your ability to continue to work with the product?

No

 

Is there any workaround available to the best of your knowledge?

No

 

Can this issue be reproduced? If so, please provide the hit rate

Yes, 100%

 

Can this issue be reproduced from the UI?

N/A

If this is a regression, please provide more details to justify this:

 

Steps to Reproduce:

1. Create a bucket and upload some objects.

2. Upload BLC policy 
[root@fa-akarattu-mgmt-host ~]# account8000 s3api get-bucket-lifecycle-configuration --bucket bucket8000
{
    "Rules": [
        {
            "Expiration":

{                 "Date": "2022-07-12T00:00:00+00:00"             }

,
            "ID": "Expiration Rule",
            "Filter":

{                 "Prefix": "file"             }

,
            "Status": "Enabled"
        }
    ]
}

3. Setup notification for the same bucket and make sure that notifications are generated for regular s3 deletion 

[root@fa-akarattu-mgmt-host ~]# account8000 s3 rm s3://bucket8000/file20.txt

Received raw data: {"Records":[{"eventVersion":"2.3","eventSource":"bright-31:s3","eventTime":"2025-06-10T10:17:39.700Z","s3":{"s3SchemaVersion":"1.0","object":

{"sequencer":"19759587874","key":"file20.txt"}

,"bucket":{"name":"bucket8000","ownerIdentity":

{"principalId":"account8000"}

,"arn":"arn:aws:s3:::bucket8000"}},"eventName":"ObjectRemoved:Delete","userIdentity":{"principalId":"account8000"},"requestParameters":{"sourceIPAddress":"::ffff:10.0.100.25"},"responseElements":{"x-amz-request-id":"mbqdagru-77mq90-146o","x-amz-id-2":"mbqdagru-77mq90-146o"}}]}

4. Now run the BLC worker and verify the notifications for deletion of objects --> FAILED

Below is run report. Objects are deleted, however no notifications received for these deletion

[root@bright-31 lifecycle]# jq . lifecycle_run_1749550803313.json
{
  "running_host": "bright-31",
  "lifecycle_run_times":

{     "run_lifecycle_start_time": 1749550803313,     "list_buckets_start_time": 1749550803332,     "list_buckets_end_time": 1749550803333,     "list_buckets_took_ms": 1,     "process_buckets_start_time": 1749550803333,     "process_buckets_end_time": 1749550803580,     "process_buckets_took_ms": 247,     "run_lifecycle_end_time": 1749550803580,     "run_lifecycle_took_ms": 267   }

,
  "total_stats":

{     "num_objects_deleted": 15,     "num_objects_delete_failed": 0,     "objects_delete_errors": [],     "num_mpu_aborted": 0,     "num_mpu_abort_failed": 0,     "mpu_abort_errors": []   }

,
  "state":

{     "is_finished": true   }

,
  "buckets_statuses": {
    "bucket7000": {
      "bucket_process_times":

{         "process_bucket_start_time": 1749550803333,         "process_bucket_end_time": 1749550803543,         "process_bucket_took_ms": 210       }

,
      "bucket_stats": {},
      "state":

{         "num_processed_objects": 0,         "is_finished": true       }

,
      "rules_statuses": {
        "Expiration-simple": {
          "state": {
            "expire":

{               "is_finished": true             }

,
            "noncurrent":

{               "is_finished": true             }

,
            "is_finished": true
          },
          "rule_process_times":

{             "process_rule_start_time": 1749550803336,             "get_candidates_start_time": 1749550803338,             "get_candidates_end_time": 1749550803543,             "get_candidates_took_ms": 205,             "process_rule_end_time": 1749550803543,             "process_rule_took_ms": 207           }

,
          "rule_stats": {}
        }
      }
    },
    "bucket567": {
      "bucket_process_times":

{         "process_bucket_start_time": 1749550803334,         "process_bucket_end_time": 1749550803541,         "process_bucket_took_ms": 207       }

,
      "bucket_stats": {},
      "state":

{         "num_processed_objects": 0,         "is_finished": true       }

,
      "rules_statuses": {
        "Expiration-simple": {
          "state": {
            "expire": {},
            "noncurrent": {},
            "is_finished": true
          },
          "rule_process_times":

{             "process_rule_start_time": 1749550803340,             "get_candidates_start_time": 1749550803340,             "get_candidates_end_time": 1749550803541,             "get_candidates_took_ms": 201,             "process_rule_end_time": 1749550803541,             "process_rule_took_ms": 201           }

,
          "rule_stats": {},
          "errors": [
            "No such file or directory"
          ]
        }
      }
    },
    "bucket-123": {
      "bucket_process_times":

{         "process_bucket_start_time": 1749550803334,         "process_bucket_end_time": 1749550803341,         "process_bucket_took_ms": 7       }

,
      "bucket_stats": {},
      "state":

{         "is_finished": true       }

,
      "rules_statuses": {}
    },
    "bucket8000": {
      "bucket_process_times":

{         "process_bucket_start_time": 1749550803334,         "process_bucket_end_time": 1749550803580,         "process_bucket_took_ms": 246       }

,
      "bucket_stats":

{         "num_objects_deleted": 15,         "num_objects_delete_failed": 0,         "objects_delete_errors": []       }

,
      "state":

{         "num_processed_objects": 15,         "is_finished": true       }

,
      "rules_statuses": {
        "Expiration Rule": {
          "state": {
            "expire":

{               "is_finished": true             }

,
            "noncurrent": {},
            "is_finished": true
          },
          "rule_process_times":

{             "process_rule_start_time": 1749550803341,             "get_candidates_start_time": 1749550803342,             "get_candidates_end_time": 1749550803554,             "get_candidates_took_ms": 212,             "delete_multiple_objects_start_time": 1749550803555,             "delete_multiple_objects_end_time": 1749550803579,             "delete_multiple_objects_took_ms": 24,             "process_rule_end_time": 1749550803580,             "process_rule_took_ms": 239           }

,
          "rule_stats":

{             "num_objects_deleted": 15,             "num_objects_delete_failed": 0,             "objects_delete_errors": []           }

        }
      }
    }
  }
}

 

The exact date and time when the issue was observed, including timezone details:

 

Actual results:

 

 

Expected results:

 

Logs collected and log location:

 

Additional info: