*Description of problem - Provide a detailed description of the issue encountered, including logs/command-output snippets and screenshots if the issue is observed in the UI: Duplicate entries in webhook log

Whenever webhook becomes unreachable the whole notification.log file resends the notification which is a duplicate of the already sent notification. We need to have mechanism of keeping the track of already sent notification

 

*The OCP platform infrastructure and deployment type (AWS, Bare Metal, VMware, etc. Please clarify if it is platform agnostic deployment), (IPI/UPI):Bare Metal

 

The ODF deployment type (Internal, External, Internal-Attached (LSO), Multicluster, DR, Provider, etc):

 

 

The version of all relevant components (OCP, ODF, RHCS, ACM whichever is applicable):

 ODF-4.18.2

 

Does this issue impact your ability to continue to work with the product?

 

 

Is there any workaround available to the best of your knowledge?

 

 

Can this issue be reproduced? If so, please provide the hit rate

 

 

Can this issue be reproduced from the UI?

 

If this is a regression, please provide more details to justify this:

 

Steps to Reproduce:

1. warp put --host=172.20.100.60m:6443 --access-key="$access_key" --secret-key="$secret_key" --obj.size=1k --duration=20m --disable-multipart  --bucket="user001-bucket1" --tls --insecure --noclear --concurrent 1 &

cat webhook-2025-04-17_error_injection.log | grep w user001-bucket1 |  tr -d '
' |  cut -c 164 | rev | cut c 48 | rev | jq | grep key | sort | uniq -d | wc -l
272

1. cat webhook-2025-04-17_error_injection.log | grep w user001-bucket1 |  tr -d '
   ' |  cut -c 164 | rev | cut c 48 | rev | jq | grep "8c(erAar/1.7kEQDbHvqz7Zqrzn.rnd"
             "key": "8c(erAar/1.7kEQDbHvqz7Zqrzn.rnd",
             "key": "8c(erAar/1.7kEQDbHvqz7Zqrzn.rnd",

1. cat webhook-2025-04-17_error_injection.log | grep w user001-bucket1 |  tr -d '
   ' |  cut -c 164 | rev | cut c 48 | rev | jq | grep "8c(erAar/1.7kEQDbHvqz7Zqrzn.rnd" -C 15
        },
         "responseElements": {         "x-amz-id-2": "m9lujgzb-85huss-15my",         "x-amz-request-id": "m9lujgzb-85huss-15my"       }

   ,
         "s3": {
           "bucket":

   Unknown macro: {          "arn"}

   ,
           "object":

   {           "eTag": "1c9844d95240d32b1f616a35a30caa8f",           "key": "8c(erAar/1.7kEQDbHvqz7Zqrzn.rnd",           "sequencer": "196458f419e",           "size": 1000,           "versionId": "mtime-d997uauhtibk-ino-5dky"         }

   ,
           "s3SchemaVersion": "1.0"
         },
         "userIdentity":

   {         "principalId": "user001"       }

       }
     ]
   }
   {
     "Records": [
      

   { --       }

   ,
         "responseElements":

   {         "x-amz-id-2": "m9lujgzb-85huss-15my",         "x-amz-request-id": "m9lujgzb-85huss-15my"       }

   ,
         "s3": {
           "bucket":

   Unknown macro: {          "arn"}

   ,
           "object":

   {           "eTag": "1c9844d95240d32b1f616a35a30caa8f",           "key": "8c(erAar/1.7kEQDbHvqz7Zqrzn.rnd",           "sequencer": "196458f419e",           "size": 1000,           "versionId": "mtime-d997uauhtibk-ino-5dky"         }

   ,
           "s3SchemaVersion": "1.0"
         },
         "userIdentity":

   {         "principalId": "user001"       }

       }
     ]
   }

2.[webhook] 2025/04/17 17:10:19 [ee630d] 200 | 0 B | 176.084µs | localhost | POST /hooks/webhook
[webhook] 2025/04/17 17:10:19 [ee630d] executing /root/webhook/test/hookecho.sh (/root/webhook/test/hookecho.sh) with arguments ["/root/webhook/test/hookecho.sh" "{\"Records\":[{\"eventName\":\"ObjectCreated:Put\",\"eventSource\":\"c83f2-dan9:s3\",\"eventTime\":\"2025-04-17T21:02:18.526Z\",\"eventVersion\":\"2.3\",\"requestParameters\":

{\"sourceIPAddress\":\"::f8\"}

,\"responseElements\":

{\"x-amz-id-2\":\"m9lujgzb-85huss-15my\",\"x-amz-request-id\":\"m9lujgzb-85huss-15my\"}

,\"s3\":{\"bucket\":{\"arn\":\"arn:aws:s3:::user001-bucket1\",\"name\":\"user001-bucket1\",\"ownerIdentity\":{\"principalId\":\"user001\"}},\"object\":

{\"eTag\":\"1c9844d95240d32b1f616a35a30caa8f\",\"key\":\"8c(erAar/1.7kEQDbHvqz7Zqrzn.rnd\",\"sequencer\":\"196458f419e\",\"size\":1000,\"versionId\":\"mtime-d997uauhtibk-ino-5dky\"}

,\"s3SchemaVersion\":\"1.0\"},\"userIdentity\":{\"principalId\":\"user001\"}}]}"] and environment [] using /tmp/webhook as cwd

[webhook] 2025/04/17 17:10:19 [ee630d] incoming HTTP POST request from 172.20.100.141:47218
[webhook] 2025/04/17 17:10:19 [ee630d] webhook got matched
[webhook] 2025/04/17 17:10:19 [ee630d] webhook hook triggered successfully
[webhook] 2025/04/17 17:10:19 [ee630d] 200 | 0 B | 176.084µs | localhost | POST /hooks/webhook
[webhook] 2025/04/17 17:10:19 [ee630d] executing /root/webhook/test/hookecho.sh (/root/webhook/test/hookecho.sh) with arguments ["/root/webhook/test/hookecho.sh" "{\"Records\":[{\"eventName\":\"ObjectCreated:Put\",\"eventSource\":\"c83f2-dan9:s3\",\"eventTime\":\"2025-04-17T21:02:18.526Z\",\"eventVersion\":\"2.3\",\"requestParameters\":

{\"sourceIPAddress\":\"::f8\"}

,\"responseElements\":

{\"x-amz-id-2\":\"m9lujgzb-85huss-15my\",\"x-amz-request-id\":\"m9lujgzb-85huss-15my\"}

,\"s3\":{\"bucket\":{\"arn\":\"arn:aws:s3:::user001-bucket1\",\"name\":\"user001-bucket1\",\"ownerIdentity\":{\"principalId\":\"user001\"}},\"object\":

{\"eTag\":\"1c9844d95240d32b1f616a35a30caa8f\",\"key\":\"8c(erAar/1.7kEQDbHvqz7Zqrzn.rnd\",\"sequencer\":\"196458f419e\",\"size\":1000,\"versionId\":\"mtime-d997uauhtibk-ino-5dky\"}

,\"s3SchemaVersion\":\"1.0\"},\"userIdentity\":{\"principalId\":\"user001\"}}]}"] and environment [] using /tmp/webhook as cwd
[webhook] 2025/04/17 17:10:19 [2f31a5] command output:
[webhook] 2025/04/17 17:10:19 [2f31a5] finished handling webhook
[webhook] 2025/04/17 17:10:19 [2f75a9] incoming HTTP POST request from 172.20.100.141:47232
[webhook] 2025/04/17 17:10:19 [2f75a9] webhook got matched
[webhoo

[webhook] 2025/04/17 17:39:29 [4d7328] incoming HTTP POST request from 172.20.100.140:44844
[webhook] 2025/04/17 17:39:29 [4d7328] webhook got matched
[webhook] 2025/04/17 17:39:29 [4d7328] webhook hook triggered successfully
[webhook] 2025/04/17 17:39:29 [4d7328] 200 | 0 B | 142.256µs | localhost | POST /hooks/webhook
[webhook] 2025/04/17 17:39:29 [4d7328] executing /root/webhook/test/hookecho.sh (/root/webhook/test/hookecho.sh) with arguments ["/root/webhook/test/hookecho.sh" "{\"Records\":[{\"eventName\":\"ObjectCreated:Put\",\"eventSource\":\"c83f2-dan9:s3\",\"eventTime\":\"2025-04-17T21:02:18.526Z\",\"eventVersion\":\"2.3\",\"requestParameters\":

{\"sourceIPAddress\":\"::f8\"}

,\"responseElements\":

{\"x-amz-id-2\":\"m9lujgzb-85huss-15my\",\"x-amz-request-id\":\"m9lujgzb-85huss-15my\"}

,\"s3\":{\"bucket\":{\"arn\":\"arn:aws:s3:::user001-bucket1\",\"name\":\"user001-bucket1\",\"ownerIdentity\":{\"principalId\":\"user001\"}},\"object\":

{\"eTag\":\"1c9844d95240d32b1f616a35a30caa8f\",\"key\":\"8c(erAar/1.7kEQDbHvqz7Zqrzn.rnd\",\"sequencer\":\"196458f419e\",\"size\":1000,\"versionId\":\"mtime-d997uauhtibk-ino-5dky\"}

,\"s3SchemaVersion\":\"1.0\"},\"userIdentity\":{\"principalId\":\"user001\"}}]}"] and environment [] using /tmp/webhook as cwd
[webhook] 2025/04/17 17:39:29 [9fc694] command output:
[webhook] 2025/04/17 17:39:29 [9fc694] finished handling webhook
[webhook] 2025/04/17 17:39:29 [9c96f8] incoming HTTP POST request from 172.20.100.140:44832
[webhook] 2025/04/17 17:39:29 [9c96f8] webhook got matched
[webhook] 2025/04/17 17:39:29 [9c96f8] webhook hook triggered successfully
[webhook] 2025/

Apr 17 17:11:09 c83f2-dan10 node[3925023]: {"timestamp":"2025-04-17T21:11:09.235Z","host":"","event":{"code":"noobaa_notification_failed","message":"Failed to send notification., value: secure_notify_event","description":"Notification failed., error: Error: connect ECONNREFUSED ","entity_type":"NODE","event_type":"WARN","scope":"NODE","severity":"WARN","state":"HEALTHY","arguments":

{"errno":-111,"code":"ECONNREFUSED","syscall":"connect","address":"","port":9000}

,"pid":3925023}}
Apr 17 17:11:09 c83f2-dan10 node[3925023]: {"timestamp":"2025-04-17T21:11:09.239Z","host":"","event":{"code":"noobaa_notification_failed","message":"Failed to send notification., value: secure_notify_event","description":"Notification failed., error: Error: connect ECONNREFUSED

3.

 

The exact date and time when the issue was observed, including timezone details:

 

Actual results:

 

 

Expected results:

 

Logs collected and log location:

 

Additional info: