Loading...

XML

Word

Printable

Type: Bug
Resolution: Not a Bug
Priority: Undefined
Fix Version/s: None
Affects Version/s: odf-4.18.2
Component/s: noobaa-nc
Labels:
None

Blocked:
False
Blocked Reason:

Hide

None

Show
None
Ready:
False
Dev Approval:
?
Docs Approval:
?
PM Approval:
?
QE Approval:
?
Status Summary:

Hide

notification logs shown at each crontab entry (2 min in my case), generated the failure_logging and append to that even when the port has been changed in the connections and for fresh uploads as well..

[root@rk23bld-22 ~]# cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.log
{"meta":\{"connect":"notify_event","name":"notify_event","bucket":"newbucket-defvfy"}
,"notif":{"Records":[{"eventVersion":"2.3","eventSource":"rk23bld-22:s3","eventTime":"2025-04-24T06:59:07.303Z","s3":{"s3SchemaVersion":"1.0","object":
{"sequencer":"1966697cfa7","key":"file_1M_11","size":1048576,"eTag":"mtime-d9eoaj5o8oow-ino-51g"}
,"bucket":{"name":"newbucket-defvfy","ownerIdentity":
{"principalId":"s3user21k1"}
,"arn":"arn:aws:s3:::newbucket-defvfy"}},"eventName":"ObjectCreated:Put","userIdentity":{"principalId":"s3user21k1"},"requestParameters":{"sourceIPAddress":"::ffff:192.168.0.77"},"responseElements":{"x-amz-request-id":"m9v0i3qw-4g7fej-a7x","x-amz-id-2":"m9v0i3qw-4g7fej-a7x"}}]}}

[root@rk23bld-22 ~]# cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478005569.log
{"meta":\{"connect":"notify_event","name":"notify_event","bucket":"newbucket-defvfy"}
,"notif":{"Records":[{"eventVersion":"2.3","eventSource":"rk23bld-22:s3","eventTime":"2025-04-24T06:59:07.303Z","s3":{"s3SchemaVersion":"1.0","object":
{"sequencer":"1966697cfa7","key":"file_1M_11","size":1048576,"eTag":"mtime-d9eoaj5o8oow-ino-51g"}
,"bucket":{"name":"newbucket-defvfy","ownerIdentity":
{"principalId":"s3user21k1"}
,"arn":"arn:aws:s3:::newbucket-defvfy"}},"eventName":"ObjectCreated:Put","userIdentity":{"principalId":"s3user21k1"},"requestParameters":{"sourceIPAddress":"::ffff:192.168.0.77"},"responseElements":{"x-amz-request-id":"m9v0i3qw-4g7fej-a7x","x-amz-id-2":"m9v0i3qw-4g7fej-a7x"}}]}}
[root@rk23bld-22 ~]# cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478005569.log
cat: /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478005569.log: No such file or directory

[root@rk23bld-22 ~]# cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478483542.log
{"meta":\{"connect":"notify_event","name":"notify_event","bucket":"newbucket-defvfy"}
,"notif":{"Records":[{"eventVersion":"2.3","eventSource":"rk23bld-22:s3","eventTime":"2025-04-24T06:59:07.303Z","s3":{"s3SchemaVersion":"1.0","object":
{"sequencer":"1966697cfa7","key":"file_1M_11","size":1048576,"eTag":"mtime-d9eoaj5o8oow-ino-51g"}
,"bucket":{"name":"newbucket-defvfy","ownerIdentity":
{"principalId":"s3user21k1"}
,"arn":"arn:aws:s3:::newbucket-defvfy"}},"eventName":"ObjectCreated:Put","userIdentity":{"principalId":"s3user21k1"},"requestParameters":{"sourceIPAddress":"::ffff:192.168.0.77"},"responseElements":{"x-amz-request-id":"m9v0i3qw-4g7fej-a7x","x-amz-id-2":"m9v0i3qw-4g7fej-a7x"}}]}}
[root@rk23bld-22 ~]# cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478483542.log
cat: /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478483542.log: No such file or directory

[root@rk23bld-22 ~]# cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478724083.log
{"meta":\{"connect":"notify_event","name":"notify_event","bucket":"newbucket-defvfy"}
,"notif":{"Records":[{"eventVersion":"2.3","eventSource":"rk23bld-22:s3","eventTime":"2025-04-24T06:59:07.303Z","s3":{"s3SchemaVersion":"1.0","object":
{"sequencer":"1966697cfa7","key":"file_1M_11","size":1048576,"eTag":"mtime-d9eoaj5o8oow-ino-51g"}
,"bucket":{"name":"newbucket-defvfy","ownerIdentity":
{"principalId":"s3user21k1"}
,"arn":"arn:aws:s3:::newbucket-defvfy"}},"eventName":"ObjectCreated:Put","userIdentity":{"principalId":"s3user21k1"},"requestParameters":{"sourceIPAddress":"::ffff:192.168.0.77"},"responseElements":{"x-amz-request-id":"m9v0i3qw-4g7fej-a7x","x-amz-id-2":"m9v0i3qw-4g7fej-a7x"}}]}}

Noobaa logs

Apr 24 09:00:05 rk23bld-23 node[866363]: 2025-04-24 09:00:05.435127 [PID-866363/TID-866456] [L0] FS::FSWorker::Execute: WARNING FileFcntlLock_wrap->_path=/mnt
/fsnotify/notification-dir/rk23bld-23_notification_logging.1745478004446.log took too long: 911.123 ms
Apr 24 09:00:05 rk23bld-23 node[866363]: [noobaa-cli/866363] [L0] core.manage_nsfs.nc_master_key_manager:: init_from_exec: get master keys response status=
OK, version=1
Apr 24 09:00:05 rk23bld-23 node[866363]: [noobaa-cli/866363] [ERROR] core.util.notifications_util:: Notify err = Error: connect EHOSTUNREACH 10.0.100.77:9080
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1610:16)
{ errno: -113, code: 'EHOSTUNREACH', syscall: 'connect', address: '10.0.100.77', por t: 9080}
Apr 24 09:04:04 rk23bld-23 node[889278]: [noobaa-cli/889278] [ERROR] core.util.notifications_util:: Notify err = Error: connect EHOSTUNREACH 10.0.100.77:9080
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1610:16)
{ errno: -113, code: 'EHOSTUNREACH', syscall: 'connect', address: '10.0.100.77', por t: 9080}
application node the Webhook is running :

[root@rkomanduri-scale-host BN]# netstat -anp |grep webhook
tcp6 0 0 :::9080 :::* LISTEN 28431/webhook

notification list which shows 9080 as the port

Name Notification Protocol Host Port Timeout
----- ----------------- — — -----------
notify_event http 10.0.100.77 9080 1000

Show
notification logs shown at each crontab entry (2 min in my case), generated the failure_logging and append to that even when the port has been changed in the connections and for fresh uploads as well.. [root@rk23bld-22 ~] # cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.log {"meta":\{"connect":"notify_event","name":"notify_event","bucket":"newbucket-defvfy"} ,"notif":{"Records":[{"eventVersion":"2.3","eventSource":"rk23bld-22:s3","eventTime":"2025-04-24T06:59:07.303Z","s3":{"s3SchemaVersion":"1.0","object": {"sequencer":"1966697cfa7","key":"file_1M_11","size":1048576,"eTag":"mtime-d9eoaj5o8oow-ino-51g"} ,"bucket":{"name":"newbucket-defvfy","ownerIdentity": {"principalId":"s3user21k1"} ,"arn":"arn:aws:s3:::newbucket-defvfy"}},"eventName":"ObjectCreated:Put","userIdentity":{"principalId":"s3user21k1"},"requestParameters":{"sourceIPAddress":"::ffff:192.168.0.77"},"responseElements":{"x-amz-request-id":"m9v0i3qw-4g7fej-a7x","x-amz-id-2":"m9v0i3qw-4g7fej-a7x"}}]}} [root@rk23bld-22 ~] # cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478005569.log {"meta":\{"connect":"notify_event","name":"notify_event","bucket":"newbucket-defvfy"} ,"notif":{"Records":[{"eventVersion":"2.3","eventSource":"rk23bld-22:s3","eventTime":"2025-04-24T06:59:07.303Z","s3":{"s3SchemaVersion":"1.0","object": {"sequencer":"1966697cfa7","key":"file_1M_11","size":1048576,"eTag":"mtime-d9eoaj5o8oow-ino-51g"} ,"bucket":{"name":"newbucket-defvfy","ownerIdentity": {"principalId":"s3user21k1"} ,"arn":"arn:aws:s3:::newbucket-defvfy"}},"eventName":"ObjectCreated:Put","userIdentity":{"principalId":"s3user21k1"},"requestParameters":{"sourceIPAddress":"::ffff:192.168.0.77"},"responseElements":{"x-amz-request-id":"m9v0i3qw-4g7fej-a7x","x-amz-id-2":"m9v0i3qw-4g7fej-a7x"}}]}} [root@rk23bld-22 ~] # cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478005569.log cat: /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478005569.log: No such file or directory [root@rk23bld-22 ~] # cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478483542.log {"meta":\{"connect":"notify_event","name":"notify_event","bucket":"newbucket-defvfy"} ,"notif":{"Records":[{"eventVersion":"2.3","eventSource":"rk23bld-22:s3","eventTime":"2025-04-24T06:59:07.303Z","s3":{"s3SchemaVersion":"1.0","object": {"sequencer":"1966697cfa7","key":"file_1M_11","size":1048576,"eTag":"mtime-d9eoaj5o8oow-ino-51g"} ,"bucket":{"name":"newbucket-defvfy","ownerIdentity": {"principalId":"s3user21k1"} ,"arn":"arn:aws:s3:::newbucket-defvfy"}},"eventName":"ObjectCreated:Put","userIdentity":{"principalId":"s3user21k1"},"requestParameters":{"sourceIPAddress":"::ffff:192.168.0.77"},"responseElements":{"x-amz-request-id":"m9v0i3qw-4g7fej-a7x","x-amz-id-2":"m9v0i3qw-4g7fej-a7x"}}]}} [root@rk23bld-22 ~] # cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478483542.log cat: /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478483542.log: No such file or directory [root@rk23bld-22 ~] # cat /mnt/fsnotify/notification-dir/rk23bld-23_notification_logging.failure.1745478724083.log {"meta":\{"connect":"notify_event","name":"notify_event","bucket":"newbucket-defvfy"} ,"notif":{"Records":[{"eventVersion":"2.3","eventSource":"rk23bld-22:s3","eventTime":"2025-04-24T06:59:07.303Z","s3":{"s3SchemaVersion":"1.0","object": {"sequencer":"1966697cfa7","key":"file_1M_11","size":1048576,"eTag":"mtime-d9eoaj5o8oow-ino-51g"} ,"bucket":{"name":"newbucket-defvfy","ownerIdentity": {"principalId":"s3user21k1"} ,"arn":"arn:aws:s3:::newbucket-defvfy"}},"eventName":"ObjectCreated:Put","userIdentity":{"principalId":"s3user21k1"},"requestParameters":{"sourceIPAddress":"::ffff:192.168.0.77"},"responseElements":{"x-amz-request-id":"m9v0i3qw-4g7fej-a7x","x-amz-id-2":"m9v0i3qw-4g7fej-a7x"}}]}} Noobaa logs Apr 24 09:00:05 rk23bld-23 node [866363] : 2025-04-24 09:00:05.435127 [PID-866363/TID-866456] [L0] FS::FSWorker::Execute: WARNING FileFcntlLock_wrap->_path=/mnt /fsnotify/notification-dir/rk23bld-23_notification_logging.1745478004446.log took too long: 911.123 ms Apr 24 09:00:05 rk23bld-23 node [866363] : [noobaa-cli/866363] [L0] core.manage_nsfs.nc_master_key_manager:: init_from_exec: get master keys response status= OK, version=1 Apr 24 09:00:05 rk23bld-23 node [866363] : [noobaa-cli/866363] [ERROR] core.util.notifications_util:: Notify err = Error: connect EHOSTUNREACH 10.0.100.77:9080 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1610:16) { errno: -113, code: 'EHOSTUNREACH', syscall: 'connect', address: '10.0.100.77', por t: 9080} Apr 24 09:04:04 rk23bld-23 node [889278] : [noobaa-cli/889278] [ERROR] core.util.notifications_util:: Notify err = Error: connect EHOSTUNREACH 10.0.100.77:9080 at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1610:16) { errno: -113, code: 'EHOSTUNREACH', syscall: 'connect', address: '10.0.100.77', por t: 9080} application node the Webhook is running : [root@rkomanduri-scale-host BN] # netstat -anp |grep webhook tcp6 0 0 :::9080 :::* LISTEN 28431/webhook notification list which shows 9080 as the port Name Notification Protocol Host Port Timeout ----- ----------------- — — ----------- notify_event http 10.0.100.77 9080 1000
Target Release:

odf-4.19

Regression:
None

SFDC Cases Links:
SFDC Cases Open:
SFDC Cases Counter:

Description of problem - Provide a detailed description of the issue encountered, including logs/command-output snippets and screenshots if the issue is observed in the UI:

Tried on the noobaa-5.18.2-20250418 d/s build.

The OCP platform infrastructure and deployment type (AWS, Bare Metal, VMware, etc. Please clarify if it is platform agnostic deployment), (IPI/UPI):

The ODF deployment type (Internal, External, Internal-Attached (LSO), Multicluster, DR, Provider, etc):

The version of all relevant components (OCP, ODF, RHCS, ACM whichever is applicable):

Does this issue impact your ability to continue to work with the product?

Is there any workaround available to the best of your knowledge?

Can this issue be reproduced? If so, please provide the hit rate

Can this issue be reproduced from the UI?

If this is a regression, please provide more details to justify this:

Steps to Reproduce:

1. Enable bucket notification on a bucket, upload IO and observe in the Webhook (target) server

2. Change the port number via the mms3 or the notification cli

3. Upload an object, it would continue to send to the new port on the host, but unable to do so

The exact date and time when the issue was observed, including timezone details:

Actual results:

Expected results:

Logs collected and log location:

Additional info:

Assignee:: Nimrod Becker

Reporter:: Ravi Kumar Komanduri (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 11 Start watching this issue

Created:: 2025/04/24 4:40 PM

Updated:: 2025/04/25 4:27 PM

Resolved:: 2025/04/25 4:27 PM

Details

Description

Attachments

Easy Agile Planning Poker

Activity

People

Dates

PagerDuty