While performing testing for bucket notifications, we get Access Denied error while using warp tool:

upload error: Access Denied       01m27s / 05m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  29.02%
upload error: Access Denied       01m27s / 05m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  29.18%
upload error: Access Denied       01m28s / 05m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  29.55%
upload error: Access Denied       01m26s / 05m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  28.88%
upload error: Access Denied       01m28s / 05m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  29.42%
upload error: Access Denied       01m28s / 05m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  29.38%
upload error: Access Denied       01m29s / 05m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  29.67%
upload error: Access Denied       01m28s / 05m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  29.63%
upload error: Access Denied       01m29s / 05m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  29.88%
upload error: Access Denied       01m29s / 05m00s ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓█░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░┃  29.69%
upload error: Access Denied       01m28s / 05m00s ┃▓

warp workload was executed as follows:

warp put --host=172.20.100.60:6443 --access-key="$access_key" --secret-key="$secret_key" --obj.size=1k --duration=5m --disable-multipart  --bucket="user001-bucket1" --tls --insecure --noclear --concurrent 1

workload was distributed as follows:

250 warp instances per app node, distributed along 4 app nodes for a total of 1,000 concurrent instances. Approximately 62 instances per ces IP.

Errors in noobaa.log looks as follows:

{*}Apr 10 16:51:59 c83f2-dan10 [4003905]: [nsfs/4003905] ^{[[36m  [LOG]}[[39m CONSOLE:: { timestamp: ^{[[33m1744318319883}[[39m, group_name: ^{[[32m'default-endpoint-group'}[[39m, hostname: ^{[[32m'c83f2-dan10'}[[39m, cpu:

{ count: ^[[33m64^[[39m, usage: ^[[33m0.10812910155015307^[[39m }

, memory: { total: ^{[[33m8356247424}[[39m, used: ^{[[33m177111040}[[39m }, s3_ops: { usage:

{ total_calls: ^[[33m2995^[[39m, put_object: ^[[33m2995^[[39m }

, errors: { total_errors: ^[[33m139[[39m, AccessDenied: ^[[33m139[[39m } }, bandwidth: [ { bucket: ^{[[32m'user032-bucket7'}{*}[[39m, access_key: ^{[[32m'7nOMRmill3vvu6BUtBkA'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m198[[39m, write_bytes: ^[[33m198000[[39m }, { bucket: ^{[[32m'user052-bucket1'}[[39m, access_key: ^{[[32m'ojviSaEXb1KtVpfBT5d9'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m148[[39m, write_bytes: ^[[33m148000[[39m }, { bucket: ^{[[32m'user090-bucket4'}[[39m, access_key: ^{[[32m'Q13r3GpUfInc0qt0mpZy'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m164[[39m, write_bytes: ^[[33m164000[[39m }, { bucket: ^{[[32m'user084-bucket1'}[[39m, access_key: ^{[[32m'HBse2NRhVN0k5xdXsJJf'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m213[[39m, write_bytes: ^[[33m213000[[39m }, { bucket: ^{[[32m'user045-bucket6'}[[39m, access_key: ^{[[32m'yRWyCTRoLnKFVNWUtgXw'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m199[[39m, write_bytes: ^[[33m199000[[39m }, { bucket: ^{[[32m'user020-bucket1'}[[39m, access_key: ^{[[32m'DAObTHBjJQDMqgQOqjge'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m197[[39m, write_bytes: ^[[33m197000[[39m }, { bucket: ^{[[32m'user013-bucket6'}[[39m, access_key: ^{[[32m'6Xatty2aT6DUUdINYgy3'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m197[[39m, write_bytes: ^[[33m197000[[39m }, { bucket: ^{[[32m'user026-bucket4'}[[39m, access_key: ^{[[32m'6fW0ykuW8ldSJc4bZig4'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m198[[39m, write_bytes: ^[[33m198000[[39m }, { bucket: ^{[[32m'user096-bucket6'}[[39m, access_key: ^{[[32m'6kk1X5FpcthUVzsEfI0Y'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m244[[39m, write_bytes: ^[[33m244000[[39m }, { bucket: ^{[[32m'user076-bucket4'}[[39m, access_key: ^{[[32m'r9mKMQ9eOIftNMJ96ucC'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m212[[39m, write_bytes: ^[[33m212000[[39m }, { bucket: ^{[[32m'user007-bucket3'}[[39m, access_key: ^{[[32m'g6EBlNviB5iAHqhcTfgi'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m195[[39m, write_bytes: ^[[33m195000[[39m }, { bucket: ^{[[32m'user058-bucket4'}[[39m, access_key: ^{[[32m'lw9tL9stABUIqqzuQ7QM'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m200[[39m, write_bytes: ^[[33m200000[[39m }, { bucket: ^{[[32m'user039-bucket3'}[[39m, access_key: ^{[[32m'bqHNcDgCfmN5Iozj94Wa'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m197[[39m, write_bytes: ^[[33m197000[[39m }, { bucket: ^{[[32m'user064-bucket7'}[[39m, access_key: ^{[[32m'eVjmgPgD8HQWi9mgqGrD'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m155[[39m, write_bytes: ^[[33m155000[[39m }, { bucket: ^{[[32m'user071-bucket3'}[[39m, access_key: ^{[[32m'M8IEzFHhe3ltQo67HrEp'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m150[[39m, write_bytes: ^[[33m150000[[39m } ] }
Apr 10 16:51:59 c83f2-dan10 [4003965]: [nsfs/4003965] ^{[[36m  [LOG]}[[39m CONSOLE:: { timestamp: ^{[[33m1744318319884}[[39m, group_name: ^{[[32m'default-endpoint-group'}[[39m, hostname: ^{[[32m'c83f2-dan10'}[[39m, cpu:

{ count: ^[[33m64^[[39m, usage: ^[[33m0.1109802418356855^[[39m }

, memory: { total: ^{[[33m8356247424}[[39m, used: ^{[[33m133599232}[[39m }, s3_ops: { usage:

{ total_calls: ^[[33m3114^[[39m, put_object: ^[[33m3114^[[39m }

, errors: { total_errors: ^[[33m214[[39m, AccessDenied: ^[[33m214[[39m } }, bandwidth: [ { bucket: ^{[[32m'user065-bucket1'}[[39m, access_key: ^{[[32m'uBGnGxq0iYpx4Cs8xBGL'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m156[[39m, write_bytes: ^[[33m156000[[39m }, { bucket: ^{[[32m'user084-bucket4'}[[39m, access_key: ^{[[32m'HBse2NRhVN0k5xdXsJJf'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m211[[39m, write_bytes: ^[[33m211000[[39m }, { bucket: ^{[[32m'user097-bucket4'}[[39m, access_key: ^{[[32m'lKKkQ0NVDGHGd4Btgp1M'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m246[[39m, write_bytes: ^[[33m246000[[39m }, { bucket: ^{[[32m'user058-bucket6'}[[39m, access_key: ^{[[32m'lw9tL9stABUIqqzuQ7QM'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m199[[39m, write_bytes: ^[[33m199000[[39m }, { bucket: ^{[[32m'user090-bucket6'}[[39m, access_key: ^{[[32m'Q13r3GpUfInc0qt0mpZy'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m213[[39m, write_bytes: ^[[33m213000[[39m }, { bucket: ^{[[32m'user052-bucket4'}[[39m, access_key: ^{[[32m'ojviSaEXb1KtVpfBT5d9'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m198[[39m, write_bytes: ^[[33m198000[[39m }, { bucket: ^{[[32m'user026-bucket6'}[[39m, access_key: ^{[[32m'6fW0ykuW8ldSJc4bZig4'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m141[[39m, write_bytes: ^[[33m141000[[39m }, { bucket: ^{[[32m'user033-bucket1'}[[39m, access_key: ^{[[32m'jpxx0BvgG8paql25ZIOI'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m86[[39m, write_bytes: ^[[33m86000[[39m }, { bucket: ^{[[32m'user077-bucket6'}[[39m, access_key: ^{[[32m'MUMCz4T0x9NJAuVfioEt'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m211[[39m, write_bytes: ^[[33m211000[[39m }, { bucket: ^{[[32m'user001-bucket1'}[[39m, access_key: ^{[[32m'y9JBwC3d0yDMfXv4US4h'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m192[[39m, write_bytes: ^[[33m192000[[39m }, { bucket: ^{[[32m'user020-bucket3'}[[39m, access_key: ^{[[32m'DAObTHBjJQDMqgQOqjge'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m193[[39m, write_bytes: ^[[33m193000[[39m }, { bucket: ^{[[32m'user013-bucket7'}[[39m, access_key: ^{[[32m'6Xatty2aT6DUUdINYgy3'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m192[[39m, write_bytes: ^[[33m192000[[39m }, { bucket: ^{[[32m'user007-bucket4'}[[39m, access_key: ^{[[32m'g6EBlNviB5iAHqhcTfgi'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m138[[39m, write_bytes: ^[[33m138000[[39m }, { bucket: ^{[[32m'user071-bucket4'}[[39m, access_key: ^{[[32m'M8IEzFHhe3ltQo67HrEp'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m200[[39m, write_bytes: ^[[33m200000[[39m }, { bucket: ^{[[32m'user045-bucket7'}[[39m, access_key: ^{[[32m'yRWyCTRoLnKFVNWUtgXw'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m196[[39m, write_bytes: ^[[33m196000[[39m }, { bucket: ^{[[32m'user039-bucket4'}[[39m, access_key: ^{[[32m'bqHNcDgCfmN5Iozj94Wa'}[[39m, read_count: ^[[33m0[[39m, read_bytes: ^[[33m0[[39m, write_count: ^[[33m141[[39m, write_bytes: ^[[33m141000[[39m } ] }
Apr 10 16:51:59 c83f2-dan10 [4003863]: [nsfs/4003863] ^{[[36m   [L0]}[[39m core.endpoint.endpoint:: Sending endpoint report: { group_name: ^{[[32m'default-endpoint-group'}[[39m, hostname: ^{[[32m'c83f2-dan10'}[[39m }

warp drops between 35-50 files per instance. For a total of 8,750-12,500 files dropped.

If this is a regression, please provide more details to justify this:

 

Steps to Reproduce:

1. Create account

2. Create buckets

3. use warp to upload objects

 

The exact date and time when the issue was observed, including timezone details:

 

Actual results:

 

 

Expected results:

All testcases of uploading object should pass.

Logs collected and log location:

Logs, traces and so on are copied to the new DUMPS server: /u/DUMPS/D.347191

Additional info: