Description of problem - Provide a detailed description of the issue encountered, including logs/command-output snippets and screenshots if the issue is observed in the UI:

Setting a replication policy with `"sync_versions":true` and `"sync_deletions":true` between two versioning-enabled buckets is expected to replicate both versions and delete markers. However, while versions are replicated as expected, delete markers are never replicated.

The result is that objects that deletions on the source bucket are not getting replicated to the target bucket as expected.

The OCP platform infrastructure and deployment type (AWS, Bare Metal, VMware, etc. Please clarify if it is platform agnostic deployment), (IPI/UPI):

IBM Cloud - IPI

The ODF deployment type (Internal, External, Internal-Attached (LSO), Multicluster, DR, Provider, etc):

Internal

The version of all relevant components (OCP, ODF, RHCS, ACM whichever is applicable):

OCP: 4.18.0-0.nightly-2024-11-15-113437
ODF: 4.18.0-53
noobaa core: 5.18.0-aaeeb19
noobaa operator: 5.18.0-b62bb7eb535d16c2adb956540999d21c5ed8d86a
noobaa cli: 5.18.0

Does this issue impact your ability to continue to work with the product?

This fails the happy path testing of https://issues.redhat.com/browse/RHSTOR-4129, as instructed by lmauda:

Happy path validation:

- Set replication with version on between 2 buckets with version enabled.
- Write objects with versions
- delete some object version


see that all the versions are replicated.

  

Is there any workaround available to the best of your knowledge?

Nope

Can this issue be reproduced? If so, please provide the hit rate

100%

Can this issue be reproduced from the UI?

N/A

If this is a regression, please provide more details to justify this:

No - this is a new feature

Steps to Reproduce:

1. Create two buckets via OBCs:

$ mcg-cli obc create --exact source-bucket
...
$ mcg-cli obc create --exact target-bucket
...

2. Enable versioning on both:

$ my_s3api_alias put-bucket-versioning --bucket source-bucket --versioning-configuration Status=Enabled
$ my_s3api_alias put-bucket-versioning --bucket target-bucket --versioning-configuration Status=Enabled

3. Patch a replication policy with versioning and deletion syncs enabled on the source bucket's OBC:

$ oc patch obc source-bucket --type=merge -p '{"spec": {"additionalConfig": {"replicationPolicy": "{\"rules\":[{\"rule_id\":\"replication-rule-c\",\"destination_bucket\":\"target-bucket\", \"sync_versions\": true, \"sync_deletions\": true}]}"}}}' 

4. Write multiple versions of the same object to the source bucket:

$ echo "a" | my_s3_alias cp - s3://source-bucket/test_obj.txt
$ echo "b" | my_s3_alias cp - s3://source-bucket/test_obj.txt 
$ echo "c" | my_s3_alias cp - s3://source-bucket/test_obj.txt  

5. Wait a few minutes and verify that the versions were replicated to the target bucket by comparing the ETags for the object:

$ my_s3api_alias list-object-versions --bucket source-bucket --prefix test_obj.txt | jq -r ".Versions[].ETag"
"5536fc9343e9ab0aca071354084dd3cc"
"d6d88f2e50080b9602da53dac1102762"
"c576ec4297a7bdacc878e0061192441e" 

$ my_s3api_alias list-object-versions --bucket target-bucket --prefix test_obj_c.txt | jq -r ".Versions[].ETag"
"5536fc9343e9ab0aca071354084dd3cc"
"d6d88f2e50080b9602da53dac1102762"
"c576ec4297a7bdacc878e0061192441e" 

6. Delete the object on the source bucket and confirm the creation of the delete marker:

$ my_s3_alias rm s3://source-bucket/test_obj.txt 
$ $ my_s3api_alias list-object-versions --bucket source-bucket --prefix test_obj_c.txt | jq -r ".DeleteMarkers"
[
  {
    "Owner": {
      "DisplayName": "operator@noobaa.io",
      "ID": "6739d5cce8737a001da1925f"
    },
    "Key": "test_obj.txt",
    "VersionId": "nbver-25",
    "IsLatest": true,
    "LastModified": "2024-11-19T09:39:59+00:00"
  }
]

7. Wait a few minutes and check whether the delete marker was replicated to the target bucket, and whether the object appears as deleted on it: 

$ my_s3api_alias list-object-versions --bucket target-bucket --prefix test_obj.txt | jq -r ".DeleteMarkers" 
null

$ my_s3_alias ls s3://target-bucket/test_obj.txt
2024-11-19 11:38:46          4 test_obj.txt

The exact date and time when the issue was observed, including timezone details:

November 19th, 14:35

Actual results:

The versions get replicated as expected, but at step #7 the delete marker does not get replicated to the target bucket, and the object is still listed on the target bucket.

Expected results:

The delete marker should also get replicated if isLatest=true still applies to it

Logs collected and log location:

must-gather logs: https://drive.google.com/drive/folders/1StvMIgo5amUhtr5Aa5tfixPBNRu7qBoB?usp=drive_link 

Note that the NOOBAA_LOG_LEVEL was set to "all" before their creation.

Additional info: