Description of problem (please be detailed as possible and provide log
snippests):

A problem was encountered going between setting ceph messenger v2 from defaults to disabling secure mode completely and then enabling secure mode. A workaround is specified below on how to get around this problem. Much time was spent trying to figure this out. I could not find anywhere in documentation showing the proper method to go between these three modes.

This should be documented on proper method to switch between ceph msgr v2 secure mode enabled vs. disabled. It was not found in current documentation.

Here are the scripts to enable/disable msgv2 showing when the error occurred. It is not recommended to use this individual "ceph config set .." method to control mode settings.

Enable msgr v2 secure mode:

#!/bin/bash
set -x
ceph config set osd ms_bind_msgr1 false
ceph config set mgr ms_bind_msgr1 false
ceph config set mon ms_bind_msgr1 false
ceph config set osd ms_bind_msgr2 true
ceph config set mgr ms_bind_msgr2 true
ceph config set mon ms_bind_msgr2 true
ceph config set mon ms_cluster_mode "secure"
ceph config set mgr ms_cluster_mode "secure"
ceph config set osd ms_cluster_mode "secure"
ceph config set osd ms_service_mode "secure"
ceph config set mgr ms_service_mode "secure"
ceph config set mon ms_service_mode "secure"
ceph config set mon ms_client_mode "secure"
ceph config set mgr ms_client_mode "secure"
ceph config set osd ms_client_mode "secure"
ceph config set mon ms_mon_cluster_mode "secure"
ceph config set mgr ms_mon_cluster_mode "secure"
ceph config set osd ms_mon_cluster_mode "secure"
ceph config set mgr ms_mon_service_mode "secure"
ceph config set mon ms_mon_service_mode "secure"
ceph config set osd ms_mon_service_mode "secure"
ceph config set mgr ms_mon_client_mode "secure"
ceph config set mon ms_mon_client_mode "secure"
ceph config set osd ms_mon_client_mode "secure"

ceph config set client ms_bind_msgr1 false
ceph config set client ms_bind_msgr2 true
ceph config set client ms_cluster_mode "secure"
ceph config set client ms_service_mode "secure"
ceph config set client ms_client_mode "secure"
ceph config set client ms_mon_cluster_mode "secure"
ceph config set client ms_mon_service_mode "secure"
ceph config set client ms_mon_client_mode "secure"

Disable msgr v2 secure mode:

#!/bin/bash
set -x
ceph config set osd ms_bind_msgr1 false
ceph config set mgr ms_bind_msgr1 false
ceph config set mon ms_bind_msgr1 false
ceph config set osd ms_bind_msgr2 true
ceph config set mgr ms_bind_msgr2 true
ceph config set mon ms_bind_msgr2 true
ceph config set mon ms_cluster_mode "crc"
ceph config set mgr ms_cluster_mode "crc"
ceph config set osd ms_cluster_mode "crc"
ceph config set osd ms_service_mode "crc"
ceph config set mgr ms_service_mode "crc"
ceph config set mon ms_service_mode "crc"
ceph config set mon ms_client_mode "crc"
ceph config set mgr ms_client_mode "crc"
ceph config set osd ms_client_mode "crc"
ceph config set mon ms_mon_cluster_mode "crc"
ceph config set mgr ms_mon_cluster_mode "crc"
ceph config set osd ms_mon_cluster_mode "crc"
ceph config set mgr ms_mon_service_mode "crc"
ceph config set mon ms_mon_service_mode "crc"
ceph config set osd ms_mon_service_mode "crc"
ceph config set mgr ms_mon_client_mode "crc"
ceph config set mon ms_mon_client_mode "crc"
ceph config set osd ms_mon_client_mode "crc"

ceph config set client ms_bind_msgr1 false
ceph config set client ms_bind_msgr2 true
ceph config set client ms_cluster_mode "crc"
ceph config set client ms_service_mode "crc"
ceph config set client ms_client_mode "crc"
ceph config set client ms_mon_cluster_mode "crc"
ceph config set client ms_mon_service_mode "crc"
ceph config set client ms_mon_client_mode "crc"

First, the disable script was run and it was successful.
Some tests were run.

The error showed when running the enable script and it occurred at this line below.

++ ceph config set osd ms_bind_msgr1 false
++ ceph config set mgr ms_bind_msgr1 false
++ ceph config set mon ms_bind_msgr1 false
++ ceph config set osd ms_bind_msgr2 true
++ ceph config set mgr ms_bind_msgr2 true
++ ceph config set mon ms_bind_msgr2 true
++ ceph config set mon ms_cluster_mode secure
++ ceph config set mgr ms_cluster_mode secure
++ ceph config set osd ms_cluster_mode secure
++ ceph config set osd ms_service_mode secure
++ ceph config set mgr ms_service_mode secure
++ ceph config set mon ms_service_mode secure
++ ceph config set mon ms_client_mode secure
++ ceph config set mgr ms_client_mode secure
++ ceph config set osd ms_client_mode secure
++ ceph config set mon ms_mon_cluster_mode secure
++ ceph config set mgr ms_mon_cluster_mode secure
++ ceph config set osd ms_mon_cluster_mode secure
++ ceph config set mgr ms_mon_service_mode secure
++ ceph config set mon ms_mon_service_mode secure
++ ceph config set osd ms_mon_service_mode secure
2023-01-11T22:12:18.188+0000 7f22e359e700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-01-11T22:12:18.188+0000 7f22e259c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-01-11T22:12:18.188+0000 7f22e2d9d700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-01-11T22:12:18.188+0000 7f22e9221700 0 librados: client.admin authentication error (13) Permission denied
[errno 13] RADOS permission denied (error connecting to the cluster)
++ ceph config set mgr ms_mon_client_mode secure
2023-01-11T22:12:18.293+0000 7ff523fff700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-01-11T22:12:18.294+0000 7ff528e25700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-01-11T22:12:18.294+0000 7ff5237fe700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-01-11T22:12:18.294+0000 7ff52a888700 0 librados: client.admin authentication error (13) Permission denied
[errno 13] RADOS permission denied (error connecting to the cluster)
++ ceph config set mon ms_mon_client_mode secure
2023-01-11T22:12:18.397+0000 7f23ea321700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-01-11T22:12:18.398+0000 7f23e9b20700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-01-11T22:12:18.398+0000 7f23eab22700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2,1]
2023-01-11T22:12:18.398+0000 7f23ec585700 0 librados: client.admin authentication error (13) Permission denied
[errno 13] RADOS permission denied (error connecting to the cluster)

At this point the system became unusable to determine the values or to set to any other values for these variables (ceph config set/get)

This workaround was found to get around this problem.

3 config files were made, i.e.:

sec.default:

[global]
ms_bind_msgr1=true
ms_bind_msgr2=true
ms_cluster_mode="crc secure"
ms_service_mode="crc secure"
ms_client_mode="crc secure"
ms_mon_cluster_mode="crc secure"
ms_mon_service_mode="crc secure"
ms_mon_client_mode="crc secure"

sec.off:

[global]
ms_bind_msgr1=true
ms_bind_msgr2=true
ms_cluster_mode="crc"
ms_service_mode="crc"
ms_client_mode="crc"
ms_mon_cluster_mode="crc"
ms_mon_service_mode="crc"
ms_mon_client_mode="crc"

sec.on:

[global]
ms_bind_msgr1=false
ms_bind_msgr2=true
ms_cluster_mode="secure"
ms_service_mode="secure"
ms_client_mode="secure"
ms_mon_cluster_mode="secure"
ms_mon_service_mode="secure"
ms_mon_client_mode="secure"

To go between settings, first run

#!/bin/bash
set -x
ceph config rm global ms_bind_msgr1
ceph config rm global ms_bind_msgr2
ceph config rm global ms_cluster_mode
ceph config rm global ms_service_mode
ceph config rm global ms_client_mode
ceph config rm global ms_mon_cluster_mode
ceph config rm global ms_mon_service_mode
ceph config rm global ms_mon_client_mode

Then:

ceph config assimilate-conf -i my.conf.file, i.e.
ceph config assimilate-conf -i sec.default

Version of all relevant components (if applicable):

OCP 4.12.0.rc4
ODF 4.12.0-145

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

No

Is there any workaround available to the best of your knowledge?

Yes - described in description

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?

3

Can this issue reproducible?

yes

Can this issue reproduce from the UI?

N/A

If this is a regression, please provide more details to justify this:

Steps to Reproduce:
1. Run indiv ceph config set commands to disable secure mode
2. Run indiv ceph config set commands to enable secure mode
3. Error shows

Actual results:

Error

Expected results:

Should change between modes

Additional info: