Goal

As a user I would like to enable FIPS mode in my OSP17.1/rhel9.2 environment and set a bgp password that uses a compliant algorithm (not MD5).

Right now if one enables FIPS the password is ignored because:
2024/10/21 03:13:23 BGP: [KTTWK-0CPJ7][EC 100663303] sockopt_tcp_signature: setsockopt(21): Cannot allocate memory
2024/10/21 03:13:23 BGP: [NWGVJ-FEW9F][EC 33554495] Unable to set TCP MD5 option on socket for peer fe80::7060:4d06:5c2a:3196 (sock=21): Cannot allocate memory

Other vendors (Cisco/Juniper) seem to support additional algorithms:

https://community.cisco.com/t5/networking-knowledge-base/configuring-authentication-for-bgp/ta-p/3108287

https://www.juniper.net/documentation/us/en/software/junos/bgp/topics/topic-map/bgp_security.html#id-example-configuring-router-authentication-for-bgp

It would be great if we had similar capabilities in frr.

Acceptance criteria

• enable FIPS in rhel9.x
• set appropriate bgp password and algorithm
• verify that bgp sessions are established and there are no errors in the logs related to passwords or similar