Currently, OpenSSL engine support (openssl-pkcs11) on RHEL 9 does not support the RSA padding mode (RSA_PKCS1_WITH_TLS_PADDING) required for Ex=RSA ciphers in TLS 1.2, causing negotiation failures. This issue is resolved in RHEL 10 by replacing the deprecated engine with the new PKCS#11 provider module.

We request backporting this PKCS#11 provider fix to RHEL 9 to enable customers to use Ex=RSA ciphers seamlessly with hardware security modules (HSMs) and PKCS#11 engines without requiring immediate migration to RHEL 10.

What were you trying to do that didn't work?

Attempt to establish a TLS 1.2 connection using OpenSSL with the openssl-pkcs11 engine on RHEL 9, specifically using Ex=RSA ciphers (e.g., AES256-SHA256). The connection fails due to unsupported RSA padding mode in the engine.

4047B65C567F0000:error:02000090:rsa routines:pkey_rsa_ctrl:illegal or unsupported padding mode:crypto/rsa/rsa_pmeth.c:478:
4047B65C567F0000:error:03000093:digital envelope routines:evp_pkey_ctx_ctrl_int:command not supported:crypto/evp/pmeth_lib.c:1360:
4047B65C567F0000:error:0A000093:SSL routines:tls_process_cke_rsa:decryption failed:ssl/statem/statem_srvr.c:3032:

What is the impact of this issue to you?

Inability to use hardware security modules (HSMs) with Ex=RSA ciphers on RHEL 9.
TLS handshake failures causing service disruptions or inability to secure communications.
Migration to RHEL 10 is required to resolve the issue, which may not be immediately feasible.

Please provide the package NVR for which the bug is seen:

openssl-3.5.1-2.el9.x86_64
openssl-pkcs11-0.4.11-9.el9.x86_64

How reproducible is this bug?:

Consistently reproducible on RHEL 9 systems when using the openssl-pkcs11 engine with Ex=RSA ciphers during TLS handshak

Steps to reproduce

Configure OpenSSL with openssl-pkcs11 engine on RHEL 9.
Generate RSA keys and certificates.
Run OpenSSL server with AES256-SHA256 cipher and engine enabled.
Connect with OpenSSL client using TLS 1.2 and the same cipher.
Observe TLS handshake failure and error logs related to RSA padding.

Expected results
Successful TLS 1.2 handshake using AES256-SHA256 cipher with the PKCS#11 engine, enabling secure communication with HSM-backed keys.

Actual results
TLS handshake fails with errors such as:

4047B65C567F0000:error:02000090:rsa routines:pkey_rsa_ctrl:illegal or unsupported padding mode:crypto/rsa/rsa_pmeth.c:478:
4047B65C567F0000:error:03000093:digital envelope routines:evp_pkey_ctx_ctrl_int:command not supported:crypto/evp/pmeth_lib.c:1360:
4047B65C567F0000:error:0A000093:SSL routines:tls_process_cke_rsa:decryption failed:ssl/statem/statem_srvr.c:3032: