In FIPS mode, OpenSSL must not perform any cryptographic operations when rsa_keygen_pairwise_test fails

What were you trying to do that didn't work?

When rsa_keygen_pairwise_test in crypto/rsa/rsa_gen.c fails (e.g., if it is made to fail for testing purposes using gdb), according to ISO 19790 AS10.09 the module shall not perform "any" cryptographic operations or output data in error state.

However, EVP_DigestUpdate and EVP_EncryptUpdate still fail. I filed https://github.com/openssl/openssl/pull/22506 upstream for this, but upstream disagrees. We have asked for clarification from CMVP. While we wait for clarification to settle this, we should patch this downstream to avoid blocking our submission.

Please provide the package NVR for which bug is seen:

openssl-3.0.7-18.el9_2

How reproducible:

See FIPS-182

Expected results

EVP_DigestUpdate() and EVP_EncryptUpdate() should not work

Actual results

EVP_DigestUpdate() and EVP_EncryptUpdate() work