Description of problem:

There are many failures in libkcapi self-test in FIPS mode. When FIPS mode is disabled the all tests pass. Self-test is actually testing kcapi and hence it might indicate problems in Kernel Crypto API itself.

Version-Release number of selected component (if applicable):

libkcapi-1.3.1-3.el9

How reproducible:

100% when FIPS mode is enabled.

Steps to Reproduce:

1. Install likcapi-tests (from the buildroot repository).

2. Disable 3DES cases

1. sed -i 's/HASHEXEC\="1 2/HASHEXEC\="2/g' /usr/libexec/libkcapi/test.sh
2. sed -i 's/SYMEXEC\="1 2 3 4 5 6 7/SYMEXEC\="1 2 3/g' /usr/libexec/libkcapi/test.sh

3. Execute all tests:

1. /usr/libexec/libkcapi/test.sh
2. /usr/libexec/libkcapi/kcapi-enc-test.sh
3. /usr/libexec/libkcapi/kcapi-enc-test-large.sh
4. /usr/libexec/libkcapi/kcapi-dgst-test.sh
5. /usr/libexec/libkcapi/hasher-test.sh
6. /usr/libexec/libkcapi/kcapi-convenience.sh
7. /usr/libexec/libkcapi/kcapi-fuzz-test.sh

Actual results:

1. /usr/libexec/libkcapi/test.sh
   ...
   <fail>

1. /usr/libexec/libkcapi/kcapi-enc-test.sh
   ...
   Number of failures: 60

1. /usr/libexec/libkcapi/kcapi-enc-test-large.sh
   Number of failures: 0

1. /usr/libexec/libkcapi/kcapi-dgst-test.sh
   ...
   Number of failures: 64

1. /usr/libexec/libkcapi/hasher-test.sh
   ...
   Number of failures: 8

1. /usr/libexec/libkcapi/kcapi-convenience.sh
   ...
   Number of failures: 1

1. /usr/libexec/libkcapi/kcapi-fuzz-test.sh
   ...
   Number of failures: 0

Expected results:

All tests should pass.

Additional info:

The first test (test.sh) fails because is uses too short keys for SHA tests (for some reason RHEL-9 kernel needs longer keys in FIPS than in RHEL-8 FIPS or RHEL-9 non-FIPS):

1. strace /usr/libexec/libkcapi/kcapi -m -x 8 -c 'hmac(sha1)' -k 73616c74 -p "70617373776f7264" -d 1 -b 20
   ...
   setsockopt(3, SOL_ALG, ALG_SET_KEY, "password", 8) = -1 EINVAL (Invalid argument)
   ...

With longer password it will work (worked with 'RedHatEnterpriseLinux'). However, the other tests fails for different reasons I had no time to investigate.