-
Bug
-
Resolution: Done
-
Major
-
None
-
None
-
None
Recently we enabled barbican in NextGen deployment to verify glance image signature verification but it is not working as expected. While uploading the signed image we got "HTTP 400 with unable to retrieve certificate UUID for the image" error in return with below stacktrace/logs in glance-api log file.
2024-01-15 10:37:55.664 1065621 ERROR castellan.key_manager.barbican_key_manager [None req-ee3c6146-c695-4486-aca4-a329a7f56ef6 f94da4c1f0ef4c5b8a69160031af896d f2d3d6c78c6448a0ad359f606b35595b - - default default] Error creating Barbican client: Unable to establish connection to http://barbican-internal.openstack.svc:9311: HTTPConnectionPool(host='barbican-internal.openstack.svc', port=9311): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd90a5119d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED')): keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://barbican-internal.openstack.svc:9311: HTTPConnectionPool(host='barbican-internal.openstack.svc', port=9311): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd90a5119d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED')) 2024-01-15 10:37:55.667 1065621 ERROR cursive.signature_utils [None req-ee3c6146-c695-4486-aca4-a329a7f56ef6 f94da4c1f0ef4c5b8a69160031af896d f2d3d6c78c6448a0ad359f606b35595b - - default default] Unable to retrieve certificate with ID 96bc3e29-8c65-4358-9a55-0a6cdc6b7709: Key manager error: Unable to establish connection to http://barbican-internal.openstack.svc:9311: HTTPConnectionPool(host='barbican-internal.openstack.svc', port=9311): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd90a5119d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED')): castellan.common.exception.KeyManagerError: Key manager error: Unable to establish connection to http://barbican-internal.openstack.svc:9311: HTTPConnectionPool(host='barbican-internal.openstack.svc', port=9311): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fd90a5119d0>: Failed to establish a new connection: [Errno 111] ECONNREFUSED')) 2024-01-15 10:37:55.669 1065621 ERROR glance.api.v2.image_data [None req-ee3c6146-c695-4486-aca4-a329a7f56ef6 f94da4c1f0ef4c5b8a69160031af896d f2d3d6c78c6448a0ad359f606b35595b - - default default] Signature verification failed for image cf29b441-26b4-447d-9bbf-6fc057d6ad9e: Signature verification for the image failed: Unable to retrieve certificate with ID: 96bc3e29-8c65-4358-9a55-0a6cdc6b7709.: cursive.exception.SignatureVerificationError: Signature verification for the image failed: Unable to retrieve certificate with ID: 96bc3e29-8c65-4358-9a55-0a6cdc6b7709.
We didn't notice any issue on barbican side which was returning "200 OK" response. Debugging further we found that the error "InvalidSignature" is raised from "_rsa_sig_verify()" of OpenSSL package without any valid information or stacktrace around it (/usr/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/rsa.py(286)_rsa_sig_verify()). Same was wrapped in cursive package with "SignatureVerificationError" and returned to glance.
This feature works well with devstack environment (based on Ubuntu22.04).
We noticed that we also have different `openssl` versions, but looks like it still works in devstack.
{code:bash}
NG environment version: (CentOS 9 stream)
OpenSSL 3.0.7 1 Nov 2022 (Library: OpenSSL 3.0.7 1 Nov 2022)
Devstack envrionment: (Ubuntu 22.04)
OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)
We are planning to test NG envrionment with OpenSSL 3.0.2 version to rule out possible version related error.
- links to
- mentioned on
