osus deployment fails verifying registry certificate

4.10.z

Always

1.  Deploy and OSUS graph pod [1] using a local Quay registry that has a valid certificate that is provided from LetsEncrypt

2.  graph update trigger fails stating that it is unable to get the issuer certificate

[1] https://access.redhat.com/documentation/en-us/openshift_container_platform/4.10/html-single/updating_clusters/index#update-service-create-service 

[2023-10-14T15:32:58Z DEBUG graph_builder::graph] graph update triggered
[2023-10-14T15:32:58Z TRACE cincinnati::plugins] Running next plugin 'release-scrape-dockerv2'
[2023-10-14T15:32:58Z ERROR graph_builder::graph] failed to fetch all release metadata from mirror.syangsao.net:8443/ocp4/openshift/release
[2023-10-14T15:32:58Z ERROR graph_builder::graph] http transport error: error sending request for url (https://mirror.syangsao.net:8443/v2/): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:32:58Z ERROR graph_builder::graph] error sending request for url (https://mirror.syangsao.net:8443/v2/): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:32:58Z ERROR graph_builder::graph] error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:32:58Z ERROR graph_builder::graph] error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:32:58Z ERROR graph_builder::graph] error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:

Certificate validation should work, especially for a trusted certificate provided by LetsEncrypt

oc logs osus-test-5bcf488987-t64w9
Defaulted container "graph-builder" out of: graph-builder, policy-engine, graph-data (init)
[2023-10-14T15:22:58Z DEBUG graph_builder] application settings:
    AppSettings {
        address: ::,
        credentials_path: None,
        mandatory_client_parameters: {},
        manifestref_key: "io.openshift.upgrades.graph.release.manifestref",
        path_prefix: "",
        pause_secs: 300s,
        scrape_timeout_secs: None,
        port: 8080,
        registry: "quay.io",
        repository: "openshift-release-dev/ocp-release",
        status_address: ::,
        status_port: 9080,
        verbosity: Trace,
        fetch_concurrency: 16,
        metrics_required: {
            "graph_upstream_raw_releases",
        },
        plugin_settings: [
            ReleaseScrapeDockerv2Settings {
                registry: "mirror.syangsao.net:8443",
                repository: "ocp4/openshift/release",
                manifestref_key: "io.openshift.upgrades.graph.release.manifestref",
                fetch_concurrency: 16,
                username: None,
                password: None,
                credentials_path: Some(
                    "/var/lib/cincinnati/registry-credentials/.dockerconfigjson",
                ),
            },
            OpenshiftSecondaryMetadataParserSettings {
                data_directory: "/var/lib/cincinnati/graph-data",
                key_prefix: "io.openshift.upgrades.graph",
                default_arch: "amd64",
                disallowed_errors: {},
            },
            EdgeAddRemovePlugin {
                key_prefix: "io.openshift.upgrades.graph",
                remove_all_edges_value: "*",
                remove_consumed_metadata: false,
                include_conditional_edges: true,
            },
        ],
        tracing_endpoint: None,
    }
[2023-10-14T15:22:58Z DEBUG graph_builder::graph] graph update triggered
[2023-10-14T15:22:58Z TRACE cincinnati::plugins] Running next plugin 'release-scrape-dockerv2'
[2023-10-14T15:22:58Z ERROR graph_builder::graph] failed to fetch all release metadata from mirror.syangsao.net:8443/ocp4/openshift/release
[2023-10-14T15:22:58Z ERROR graph_builder::graph] http transport error: error sending request for url (https://mirror.syangsao.net:8443/v2/): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:22:58Z ERROR graph_builder::graph] error sending request for url (https://mirror.syangsao.net:8443/v2/): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:22:58Z ERROR graph_builder::graph] error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:22:58Z ERROR graph_builder::graph] error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:22:58Z ERROR graph_builder::graph] error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:
[2023-10-14T15:27:58Z DEBUG graph_builder::graph] graph update triggered
[2023-10-14T15:27:58Z TRACE cincinnati::plugins] Running next plugin 'release-scrape-dockerv2'
[2023-10-14T15:27:58Z ERROR graph_builder::graph] failed to fetch all release metadata from mirror.syangsao.net:8443/ocp4/openshift/release
[2023-10-14T15:27:58Z ERROR graph_builder::graph] http transport error: error sending request for url (https://mirror.syangsao.net:8443/v2/): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:27:58Z ERROR graph_builder::graph] error sending request for url (https://mirror.syangsao.net:8443/v2/): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:27:58Z ERROR graph_builder::graph] error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:27:58Z ERROR graph_builder::graph] error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:27:58Z ERROR graph_builder::graph] error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:
[2023-10-14T15:32:58Z DEBUG graph_builder::graph] graph update triggered
[2023-10-14T15:32:58Z TRACE cincinnati::plugins] Running next plugin 'release-scrape-dockerv2'
[2023-10-14T15:32:58Z ERROR graph_builder::graph] failed to fetch all release metadata from mirror.syangsao.net:8443/ocp4/openshift/release
[2023-10-14T15:32:58Z ERROR graph_builder::graph] http transport error: error sending request for url (https://mirror.syangsao.net:8443/v2/): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:32:58Z ERROR graph_builder::graph] error sending request for url (https://mirror.syangsao.net:8443/v2/): error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:32:58Z ERROR graph_builder::graph] error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:32:58Z ERROR graph_builder::graph] error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915: (unable to get issuer certificate)
[2023-10-14T15:32:58Z ERROR graph_builder::graph] error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1915:

# Certificate is valid

curl -k -v -s https://mirror.syangsao.net:8443/v2/
*   Trying 192.168.40.15:8443...
* Connected to mirror.syangsao.net (192.168.40.15) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=mirror.syangsao.net
*  start date: Jul 31 00:00:00 2023 GMT
*  expire date: Oct 29 23:59:59 2023 GMT
*  issuer: C=AT; O=ZeroSSL; CN=ZeroSSL ECC Domain Secure Site CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* Using Stream ID: 1 (easy handle 0x556164192850)
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET /v2/ HTTP/2
> Host: mirror.syangsao.net:8443
> user-agent: curl/7.76.1
> accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
< HTTP/2 401
< server: nginx/1.20.1
< date: Sat, 14 Oct 2023 15:39:07 GMT
< content-type: text/html; charset=utf-8
< content-length: 4
< www-authenticate: Bearer realm="https://mirror.syangsao.net:8443/v2/auth",service="mirror.syangsao.net:8443"
< docker-distribution-api-version: registry/2.0
<
* Connection #0 to host mirror.syangsao.net left intact
true