-
Bug
-
Resolution: Done
-
Normal
-
None
-
None
When running the tests in this file https://github.com/stackrox/stackrox/blob/master/pkg/clientconn/service_cert_fallback_verifier.go I found they kept failing.
One reason is as follows:
=== RUN TestManager/TestExtraCertIssuedInStackRoxNamespace tlsconfig: 2025/01/13 16:51:33.200903 tlsconfig.go:235: Warn: Internal TLS certificates are not valid for all cluster-internal DNS names due to deployment in alternative namespace, issuing ephemeral certificate with adequate DNS names {"namespace": "alt-ns", "internalDNSNames": ["central.alt-ns", "central.alt-ns.svc"]} pkg/mtls/certwatch: 2025/01/13 16:51:33.201519 certwatch.go:86: Error: Error watching TLS certificate directory "/run/secrets/stackrox.io/default-tls-cert": reading contents of directory /run/secrets/stackrox.io/default-tls-cert: open /run/secrets/stackrox.io/default-tls-cert: no such file or directory. Not updating TLS certificates! manager_impl_test.go:129: Error Trace: /Users/rtannenb/go/src/github.com/stackrox/stackrox/central/tlsconfig/manager_impl_test.go:129 /Users/rtannenb/go/src/github.com/stackrox/stackrox/central/tlsconfig/manager_impl_test.go:74 Error: Should be in error chain: expected: %!q(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference) in chain: "x509: “my-central.example.org” certificate is not standards compliant" Test: TestManager/TestExtraCertIssuedInStackRoxNamespace manager_impl_test.go:129: Error Trace: /Users/rtannenb/go/src/github.com/stackrox/stackrox/central/tlsconfig/manager_impl_test.go:129 /Users/rtannenb/go/src/github.com/stackrox/stackrox/central/tlsconfig/manager_impl_test.go:74 Error: Should be in error chain: expected: %!q(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference) in chain: "x509: “my-central.example.org” certificate is not standards compliant" Test: TestManager/TestExtraCertIssuedInStackRoxNamespace --- FAIL: TestManager/TestExtraCertIssuedInStackRoxNamespace (0.01s)
Ignoring the panic message, we can see this is because of Apple's new-ish policy of labeling any certs valid for longer than 398 days are invalid.
Once that's resolved, I still get the following:
=== RUN TestManager/TestExtraCertIssuedInStackRoxNamespace tlsconfig: 2025/01/13 16:52:54.307789 tlsconfig.go:235: Warn: Internal TLS certificates are not valid for all cluster-internal DNS names due to deployment in alternative namespace, issuing ephemeral certificate with adequate DNS names {"namespace": "alt-ns", "internalDNSNames": ["central.alt-ns", "central.alt-ns.svc"]} pkg/mtls/certwatch: 2025/01/13 16:52:54.308462 certwatch.go:86: Error: Error watching TLS certificate directory "/run/secrets/stackrox.io/default-tls-cert": reading contents of directory /run/secrets/stackrox.io/default-tls-cert: open /run/secrets/stackrox.io/default-tls-cert: no such file or directory. Not updating TLS certificates! manager_impl_test.go:129: Error Trace: /Users/rtannenb/go/src/github.com/stackrox/stackrox/central/tlsconfig/manager_impl_test.go:129 /Users/rtannenb/go/src/github.com/stackrox/stackrox/central/tlsconfig/manager_impl_test.go:74 Error: Should be in error chain: expected: %!q(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference) in chain: "x509: certificate signed by unknown authority" Test: TestManager/TestExtraCertIssuedInStackRoxNamespace
I traced this all the way to https://github.com/golang/go/blob/go1.22.10/src/crypto/x509/verify.go#L770 which tells me we tried to verify the cert against system CAs. This fails because the system does not know about our self-signed cert which we use for the test.
To test this out some more, I modified VerifyPeerCertificates in https://github.com/stackrox/stackrox/blob/master/pkg/clientconn/service_cert_fallback_verifier.go as follows:
// PRETEND THIS IS NOT A SERVICE CERT SO WE CAN MOVE ON if systemVerifyErr == nil /* || !isServiceCert(leaf, v.subject) */ { return systemVerifyErr }
This now gives me
=== RUN TestManager/TestExtraCertIssuedInStackRoxNamespace tlsconfig: 2025/01/13 16:58:12.743847 tlsconfig.go:235: Warn: Internal TLS certificates are not valid for all cluster-internal DNS names due to deployment in alternative namespace, issuing ephemeral certificate with adequate DNS names {"namespace": "alt-ns", "internalDNSNames": ["central.alt-ns", "central.alt-ns.svc"]} pkg/mtls/certwatch: 2025/01/13 16:58:12.744476 certwatch.go:86: Error: Error watching TLS certificate directory "/run/secrets/stackrox.io/default-tls-cert": reading contents of directory /run/secrets/stackrox.io/default-tls-cert: open /run/secrets/stackrox.io/default-tls-cert: no such file or directory. Not updating TLS certificates! manager_impl_test.go:129: Error Trace: /Users/rtannenb/go/src/github.com/stackrox/stackrox/central/tlsconfig/manager_impl_test.go:129 /Users/rtannenb/go/src/github.com/stackrox/stackrox/central/tlsconfig/manager_impl_test.go:74 Error: Should be in error chain: expected: %!q(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference) in chain: "verifying Central certificate errors: [x509: certificate signed by unknown authority, x509: certificate is not valid for any names, but wanted to match central.stackrox]" Test: TestManager/TestExtraCertIssuedInStackRoxNamespace
This is also odd. Notice: "x509: certificate is not valid for any names, but wanted to match central.stackrox]" IS exactly what we'd see in x509.HostnameError message.
I'm not quite sure how to resolve it from here, so I am creating this ticket. I can resolve the macOS cert expiry limitations, but I'm hoping someone with more cert expertise can takeover from here.
- relates to
-
-
- Closed
-
