-
Bug
-
Resolution: Unresolved
-
Critical
-
None
-
1.37.0
-
None
-
None
-
False
-
-
False
-
-
With internal encryption enabled
spec:
config:
certmanager:
clusterLocalIssuerRef: |
kind: ClusterIssuer
name: knative-selfsigned-issuer
systemInternalIssuerRef: |
kind: ClusterIssuer
name: knative-selfsigned-issuer
features:
kubernetes.podspec-init-containers: Enabled
network:
cluster-local-domain-tls: Enabled
system-internal-tls: Enabled
The queue-proxy logs show a
> 2025-10-16T13:44:32.745982209Z 2025/10/16 13:44:32 http: TLS handshake error from 10.131.0.2:56672: client sent an HTTP request to an HTTPS server
error.
The problem seems to be that the pre-stop hook that is added by knative to the user-container
lifecycle:
preStop:
httpGet:
path: /wait-for-drain
port: 8022
scheme: HTTP
fails, as the queue-proxy admin port is HTTPS, not HTTP.
Presumably, that causes the /wait-for-drain pre-stop hook to fail immediately, the user-container is then immediately terminated (instead of waiting for the draining), which would explain the dial tcp 127.0.0.1:8080: connect: connection refused error in the queue-proxy reverseproxy just after TERM is received.
2025-10-16T13:44:32.745982209Z 2025/10/16 13:44:32 http: TLS handshake error from 10.131.0.2:56672: client sent an HTTP request to an HTTPS server
2025-10-16T13:44:32.746618297Z {"severity":"INFO","timestamp":"2025-10-16T13:44:32.746541915Z","logger":"queueproxy","caller":"sharedmain/main.go:305","message":"Received TERM signal, attempting to gracefully shutdown servers.","commit":"3526d22-dirty","knative.dev/key":"default/load-test-always-00001","knative.dev/pod":"load-test-always-00001-deployment-79645cc79f-xw4gt"}
2025-10-16T13:44:32.746641758Z {"severity":"INFO","timestamp":"2025-10-16T13:44:32.746608833Z","logger":"queueproxy","caller":"sharedmain/main.go:306","message":"Sleeping 30s to allow K8s propagation of non-ready state","commit":"3526d22-dirty","knative.dev/key":"default/load-test-always-00001","knative.dev/pod":"load-test-always-00001-deployment-79645cc79f-xw4gt"}
2025-10-16T13:44:32.750895739Z {"severity":"ERROR","timestamp":"2025-10-16T13:44:32.750768505Z","logger":"queueproxy","caller":"network/error_handler.go:33","message":"error reverse proxying request; sockstat: sockets: used 69\nTCP: inuse 17 orphan 0 tw 22 alloc 367 mem 564\nUDP: inuse 0 mem 512\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n","commit":"3526d22-dirty","knative.dev/key":"default/load-test-always-00001","knative.dev/pod":"load-test-always-00001-deployment-79645cc79f-xw4gt","error":"dial tcp 127.0.0.1:8080: connect: connection refused","stacktrace":"knative.dev/pkg/network.ErrorHandler.func1\n\t/workspace/vendor/knative.dev/pkg/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\t/usr/lib/golang/src/net/http/httputil/reverseproxy.go:486\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\t/workspace/pkg/queue/request_metric.go:201\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\t/workspace/pkg/queue/handler.go:76\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2220\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\t/workspace/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2220\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\t/workspace/pkg/http/handler/timeout.go:118"}
2025-10-16T13:44:32.751656821Z {"severity":"ERROR","timestamp":"2025-10-16T13:44:32.75157073Z","logger":"queueproxy","caller":"network/error_handler.go:33","message":"error reverse proxying request; sockstat: sockets: used 69\nTCP: inuse 17 orphan 0 tw 22 alloc 367 mem 564\nUDP: inuse 0 mem 512\nUDPLITE: inuse 0\nRAW: inuse 0\nFRAG: inuse 0 memory 0\n","commit":"3526d22-dirty","knative.dev/key":"default/load-test-always-00001","knative.dev/pod":"load-test-always-00001-deployment-79645cc79f-xw4gt","error":"dial tcp 127.0.0.1:8080: connect: connection refused","stacktrace":"knative.dev/pkg/network.ErrorHandler.func1\n\t/workspace/vendor/knative.dev/pkg/network/error_handler.go:33\nnet/http/httputil.(*ReverseProxy).ServeHTTP\n\t/usr/lib/golang/src/net/http/httputil/reverseproxy.go:486\nknative.dev/serving/pkg/queue.(*appRequestMetricsHandler).ServeHTTP\n\t/workspace/pkg/queue/request_metric.go:201\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ProxyHandler.func3\n\t/workspace/pkg/queue/handler.go:76\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2220\nknative.dev/serving/pkg/queue/sharedmain.mainHandler.ForwardedShimHandler.func4\n\t/workspace/pkg/queue/forwarded_shim.go:54\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/lib/golang/src/net/http/server.go:2220\nknative.dev/serving/pkg/http/handler.(*timeoutHandler).ServeHTTP.func4\n\t/workspace/pkg/http/handler/timeout.go:118"}
The problem can be reproduced on modified knative serving performance benchmark "load-test" (modified to use https requests)
- blocks
-
SRVKS-1130 GA for All hops encrypted amongst internal and external services
-
- In Progress
-
