-
Bug
-
Resolution: Done
-
Critical
-
1.25.0
-
None
When KnativeServing is configured with:
spec:
config:
network:
internal-encryption: 'true'
openshift-ingress-default-certificate: ut-1-25-75
then any newly deployed Knative Service is "Uninitialized".
The ksvc error message is: "Waiting for load balancer to be ready"
The knative-openshift-ingress throws:
"Failed to generate routes from ingress unable to find Ingress LoadBalancer with DomainInternal set"
Note: The config.network configuration is there from the beginning. It's NOT added later via patch.
The following configuration is used for the custom ingress certificate:
--- apiVersion: v1 kind: Secret type: kubernetes.io/tls data: tls.crt: XXX tls.key: YYY metadata: name: ut-1-25-75 namespace: openshift-ingress --- apiVersion: operator.openshift.io/v1 kind: IngressController metadata: name: default namespace: openshift-ingress-operator spec: replicas: 2 defaultCertificate: name: ut-1-25-75
Restarting the 3-scale-kourier-gateway Pods helps and the knative service becomes available again. But deploying a new ksvc again leads to the same state and the restart is required again.
Relevant part of the kourier gateway logs after creating a knative service:
[2022-10-11 11:05:02.496][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.497][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.497][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.497][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.497][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.497][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.497][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.497][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.499][1][info][upstream] [external/envoy/source/common/upstream/cds_api_helper.cc:30] cds: add 2 cluster(s), remove 2 cluster(s) [2022-10-11 11:05:02.499][1][info][upstream] [external/envoy/source/common/upstream/cds_api_helper.cc:67] cds: added/updated 0 cluster(s), skipped 2 unmodified cluster(s) [2022-10-11 11:05:02.576][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.576][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.577][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.577][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.577][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.577][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.577][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.577][1][warning][misc] [external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.route.v3.HeaderMatcher Using deprecated option 'envoy.config.route.v3.HeaderMatcher.exact_match' from file route_components.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override. [2022-10-11 11:05:02.579][1][info][upstream] [external/envoy/source/common/upstream/cds_api_helper.cc:30] cds: add 2 cluster(s), remove 2 cluster(s) [2022-10-11 11:05:02.579][1][info][upstream] [external/envoy/source/common/upstream/cds_api_helper.cc:67] cds: added/updated 0 cluster(s), skipped 2 unmodified cluster(s) [2022-10-11T11:05:02.699Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "3564fa8a-15f1-4434-a360-e4cc82d5f06a" "hello.serving-tests2.svc" "10.130.2.29:8112" [2022-10-11T11:05:02.698Z] "GET /healthz HTTP/1.1" 503 UF 0 209 5 - "-" "Knative-Ingress-Probe" "598887ef-8df1-49b0-b1ce-f5da3597606d" "hello.serving-tests2.svc.cluster.local" "10.130.2.29:8112" [2022-10-11T11:05:02.699Z] "GET /healthz HTTP/1.1" 503 UF 0 209 8 - "-" "Knative-Ingress-Probe" "f3279e44-4c2c-46aa-a455-80f70fb1a70e" "hello-serving-tests2.apps.ut-1-25-75.ci-aws.rhocf-dev.net" "10.129.2.66:8112" [2022-10-11T11:05:02.698Z] "GET /healthz HTTP/1.1" 503 UF 0 209 9 - "-" "Knative-Ingress-Probe" "b0e8cfef-d6a5-4b85-9b3c-e11326b79db1" "hello.serving-tests2" "10.129.2.66:8112" [2022-10-11T11:05:02.755Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "61d6134e-70e5-46ea-a42b-b1e2a487b4c5" "hello.serving-tests2.svc" "10.130.2.29:8112" [2022-10-11T11:05:02.756Z] "GET /healthz HTTP/1.1" 503 UF 0 209 3 - "-" "Knative-Ingress-Probe" "c6039605-8947-4e8c-8712-2656d1dd602d" "hello.serving-tests2.svc.cluster.local" "10.130.2.29:8112" [2022-10-11T11:05:02.760Z] "GET /healthz HTTP/1.1" 503 UF 0 209 5 - "-" "Knative-Ingress-Probe" "2650cd05-bc45-460d-9d93-1ec60ded918e" "hello-serving-tests2.apps.ut-1-25-75.ci-aws.rhocf-dev.net" "10.129.2.66:8112" [2022-10-11T11:05:02.769Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "731d801f-4406-4adb-9d8e-62a1be7388ba" "hello.serving-tests2" "10.130.2.29:8112" [2022-10-11T11:05:02.860Z] "GET /healthz HTTP/1.1" 503 UF 0 209 6 - "-" "Knative-Ingress-Probe" "d2b200ed-3601-473f-887c-c24dcebff514" "hello.serving-tests2.svc" "10.129.2.66:8112" [2022-10-11T11:05:02.871Z] "GET /ready HTTP/1.1" 200 - 0 5 0 - "-" "kube-probe/1.24" "0bb8cd65-2517-45ef-99e6-61933a3523f8" "internalkourier" "-" [2022-10-11T11:05:02.871Z] "GET /ready HTTP/1.1" 200 - 0 5 0 0 "-" "kube-probe/1.24" "0bb8cd65-2517-45ef-99e6-61933a3523f8" "internalkourier" "/tmp/envoy.admin" [2022-10-11T11:05:02.868Z] "GET /healthz HTTP/1.1" 503 UF 0 209 4 - "-" "Knative-Ingress-Probe" "5a5da88d-2a76-4720-b14b-a05c4a537ff2" "hello-serving-tests2.apps.ut-1-25-75.ci-aws.rhocf-dev.net" "10.130.2.29:8112" [2022-10-11T11:05:02.863Z] "GET /healthz HTTP/1.1" 503 UF 0 209 9 - "-" "Knative-Ingress-Probe" "4cb159fa-ee19-469c-9a31-a5c1fc9291f0" "hello.serving-tests2.svc.cluster.local" "10.129.2.66:8112" [2022-10-11T11:05:02.874Z] "GET /healthz HTTP/1.1" 503 UF 0 209 5 - "-" "Knative-Ingress-Probe" "08c73777-28be-4c1c-97c1-6e44bf3fd3d5" "hello.serving-tests2" "10.129.2.66:8112" [2022-10-11T11:05:03.070Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "c18af68f-4e2c-4536-94eb-484a4f6e1ba0" "hello.serving-tests2.svc" "10.130.2.29:8112" [2022-10-11T11:05:03.075Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "cf6115ea-a07c-4f01-84ea-b0bb5b7da4b9" "hello-serving-tests2.apps.ut-1-25-75.ci-aws.rhocf-dev.net" "10.130.2.29:8112" [2022-10-11T11:05:03.075Z] "GET /healthz HTTP/1.1" 503 UF 0 209 4 - "-" "Knative-Ingress-Probe" "85a73afb-9fed-4fa3-9957-c10f4fad94b3" "hello.serving-tests2.svc.cluster.local" "10.129.2.66:8112" [2022-10-11T11:05:03.083Z] "GET /healthz HTTP/1.1" 503 UF 0 209 4 - "-" "Knative-Ingress-Probe" "40e32075-09e9-4f01-827a-d52cbd25dcac" "hello.serving-tests2" "10.129.2.66:8112" [2022-10-11T11:05:03.475Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "08648e4b-0845-4fac-9716-cf7fe467bdc9" "hello.serving-tests2.svc" "10.130.2.29:8112" [2022-10-11T11:05:03.480Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "d95d2cc8-cce1-4f00-8e80-4154c41a9256" "hello-serving-tests2.apps.ut-1-25-75.ci-aws.rhocf-dev.net" "10.130.2.29:8112" [2022-10-11T11:05:03.482Z] "GET /healthz HTTP/1.1" 503 UF 0 209 3 - "-" "Knative-Ingress-Probe" "1bfb8379-40e1-4c3f-ab52-c4f7ce68c5b2" "hello.serving-tests2.svc.cluster.local" "10.130.2.29:8112" [2022-10-11T11:05:03.489Z] "GET /healthz HTTP/1.1" 503 UF 0 209 5 - "-" "Knative-Ingress-Probe" "069b6601-03f3-4468-af7b-8d527a292a56" "hello.serving-tests2" "10.129.2.66:8112" [2022-10-11T11:05:04.281Z] "GET /healthz HTTP/1.1" 503 UF 0 209 6 - "-" "Knative-Ingress-Probe" "e7afd510-746e-4cc9-a827-0a4fd059c2ee" "hello.serving-tests2.svc" "10.129.2.66:8112" [2022-10-11T11:05:04.286Z] "GET /healthz HTTP/1.1" 503 UF 0 209 4 - "-" "Knative-Ingress-Probe" "b40b15f2-dfbc-4d21-9e76-7ecc280be322" "hello-serving-tests2.apps.ut-1-25-75.ci-aws.rhocf-dev.net" "10.129.2.66:8112" [2022-10-11T11:05:04.288Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "337824e7-d569-4969-9cda-bf58f9a1a05f" "hello.serving-tests2.svc.cluster.local" "10.130.2.29:8112" [2022-10-11T11:05:04.298Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "1a4b1346-dc76-4b7b-959f-2f98ff9ee924" "hello.serving-tests2" "10.130.2.29:8112" [2022-10-11T11:05:05.889Z] "GET /healthz HTTP/1.1" 503 UF 0 209 4 - "-" "Knative-Ingress-Probe" "90d36079-a98f-4318-be4d-c10f9c79cb39" "hello.serving-tests2.svc" "10.129.2.66:8112" [2022-10-11T11:05:05.894Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "e942404f-377e-4de8-9cab-f9cf849333f2" "hello.serving-tests2.svc.cluster.local" "10.130.2.29:8112" [2022-10-11T11:05:05.893Z] "GET /healthz HTTP/1.1" 503 UF 0 209 4 - "-" "Knative-Ingress-Probe" "6062f77f-3cd6-4303-9def-e14fdd87d3e4" "hello-serving-tests2.apps.ut-1-25-75.ci-aws.rhocf-dev.net" "10.129.2.66:8112" [2022-10-11T11:05:05.903Z] "GET /healthz HTTP/1.1" 503 UF 0 209 2 - "-" "Knative-Ingress-Probe" "f77dd0b4-b2c9-4247-8fe5-5d27d8f5abd5" "hello.serving-tests2" "10.130.2.29:8112"
knative-openshift-ingress pod logs this:
{"severity":"WARNING","timestamp":"2022-10-11T11:05:02.572367203Z","logger":"openshift-ingress-controller","caller":"ingress/ingress.go:57","message":"Failed to generate routes from ingress unable to find Ingress LoadBalancer with DomainInternal set","knative.dev/pod":"knative-openshift-ingress-f45995bc4-w4n7c","knative.dev/controller":"github.com.openshift-knative.serverless-operator.serving.ingress.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/traceid":"db769ca1-0f2b-43f2-895f-913c3dcf3678","knative.dev/key":"serving-tests2/hello"}
{"severity":"INFO","timestamp":"2022-10-11T11:05:02.572401296Z","logger":"openshift-ingress-controller","caller":"controller/controller.go:550","message":"Reconcile succeeded","knative.dev/pod":"knative-openshift-ingress-f45995bc4-w4n7c","knative.dev/controller":"github.com.openshift-knative.serverless-operator.serving.ingress.pkg.reconciler.ingress.Reconciler","knative.dev/kind":"networking.internal.knative.dev.Ingress","knative.dev/traceid":"db769ca1-0f2b-43f2-895f-913c3dcf3678","knative.dev/key":"serving-tests2/hello","duration":"123.878µs"}
- is blocked by
-
-
- Closed
-