Uploaded image for project: 'Knative Serving'
  1. Knative Serving
  2. SRVKS-150

TestDestroyPodInflight fails with pre-installed maistra

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Done
    • Icon: Undefined Undefined
    • None
    • v0.6.0, v0.7.0
    • None
    • Serverless Sprint 180

      On, OCP 4.1.2, with Maistra 0.11 pre-installed, TestDestroyPodInflight fails

      --- FAIL: TestDestroyPodInflight (41.50s)
 destroypod_test.go:50: Creating a new Route and Configuration
 util.go:34: resource {<nil> <*>{&TypeMeta{Kind:,APIVersion:,} &ObjectMeta{Name:destroy-pod-inflight-fooiwlsa,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,} {0 <nil> <nil> <*>&ObjectMeta{Name:,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,}} {{0 <nil>} { }}} <nil> <nil>}
 util.go:34: resource {<*>{&TypeMeta{Kind:,APIVersion:,} &ObjectMeta{Name:destroy-pod-inflight-fooiwlsa,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,} {0 [{ { destroy-pod-inflight-fooiwlsa <nil> 100 <nil>}}]} {{0 <nil>} {<nil> <nil> <nil>}}} <nil> <nil> <nil>}
 destroypod_test.go:58: When the Revision can have traffic routed to it, the Route is marked as Ready
 destroypod_test.go:106: Sending in a long running request
 destroypod_test.go:127: Destroying the configuration (also destroys the pods)
 destroypod_test.go:132: Something went wrong with the request: Expected response to have status 200, had 503
FAIL

      logs from the istio proxy container of the destroy-pod-inflight pod

       

      oc logs destroy-pod-inflight-fooiwlsa-8t84n-deployment-78d5fc547c-gpvg7 -f -c istio-proxy
2019-06-19T17:12:49.373725Z info FLAG: --applicationPorts="[8080,8022,9090,8012]"
2019-06-19T17:12:49.373775Z info FLAG: --binaryPath="/usr/local/bin/envoy"
2019-06-19T17:12:49.373784Z info FLAG: --concurrency="2"
2019-06-19T17:12:49.373790Z info FLAG: --configPath="/etc/istio/proxy"
2019-06-19T17:12:49.373798Z info FLAG: --connectTimeout="10s"
2019-06-19T17:12:49.373822Z info FLAG: --controlPlaneAuthPolicy="NONE"
2019-06-19T17:12:49.373831Z info FLAG: --controlPlaneBootstrap="true"
2019-06-19T17:12:49.373836Z info FLAG: --customConfigFile=""
2019-06-19T17:12:49.373842Z info FLAG: --datadogAgentAddress=""
2019-06-19T17:12:49.373848Z info FLAG: --disableInternalTelemetry="false"
2019-06-19T17:12:49.373854Z info FLAG: --discoveryAddress="istio-pilot.istio-system:15010"
2019-06-19T17:12:49.373860Z info FLAG: --domain="serving-tests.svc.cluster.local"
2019-06-19T17:12:49.373866Z info FLAG: --drainDuration="45s"
2019-06-19T17:12:49.373872Z info FLAG: --envoyMetricsServiceAddress=""
2019-06-19T17:12:49.373878Z info FLAG: --help="false"
2019-06-19T17:12:49.373883Z info FLAG: --id=""
2019-06-19T17:12:49.373889Z info FLAG: --ip=""
2019-06-19T17:12:49.373895Z info FLAG: --lightstepAccessToken=""
2019-06-19T17:12:49.373900Z info FLAG: --lightstepAddress=""
2019-06-19T17:12:49.373905Z info FLAG: --lightstepCacertPath=""
2019-06-19T17:12:49.373911Z info FLAG: --lightstepSecure="false"
2019-06-19T17:12:49.373917Z info FLAG: --log_as_json="false"
2019-06-19T17:12:49.373923Z info FLAG: --log_caller=""
2019-06-19T17:12:49.373929Z info FLAG: --log_output_level="default:info"
2019-06-19T17:12:49.373934Z info FLAG: --log_rotate=""
2019-06-19T17:12:49.373940Z info FLAG: --log_rotate_max_age="30"
2019-06-19T17:12:49.373946Z info FLAG: --log_rotate_max_backups="1000"
2019-06-19T17:12:49.373953Z info FLAG: --log_rotate_max_size="104857600"
2019-06-19T17:12:49.373959Z info FLAG: --log_stacktrace_level="default:none"
2019-06-19T17:12:49.373977Z info FLAG: --log_target="[stdout]"
2019-06-19T17:12:49.373984Z info FLAG: --parentShutdownDuration="1m0s"
2019-06-19T17:12:49.373992Z info FLAG: --proxyAdminPort="15000"
2019-06-19T17:12:49.373998Z info FLAG: --proxyLogLevel="warning"
2019-06-19T17:12:49.374005Z info FLAG: --serviceCluster="destroy-pod-inflight-fooiwlsa-8t84n.serving-tests"
2019-06-19T17:12:49.374011Z info FLAG: --serviceregistry="Kubernetes"
2019-06-19T17:12:49.374017Z info FLAG: --statsdUdpAddress=""
2019-06-19T17:12:49.374023Z info FLAG: --statusPort="15020"
2019-06-19T17:12:49.374028Z info FLAG: --templateFile=""
2019-06-19T17:12:49.374034Z info FLAG: --trust-domain=""
2019-06-19T17:12:49.374040Z info FLAG: --zipkinAddress="zipkin.istio-system:9411"
2019-06-19T17:12:49.374064Z info Version redhat@redhat-docker.io/maistra-0.11.0-6-74878c5d0f82e501871ab6302fd69e323e47f116-Clean
2019-06-19T17:12:49.374269Z info Obtained private IP [10.128.2.42 fe80::f4ff:91ff:fe5f:c96b]
2019-06-19T17:12:49.374339Z info Proxy role: &model.Proxy{ClusterID:"", Type:"sidecar", IPAddresses:[]string{"10.128.2.42", "10.128.2.42", "fe80::f4ff:91ff:fe5f:c96b"}, ID:"destroy-pod-inflight-fooiwlsa-8t84n-deployment-78d5fc547c-gpvg7.serving-tests", Locality:(*core.Locality)(nil), DNSDomain:"serving-tests.svc.cluster.local", ConfigNamespace:"", TrustDomain:"cluster.local", Metadata:map[string]string{}, SidecarScope:(*model.SidecarScope)(nil), ServiceInstances:[]*model.ServiceInstance(nil), WorkloadLabels:model.LabelsCollection(nil)}
2019-06-19T17:12:49.374363Z info PilotSAN []string(nil)
2019-06-19T17:12:49.375032Z info Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 2
configPath: /etc/istio/proxy
connectTimeout: 10s
discoveryAddress: istio-pilot.istio-system:15010
drainDuration: 45s
parentShutdownDuration: 60s
proxyAdminPort: 15000
serviceCluster: destroy-pod-inflight-fooiwlsa-8t84n.serving-tests
statNameLength: 189
tracing:
 zipkin:
 address: zipkin.istio-system:9411
2019-06-19T17:12:49.375064Z info Monitored certs: []string{"/etc/certs/cert-chain.pem", "/etc/certs/key.pem", "/etc/certs/root-cert.pem"}
2019-06-19T17:12:49.375094Z info PilotSAN []string(nil)
2019-06-19T17:12:49.375144Z info Starting proxy agent
2019-06-19T17:12:49.375260Z info Opening status port 15020
2019-06-19T17:12:49.375500Z info Received new config, resetting budget
2019-06-19T17:12:49.375521Z info watching /etc/certs/cert-chain.pem for changes
2019-06-19T17:12:49.375539Z info Reconciling retry (budget 10)
2019-06-19T17:12:49.375563Z info watching /etc/certs/key.pem for changes
2019-06-19T17:12:49.375597Z info watching /etc/certs/root-cert.pem for changes
2019-06-19T17:12:49.375583Z info Epoch 0 starting
2019-06-19T17:12:49.376673Z info Envoy command: [-c /etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster destroy-pod-inflight-fooiwlsa-8t84n.serving-tests --service-node sidecar~10.128.2.42~destroy-pod-inflight-fooiwlsa-8t84n-deployment-78d5fc547c-gpvg7.serving-tests~serving-tests.svc.cluster.local --max-obj-name-len 189 --allow-unknown-fields -l warning --concurrency 2]
[2019-06-19 17:12:49.399][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-06-19 17:12:49.399][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-06-19 17:12:49.399][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-06-19 17:12:49.404][18][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 14, no healthy upstream
[2019-06-19 17:12:49.404][18][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:49] Unable to establish new stream
2019-06-19T17:12:49.770822Z info watchFileEvents: "/etc/certs/cert-chain.pem": MODIFY|ATTRIB
2019-06-19T17:12:49.770865Z info watchFileEvents: "/etc/certs/key.pem": MODIFY|ATTRIB
2019-06-19T17:12:49.770879Z info watchFileEvents: "/etc/certs/root-cert.pem": MODIFY|ATTRIB
[2019-06-19 17:12:49.895][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Listener.use_original_dst' from file lds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-06-19 17:12:49.901][18][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
[2019-06-19 17:12:49.903][18][warning][filter] [src/envoy/http/authn/http_filter_factory.cc:102] mTLS PERMISSIVE mode is used, connection can be either plaintext or TLS, and client cert can be omitted. Please consider to upgrade to mTLS STRICT mode for more secure configuration that only allows TLS connection with client cert. See https://istio.io/docs/tasks/security/mtls-migration/
2019-06-19T17:12:50.774409Z info watchFileEvents: "/etc/certs/cert-chain.pem": MODIFY|ATTRIB
2019-06-19T17:12:50.774483Z info watchFileEvents: "/etc/certs/key.pem": MODIFY|ATTRIB
2019-06-19T17:12:50.774496Z info watchFileEvents: "/etc/certs/root-cert.pem": MODIFY|ATTRIB
2019-06-19T17:12:50.901516Z info Envoy proxy is ready
[2019-06-19 17:12:51.464][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Listener.use_original_dst' from file lds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2019-06-19 17:12:52.362][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Listener.use_original_dst' from file lds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
2019-06-19T17:12:59.771054Z info watchFileEvents: notifying
2019-06-19T17:13:14.149905Z info pilot-agent is terminating
2019-06-19T17:13:14.149967Z info Status server has successfully terminated
2019-06-19T17:13:14.150033Z error accept tcp [::]:15020: use of closed network connection
2019-06-19T17:13:14.150057Z error <nil>
2019-06-19T17:13:14.150072Z info Agent draining Proxy
2019-06-19T17:13:14.150082Z info Reconciling retry (budget 10)
2019-06-19T17:13:14.150110Z info Graceful termination period is 5s, starting...
2019-06-19T17:13:14.150126Z info Epoch 1 starting
2019-06-19T17:13:14.150178Z info Envoy command: [-c /var/lib/istio/envoy/envoy_bootstrap_drain.json --restart-epoch 1 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster destroy-pod-inflight-fooiwlsa-8t84n.serving-tests --service-node sidecar~10.128.2.42~destroy-pod-inflight-fooiwlsa-8t84n-deployment-78d5fc547c-gpvg7.serving-tests~serving-tests.svc.cluster.local --max-obj-name-len 189 --allow-unknown-fields -l warning --concurrency 2]
2019-06-19T17:13:14.150554Z info watchFileEvents has successfully terminated
2019-06-19T17:13:14.150612Z info Watcher has successfully terminated
[2019-06-19 17:13:14.173][18][warning][main] [external/envoy/source/server/server.cc:536] shutting down admin due to child startup
[2019-06-19 17:13:14.173][18][warning][main] [external/envoy/source/server/server.cc:544] terminating parent process
[2019-06-19 17:13:14.528][18][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Listener.use_original_dst' from file lds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
2019-06-19T17:13:19.150300Z info Graceful termination period complete, terminating remaining proxies.
2019-06-19T17:13:19.150345Z warn Aborting epoch 0...
2019-06-19T17:13:19.150360Z warn Aborting epoch 1...
2019-06-19T17:13:19.150365Z warn Aborted all epochs
2019-06-19T17:13:19.150370Z info Agent has successfully terminated
[2019-06-19 17:13:19.150][18][warning][main] [external/envoy/source/server/server.cc:425] caught SIGTERM
[2019-06-19 17:13:19.150][31][warning][main] [external/envoy/source/server/server.cc:425] caught SIGTERM

       

       

       

              rhn-support-knakayam Kenjiro Nakayama (Inactive)
              maschmid@redhat.com Marek Schmidt
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: