=== RUN TestPilotPlugin 2019-08-14T01:52:32.615291Z info 01:52:32 Envoy config: in /go/out/linux_amd64/release/config.conf.20227.yaml 2019-08-14T01:52:32.615987Z info 01:52:32 Pinging URL: http://localhost:20227/server_info 2019-08-14T01:52:32.616002Z info 01:52:32 HTTP GET http://localhost:20227/server_info 2019-08-14T01:52:32.616566Z info 01:52:32 Get http://localhost:20227/server_info: dial tcp [::1]:20227: connect: connection refused 2019-08-14T01:52:32.616592Z info 01:52:32 Will wait 200ms and try again. [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:205] initializing epoch 0 (hot restart version=10.200.16384.127.options=capacity=16384, num_slots=8209 hash=228984379728933363 size=2654312) [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:207] statically linked extensions: [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:209] access_loggers: envoy.file_access_log,envoy.http_grpc_access_log [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:212] filters.http: envoy.buffer,envoy.cors,envoy.ext_authz,envoy.fault,envoy.filters.http.grpc_http1_reverse_bridge,envoy.filters.http.header_to_metadata,envoy.filters.http.jwt_authn,envoy.filters.http.rbac,envoy.filters.http.tap,envoy.grpc_http1_bridge,envoy.grpc_json_transcoder,envoy.grpc_web,envoy.gzip,envoy.health_check,envoy.http_dynamo_filter,envoy.ip_tagging,envoy.lua,envoy.rate_limit,envoy.router,envoy.squash,istio_authn,jwt-auth,mixer [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:215] filters.listener: envoy.listener.original_dst,envoy.listener.original_src,envoy.listener.proxy_protocol,envoy.listener.tls_inspector [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:218] filters.network: envoy.client_ssl_auth,envoy.echo,envoy.ext_authz,envoy.filters.network.dubbo_proxy,envoy.filters.network.mysql_proxy,envoy.filters.network.rbac,envoy.filters.network.sni_cluster,envoy.filters.network.tcp_cluster_rewrite,envoy.filters.network.thrift_proxy,envoy.filters.network.zookeeper_proxy,envoy.http_connection_manager,envoy.mongo_proxy,envoy.ratelimit,envoy.redis_proxy,envoy.tcp_proxy,forward_downstream_sni,mixer,sni_verifier [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:220] stat_sinks: envoy.dog_statsd,envoy.metrics_service,envoy.stat_sinks.hystrix,envoy.statsd [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:222] tracers: envoy.dynamic.ot,envoy.lightstep,envoy.tracers.datadog,envoy.zipkin [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:225] transport_sockets.downstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:228] transport_sockets.upstream: envoy.transport_sockets.alts,envoy.transport_sockets.tap,raw_buffer,tls [2019-08-14 01:52:32.645][46220][info][main] [external/envoy/source/server/server.cc:234] buffer implementation: old (libevent) [2019-08-14 01:52:32.654][46220][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details. [2019-08-14 01:52:32.654][46220][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details. [2019-08-14 01:52:32.654][46220][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details. [2019-08-14 01:52:32.654][46220][warning][misc] [external/envoy/source/common/protobuf/utility.cc:174] Using deprecated option 'envoy.api.v2.Cluster.hosts' from file cds.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details. [2019-08-14 01:52:32.658][46220][info][main] [external/envoy/source/server/server.cc:281] admin address: 127.0.0.1:20227 [2019-08-14 01:52:32.660][46220][info][config] [external/envoy/source/server/configuration_impl.cc:50] loading 0 static secret(s) [2019-08-14 01:52:32.660][46220][info][config] [external/envoy/source/server/configuration_impl.cc:56] loading 4 cluster(s) [2019-08-14 01:52:32.675][46220][info][upstream] [external/envoy/source/common/upstream/cluster_manager_impl.cc:137] cm init: all clusters initialized [2019-08-14 01:52:32.676][46220][info][config] [external/envoy/source/server/configuration_impl.cc:60] loading 0 listener(s) [2019-08-14 01:52:32.676][46220][info][config] [external/envoy/source/server/configuration_impl.cc:85] loading tracing configuration [2019-08-14 01:52:32.676][46220][info][config] [external/envoy/source/server/configuration_impl.cc:105] loading stats sink configuration [2019-08-14 01:52:32.676][46220][info][main] [external/envoy/source/server/server.cc:462] all clusters initialized. initializing init manager [2019-08-14 01:52:32.676][46220][info][main] [external/envoy/source/server/server.cc:478] starting main dispatch loop [2019-08-14 01:52:32.688][46220][warning][config] [src/envoy/http/mixer/control.cc:50] Missing required node metadata: NODE_UID, NODE_NAMESPACE [2019-08-14 01:52:32.689][46220][info][filter] [src/istio/utils/logger.cc:82] Logger active [2019-08-14 01:52:32.689][46220][info][upstream] [external/envoy/source/server/lds_api.cc:74] lds: add/update listener 'outbound|||svc.ns3' [2019-08-14 01:52:32.697][46220][warning][config] [src/envoy/http/mixer/control.cc:50] Missing required node metadata: NODE_UID, NODE_NAMESPACE [2019-08-14 01:52:32.698][46220][info][filter] [src/istio/utils/logger.cc:82] Logger active [2019-08-14 01:52:32.698][46220][info][upstream] [external/envoy/source/server/lds_api.cc:74] lds: add/update listener 'inbound|||backend' [2019-08-14 01:52:32.702][46220][info][config] [external/envoy/source/server/listener_manager_impl.cc:1006] all dependencies initialized. starting workers [2019-08-14 01:52:32.703][46226][warning][config] [src/envoy/http/mixer/control.cc:50] Missing required node metadata: NODE_UID, NODE_NAMESPACE [2019-08-14 01:52:32.703][46226][warning][config] [src/envoy/http/mixer/control.cc:50] Missing required node metadata: NODE_UID, NODE_NAMESPACE 2019-08-14T01:52:32.816759Z info 01:52:32 Pinging URL: http://localhost:20227/server_info 2019-08-14T01:52:32.816827Z info 01:52:32 HTTP GET http://localhost:20227/server_info 2019-08-14T01:52:32.819225Z info 01:52:32 { "version": "5ea236aa3f759df29ef9209d0cf8e85bf1c8fb2e/1.11.0-dev/Clean/DEBUG/BoringSSL", "state": "LIVE", "command_line_options": { "base_id": "320", "concurrency": 1, "config_path": "/go/out/linux_amd64/release/config.conf.20227.yaml", "config_yaml": "", "allow_unknown_fields": true, "admin_address_path": "", "local_address_ip_version": "v4", "log_level": "info", "component_log_level": "", "log_format": "[%Y-%m-%d %T.%e][%t][%l][%n] %v", "log_path": "", "hot_restart_version": false, "service_cluster": "", "service_node": "", "service_zone": "", "mode": "Serve", "max_stats": "16384", "max_obj_name_len": "60", "disable_hot_restart": false, "enable_mutex_tracing": false, "restart_epoch": 0, "cpuset_threads": false, "file_flush_interval": "10s", "drain_time": "1s", "parent_shutdown_time": "1s" }, "uptime_current_epoch": "0s", "uptime_all_epochs": "0s" } 2019-08-14T01:52:32.819270Z info 01:52:32 Server is up and running... 2019-08-14T01:52:32.819277Z info 01:52:32 Mixer server listening on port 20228 [2019-08-14 01:52:33.701][46220][info][main] [external/envoy/source/server/drain_manager_impl.cc:63] shutting down parent after drain 2019-08-14T01:52:33.819525Z info 01:52:33 Http server listening on port 20229 2019-08-14T01:52:33.819678Z info 01:52:33 Pinging URL: http://localhost:20229/echo 2019-08-14T01:52:33.819686Z info 01:52:33 HTTP GET http://localhost:20229/echo 2019-08-14T01:52:33.820864Z info 01:52:33 2019-08-14T01:52:33.820879Z info 01:52:33 Server is up and running... 2019-08-14T01:52:34.821102Z info 01:52:34 HTTP GET http://localhost:20227/stats?format=json&usedonly 2019-08-14T01:52:34.824274Z info 01:52:34 {"stats":[{"name":"access_log_file.write_buffered","value":1},{"name":"access_log_file.write_completed","value":1},{"name":"access_log_file.write_total_buffered","value":0},{"name":"cluster.inbound|||backend.membership_change","value":1},{"name":"cluster.inbound|||backend.membership_degraded","value":0},{"name":"cluster.inbound|||backend.membership_healthy","value":1},{"name":"cluster.inbound|||backend.membership_total","value":1},{"name":"cluster.outbound|9091||mixer_server.membership_change","value":1},{"name":"cluster.outbound|9091||mixer_server.membership_degraded","value":0},{"name":"cluster.outbound|9091||mixer_server.membership_healthy","value":1},{"name":"cluster.outbound|9091||mixer_server.membership_total","value":1},{"name":"cluster.outbound|||svc.ns3.membership_change","value":1},{"name":"cluster.outbound|||svc.ns3.membership_degraded","value":0},{"name":"cluster.outbound|||svc.ns3.membership_healthy","value":1},{"name":"cluster.outbound|||svc.ns3.membership_total","value":1},{"name":"cluster.xds.circuit_breakers.default.cx_pool_open","value":0},{"name":"cluster.xds.circuit_breakers.default.rq_open","value":0},{"name":"cluster.xds.circuit_breakers.default.rq_pending_open","value":0},{"name":"cluster.xds.internal.upstream_rq_200","value":1},{"name":"cluster.xds.internal.upstream_rq_2xx","value":1},{"name":"cluster.xds.internal.upstream_rq_completed","value":1},{"name":"cluster.xds.membership_change","value":1},{"name":"cluster.xds.membership_degraded","value":0},{"name":"cluster.xds.membership_healthy","value":1},{"name":"cluster.xds.membership_total","value":1},{"name":"cluster.xds.upstream_cx_active","value":1},{"name":"cluster.xds.upstream_cx_http2_total","value":1},{"name":"cluster.xds.upstream_cx_rx_bytes_buffered","value":30},{"name":"cluster.xds.upstream_cx_rx_bytes_total","value":3601},{"name":"cluster.xds.upstream_cx_total","value":1},{"name":"cluster.xds.upstream_cx_tx_bytes_total","value":951},{"name":"cluster.xds.upstream_rq_200","value":1},{"name":"cluster.xds.upstream_rq_2xx","value":1},{"name":"cluster.xds.upstream_rq_active","value":1},{"name":"cluster.xds.upstream_rq_completed","value":1},{"name":"cluster.xds.upstream_rq_pending_active","value":0},{"name":"cluster.xds.upstream_rq_pending_total","value":1},{"name":"cluster.xds.upstream_rq_total","value":1},{"name":"cluster_manager.active_clusters","value":4},{"name":"cluster_manager.cluster_added","value":4},{"name":"cluster_manager.warming_clusters","value":0},{"name":"control_plane.connected_state","value":1},{"name":"http.admin.downstream_cx_active","value":1},{"name":"http.admin.downstream_cx_http1_active","value":1},{"name":"http.admin.downstream_cx_http1_total","value":1},{"name":"http.admin.downstream_cx_rx_bytes_buffered","value":122},{"name":"http.admin.downstream_cx_rx_bytes_total","value":229},{"name":"http.admin.downstream_cx_total","value":1},{"name":"http.admin.downstream_cx_tx_bytes_total","value":1132},{"name":"http.admin.downstream_rq_2xx","value":1},{"name":"http.admin.downstream_rq_active","value":1},{"name":"http.admin.downstream_rq_completed","value":1},{"name":"http.admin.downstream_rq_http1_total","value":2},{"name":"http.admin.downstream_rq_total","value":2},{"name":"http.async-client.rq_total","value":1},{"name":"http.inbound|||backend.rds.inbound|||backend.config_reload","value":1},{"name":"http.inbound|||backend.rds.inbound|||backend.update_attempt","value":2},{"name":"http.inbound|||backend.rds.inbound|||backend.update_success","value":1},{"name":"http.inbound|||backend.rds.inbound|||backend.version","value":-494510850},{"name":"http.outbound|||svc.ns3.rds.outbound|||svc.ns3.config_reload","value":1},{"name":"http.outbound|||svc.ns3.rds.outbound|||svc.ns3.update_attempt","value":2},{"name":"http.outbound|||svc.ns3.rds.outbound|||svc.ns3.update_success","value":1},{"name":"http.outbound|||svc.ns3.rds.outbound|||svc.ns3.version","value":-494510850},{"name":"listener.admin.downstream_cx_active","value":1},{"name":"listener.admin.downstream_cx_total","value":1},{"name":"listener.admin.downstream_pre_cx_active","value":0},{"name":"listener.admin.http.admin.downstream_rq_2xx","value":1},{"name":"listener.admin.http.admin.downstream_rq_completed","value":1},{"name":"listener_manager.lds.update_attempt","value":2},{"name":"listener_manager.lds.update_success","value":1},{"name":"listener_manager.lds.version","value":-494510850},{"name":"listener_manager.listener_added","value":2},{"name":"listener_manager.listener_create_success","value":2},{"name":"listener_manager.total_listeners_active","value":2},{"name":"listener_manager.total_listeners_warming","value":0},{"name":"runtime.num_keys","value":0},{"name":"server.concurrency","value":1},{"name":"server.hot_restart_epoch","value":0},{"name":"server.live","value":1},{"name":"server.version","value":6201910}]} 2019-08-14T01:52:35.824841Z info 01:52:35 HTTP GET http://localhost:20227/stats?format=json&usedonly 2019-08-14T01:52:35.828176Z info 01:52:35 {"stats":[{"name":"access_log_file.write_buffered","value":2},{"name":"access_log_file.write_completed","value":1},{"name":"access_log_file.write_total_buffered","value":146},{"name":"cluster.inbound|||backend.membership_change","value":1},{"name":"cluster.inbound|||backend.membership_degraded","value":0},{"name":"cluster.inbound|||backend.membership_healthy","value":1},{"name":"cluster.inbound|||backend.membership_total","value":1},{"name":"cluster.outbound|9091||mixer_server.membership_change","value":1},{"name":"cluster.outbound|9091||mixer_server.membership_degraded","value":0},{"name":"cluster.outbound|9091||mixer_server.membership_healthy","value":1},{"name":"cluster.outbound|9091||mixer_server.membership_total","value":1},{"name":"cluster.outbound|||svc.ns3.membership_change","value":1},{"name":"cluster.outbound|||svc.ns3.membership_degraded","value":0},{"name":"cluster.outbound|||svc.ns3.membership_healthy","value":1},{"name":"cluster.outbound|||svc.ns3.membership_total","value":1},{"name":"cluster.xds.circuit_breakers.default.cx_pool_open","value":0},{"name":"cluster.xds.circuit_breakers.default.rq_open","value":0},{"name":"cluster.xds.circuit_breakers.default.rq_pending_open","value":0},{"name":"cluster.xds.internal.upstream_rq_200","value":1},{"name":"cluster.xds.internal.upstream_rq_2xx","value":1},{"name":"cluster.xds.internal.upstream_rq_completed","value":1},{"name":"cluster.xds.membership_change","value":1},{"name":"cluster.xds.membership_degraded","value":0},{"name":"cluster.xds.membership_healthy","value":1},{"name":"cluster.xds.membership_total","value":1},{"name":"cluster.xds.upstream_cx_active","value":1},{"name":"cluster.xds.upstream_cx_http2_total","value":1},{"name":"cluster.xds.upstream_cx_rx_bytes_buffered","value":30},{"name":"cluster.xds.upstream_cx_rx_bytes_total","value":3601},{"name":"cluster.xds.upstream_cx_total","value":1},{"name":"cluster.xds.upstream_cx_tx_bytes_total","value":951},{"name":"cluster.xds.upstream_rq_200","value":1},{"name":"cluster.xds.upstream_rq_2xx","value":1},{"name":"cluster.xds.upstream_rq_active","value":1},{"name":"cluster.xds.upstream_rq_completed","value":1},{"name":"cluster.xds.upstream_rq_pending_active","value":0},{"name":"cluster.xds.upstream_rq_pending_total","value":1},{"name":"cluster.xds.upstream_rq_total","value":1},{"name":"cluster_manager.active_clusters","value":4},{"name":"cluster_manager.cluster_added","value":4},{"name":"cluster_manager.warming_clusters","value":0},{"name":"control_plane.connected_state","value":1},{"name":"http.admin.downstream_cx_active","value":1},{"name":"http.admin.downstream_cx_http1_active","value":1},{"name":"http.admin.downstream_cx_http1_total","value":1},{"name":"http.admin.downstream_cx_rx_bytes_buffered","value":122},{"name":"http.admin.downstream_cx_rx_bytes_total","value":351},{"name":"http.admin.downstream_cx_total","value":1},{"name":"http.admin.downstream_cx_tx_bytes_total","value":6189},{"name":"http.admin.downstream_rq_2xx","value":2},{"name":"http.admin.downstream_rq_active","value":1},{"name":"http.admin.downstream_rq_completed","value":2},{"name":"http.admin.downstream_rq_http1_total","value":3},{"name":"http.admin.downstream_rq_total","value":3},{"name":"http.async-client.rq_total","value":1},{"name":"http.inbound|||backend.rds.inbound|||backend.config_reload","value":1},{"name":"http.inbound|||backend.rds.inbound|||backend.update_attempt","value":2},{"name":"http.inbound|||backend.rds.inbound|||backend.update_success","value":1},{"name":"http.inbound|||backend.rds.inbound|||backend.version","value":-494510850},{"name":"http.outbound|||svc.ns3.rds.outbound|||svc.ns3.config_reload","value":1},{"name":"http.outbound|||svc.ns3.rds.outbound|||svc.ns3.update_attempt","value":2},{"name":"http.outbound|||svc.ns3.rds.outbound|||svc.ns3.update_success","value":1},{"name":"http.outbound|||svc.ns3.rds.outbound|||svc.ns3.version","value":-494510850},{"name":"listener.admin.downstream_cx_active","value":1},{"name":"listener.admin.downstream_cx_total","value":1},{"name":"listener.admin.downstream_pre_cx_active","value":0},{"name":"listener.admin.http.admin.downstream_rq_2xx","value":2},{"name":"listener.admin.http.admin.downstream_rq_completed","value":2},{"name":"listener_manager.lds.update_attempt","value":2},{"name":"listener_manager.lds.update_success","value":1},{"name":"listener_manager.lds.version","value":-494510850},{"name":"listener_manager.listener_added","value":2},{"name":"listener_manager.listener_create_success","value":2},{"name":"listener_manager.total_listeners_active","value":2},{"name":"listener_manager.total_listeners_warming","value":0},{"name":"runtime.num_keys","value":0},{"name":"server.concurrency","value":1},{"name":"server.hot_restart_epoch","value":0},{"name":"server.live","value":1},{"name":"server.version","value":6201910}]} 2019-08-14T01:52:35.828396Z info 01:52:35 HTTP GET http://localhost:20224/echo 2019-08-14T01:52:35.837529Z info 01:52:35 upstream connect error or disconnect/reset before headers. reset reason: connection failure SIGQUIT: quit PC=0x45d2e1 m=0 sigcode=0 goroutine 0 [idle]: runtime.futex(0x1b365e8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x7ffdce45d1b8, 0x410c2b, ...) /usr/local/src/runtime/sys_linux_amd64.s:530 +0x21 runtime.futexsleep(0x1b365e8, 0x7ffd00000000, 0xffffffffffffffff) /usr/local/src/runtime/os_linux.go:45 +0x4b runtime.notesleep(0x1b365e8) /usr/local/src/runtime/lock_futex.go:151 +0x9b runtime.stopm() /usr/local/src/runtime/proc.go:1952 +0xe5 runtime.findrunnable(0xc420058500, 0x0) /usr/local/src/runtime/proc.go:2415 +0x50c runtime.schedule() /usr/local/src/runtime/proc.go:2541 +0x13b runtime.park_m(0xc4203fc300) /usr/local/src/runtime/proc.go:2604 +0xb6 runtime.mcall(0x0) /usr/local/src/runtime/asm_amd64.s:351 +0x5b goroutine 1 [chan receive, 9 minutes]: testing.(*T).Run(0xc42041a0f0, 0x12ef6e9, 0xf, 0x1378c90, 0x47e5a6) /usr/local/src/testing/testing.go:825 +0x301 testing.runTests.func1(0xc42041a000) /usr/local/src/testing/testing.go:1063 +0x64 testing.tRunner(0xc42041a000, 0xc420305df8) /usr/local/src/testing/testing.go:777 +0xd0 testing.runTests(0xc4204180a0, 0x1b19db0, 0x1, 0x1, 0x411c49) /usr/local/src/testing/testing.go:1061 +0x2c4 testing.(*M).Run(0xc420416080, 0x0) /usr/local/src/testing/testing.go:978 +0x171 main.main() _testmain.go:44 +0x151 goroutine 7 [select, 9 minutes]: istio.io/istio/pilot/pkg/model.(*jwksResolver).refresher(0xc42023e300) /go/src/istio.io/istio/pilot/pkg/model/jwks_resolver.go:311 +0x107 created by istio.io/istio/pilot/pkg/model.newJwksResolver /go/src/istio.io/istio/pilot/pkg/model/jwks_resolver.go:157 +0x1d0 goroutine 18 [syscall, 9 minutes]: os/signal.signal_recv(0x0) /usr/local/src/runtime/sigqueue.go:139 +0xa6 os/signal.loop() /usr/local/src/os/signal/signal_unix.go:22 +0x22 created by os/signal.init.0 /usr/local/src/os/signal/signal_unix.go:28 +0x41 goroutine 6 [select, 9 minutes]: istio.io/istio/pkg/cache.(*ttlCache).evicter(0xc42023e280, 0x1a3185c5000) /go/src/istio.io/istio/pkg/cache/ttlCache.go:121 +0x13a created by istio.io/istio/pkg/cache.NewTTLWithCallback /go/src/istio.io/istio/pkg/cache/ttlCache.go:100 +0x1cd goroutine 27 [chan receive, 9 minutes]: istio.io/istio/mixer/test/client/env.(*TestSetup).VerifyCheck(0xc4202e51e0, 0x12ec2ab, 0xc, 0x136ec55, 0x450) /go/src/istio.io/istio/mixer/test/client/env/setup.go:294 +0x75 istio.io/istio/mixer/test/client/pilotplugin_mtls_test.TestPilotPlugin(0xc42041a0f0) /go/src/istio.io/istio/mixer/test/client/pilotplugin_mtls/pilotplugin_mtls_test.go:303 +0x6ab testing.tRunner(0xc42041a0f0, 0x1378c90) /usr/local/src/testing/testing.go:777 +0xd0 created by testing.(*T).Run /usr/local/src/testing/testing.go:824 +0x2e0 goroutine 131 [IO wait, 9 minutes]: internal/poll.runtime_pollWait(0x2b860f256f00, 0x72, 0x0) /usr/local/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc420417318, 0x72, 0xc4203d4000, 0x0, 0x0) /usr/local/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(*pollDesc).waitRead(0xc420417318, 0xffffffffffffff00, 0x0, 0x0) /usr/local/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Accept(0xc420417300, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/src/internal/poll/fd_unix.go:372 +0x1a8 net.(*netFD).accept(0xc420417300, 0xc4203cc010, 0x0, 0x0) /usr/local/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc4202da218, 0x458470, 0xc420105e58, 0xc420105e60) /usr/local/src/net/tcpsock_posix.go:136 +0x2e net.(*TCPListener).Accept(0xc4202da218, 0x137a468, 0xc420130d80, 0x1411060, 0xc4203cc010) /usr/local/src/net/tcpsock.go:259 +0x49 istio.io/istio/vendor/google.golang.org/grpc.(*Server).Serve(0xc420130d80, 0x1404ee0, 0xc4202da218, 0x0, 0x0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:557 +0x21a istio.io/istio/mixer/test/client/pilotplugin_mtls_test.TestPilotPlugin.func1(0xc420130d80, 0x1404ee0, 0xc4202da218) /go/src/istio.io/istio/mixer/test/client/pilotplugin_mtls/pilotplugin_mtls_test.go:287 +0x3f created by istio.io/istio/mixer/test/client/pilotplugin_mtls_test.TestPilotPlugin /go/src/istio.io/istio/mixer/test/client/pilotplugin_mtls/pilotplugin_mtls_test.go:286 +0x433 goroutine 149 [select, 9 minutes]: istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/pkg/server.(*server).process(0xc4203c9c20, 0x2b860f29aba8, 0xc4202d23b0, 0xc42004c540, 0x0, 0x0, 0x0, 0x0) /go/src/istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/pkg/server/server.go:192 +0x41d istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/pkg/server.(*server).handler(0xc4203c9c20, 0x2b860f29aba8, 0xc4202d23b0, 0x0, 0x0, 0x1, 0xc4202d23b0) /go/src/istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/pkg/server/server.go:322 +0xdc istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/pkg/server.(*server).StreamAggregatedResources(0xc4203c9c20, 0x1410fa0, 0xc4202d23b0, 0x2b860f25b0b8, 0xc4203c9c20) /go/src/istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/pkg/server/server.go:332 +0x6b istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2._AggregatedDiscoveryService_StreamAggregatedResources_Handler(0x128bcc0, 0xc4203c9c20, 0x140e180, 0xc4202ecb00, 0x1b53bb8, 0xc420380540) /go/src/istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2/ads.pb.go:197 +0xb2 istio.io/istio/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc420130d80, 0x1411a20, 0xc4204e0480, 0xc420449e00, 0xc4203c9cb0, 0x1b1f1c0, 0x0, 0x0, 0x0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:1124 +0x911 istio.io/istio/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420130d80, 0x1411a20, 0xc4204e0480, 0xc420449e00, 0x0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:1212 +0x12b1 istio.io/istio/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc4202de0b0, 0xc420130d80, 0x1411a20, 0xc4204e0480, 0xc420449e00) /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:686 +0x9f created by istio.io/istio/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:684 +0xa1 goroutine 146 [select, 9 minutes]: istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc4202d4100, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x133 istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc4202cd560, 0x0, 0x0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1a8 istio.io/istio/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc4204e0480) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xda created by istio.io/istio/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xe29 goroutine 147 [select, 9 minutes]: istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc4204e0480) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x23b created by istio.io/istio/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xe4e goroutine 148 [IO wait, 9 minutes]: internal/poll.runtime_pollWait(0x2b860f256e30, 0x72, 0xc4204bbb50) /usr/local/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc420416198, 0x72, 0xffffffffffffff00, 0x13f1260, 0x1ace7a0) /usr/local/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(*pollDesc).waitRead(0xc420416198, 0xc4204ec000, 0x8000, 0x8000) /usr/local/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc420416180, 0xc4204ec000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/src/internal/poll/fd_unix.go:157 +0x17d net.(*netFD).Read(0xc420416180, 0xc4204ec000, 0x8000, 0x8000, 0x11, 0x0, 0x0) /usr/local/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc4203cc010, 0xc4204ec000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/src/net/net.go:176 +0x6a bufio.(*Reader).Read(0xc420446000, 0xc420186038, 0x9, 0x9, 0x20, 0x0, 0x0) /usr/local/src/bufio/bufio.go:216 +0x238 io.ReadAtLeast(0x13ede60, 0xc420446000, 0xc420186038, 0x9, 0x9, 0x9, 0xc429e30c18, 0x29e30c1800401afe, 0x5d536960) /usr/local/src/io/io.go:309 +0x86 io.ReadFull(0x13ede60, 0xc420446000, 0xc420186038, 0x9, 0x9, 0xbf4cf83829e30c18, 0x5fd3dfb, 0x1b35640) /usr/local/src/io/io.go:327 +0x58 istio.io/istio/vendor/golang.org/x/net/http2.readFrameHeader(0xc420186038, 0x9, 0x9, 0x13ede60, 0xc420446000, 0x0, 0x0, 0x12ec5, 0xc420412068) /go/src/istio.io/istio/vendor/golang.org/x/net/http2/frame.go:237 +0x7b istio.io/istio/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420186000, 0xc4203d2340, 0xc4203d2340, 0x0, 0x0) /go/src/istio.io/istio/vendor/golang.org/x/net/http2/frame.go:492 +0xa4 istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc4204e0480, 0xc420380300, 0x137a4c0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c istio.io/istio/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc420130d80, 0x1411a20, 0xc4204e0480) /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:682 +0xeb istio.io/istio/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc420130d80, 0x1411a20, 0xc4204e0480) /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:644 +0x43 created by istio.io/istio/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:643 +0x582 goroutine 150 [select, 9 minutes]: istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc420412280, 0xc4202d8330, 0x5, 0x5, 0xa4e020, 0x0, 0xc42045c17e) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/transport.go:142 +0x1e0 istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc420412280, 0xc4202d8330, 0x5, 0x5, 0xbf, 0x0, 0x87) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/transport.go:131 +0x67 istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc4203804e0, 0xc4202d8330, 0x5, 0x5, 0xbf, 0xc4204bcc20, 0x8795b8) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/transport.go:401 +0x55 io.ReadAtLeast(0x13eea00, 0xc4203804e0, 0xc4202d8330, 0x5, 0x5, 0x5, 0xc4204e0480, 0xc420449e00, 0x5) /usr/local/src/io/io.go:309 +0x86 io.ReadFull(0x13eea00, 0xc4203804e0, 0xc4202d8330, 0x5, 0x5, 0xc4204bccf0, 0x41039d, 0x1178c60) /usr/local/src/io/io.go:327 +0x58 istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc420449e00, 0xc4202d8330, 0x5, 0x5, 0xc42045c0c0, 0xbf, 0xbf) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/transport.go:385 +0xbf istio.io/istio/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc4202d8320, 0x400000, 0xbf, 0xbf, 0x129fb40, 0xc42023e000, 0x0, 0x0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/rpc_util.go:476 +0x65 istio.io/istio/vendor/google.golang.org/grpc.recv(0xc4202d8320, 0x2b860f29ab80, 0x1b53bb8, 0xc420449e00, 0x0, 0x0, 0x129fb40, 0xc420416480, 0x400000, 0x0, ...) /go/src/istio.io/istio/vendor/google.golang.org/grpc/rpc_util.go:602 +0x4d istio.io/istio/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc4202ecb00, 0x129fb40, 0xc420416480, 0x0, 0x0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/stream.go:1004 +0x124 istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2.(*aggregatedDiscoveryServiceStreamAggregatedResourcesServer).Recv(0xc4202d23b0, 0xc4204bcfa8, 0x0, 0x0) /go/src/istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/envoy/service/discovery/v2/ads.pb.go:216 +0x62 istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/pkg/server.(*server).handler.func1(0x2b860f29aba8, 0xc4202d23b0, 0xc4202de138, 0xc42004c540) /go/src/istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/pkg/server/server.go:310 +0x50 created by istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/pkg/server.(*server).handler /go/src/istio.io/istio/vendor/github.com/envoyproxy/go-control-plane/pkg/server/server.go:308 +0x99 goroutine 46 [IO wait, 9 minutes]: internal/poll.runtime_pollWait(0x2b860f256bc0, 0x72, 0x0) /usr/local/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc42023e498, 0x72, 0xc42003a900, 0x0, 0x0) /usr/local/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(*pollDesc).waitRead(0xc42023e498, 0xffffffffffffff00, 0x0, 0x0) /usr/local/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Accept(0xc42023e480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/src/internal/poll/fd_unix.go:372 +0x1a8 net.(*netFD).accept(0xc42023e480, 0x4019c7, 0xc42030e1c0, 0x1143700) /usr/local/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc42000e068, 0x458470, 0xc4204b9e98, 0xc4204b9ea0) /usr/local/src/net/tcpsock_posix.go:136 +0x2e net.(*TCPListener).Accept(0xc42000e068, 0x137abc0, 0xc42030e140, 0x14081e0, 0xc420440fc0) /usr/local/src/net/tcpsock.go:259 +0x49 net/http.(*Server).Serve(0xc4204dc0d0, 0x1404ee0, 0xc42000e068, 0x0, 0x0) /usr/local/src/net/http/server.go:2773 +0x1a5 net/http.Serve(0x1404ee0, 0xc42000e068, 0x0, 0x0, 0x0, 0xc4203d8380) /usr/local/src/net/http/server.go:2392 +0x73 istio.io/istio/mixer/test/client/env.(*HTTPServer).Start.func1(0xc420440b40, 0xc42037a660) /go/src/istio.io/istio/mixer/test/client/env/http_server.go:155 +0xc1 created by istio.io/istio/mixer/test/client/env.(*HTTPServer).Start /go/src/istio.io/istio/mixer/test/client/env/http_server.go:152 +0x6b goroutine 151 [IO wait, 9 minutes]: internal/poll.runtime_pollWait(0x2b860f256c90, 0x72, 0x0) /usr/local/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc420416518, 0x72, 0xc4202d4400, 0x0, 0x0) /usr/local/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(*pollDesc).waitRead(0xc420416518, 0xffffffffffffff00, 0x0, 0x0) /usr/local/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Accept(0xc420416500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/src/internal/poll/fd_unix.go:372 +0x1a8 net.(*netFD).accept(0xc420416500, 0xc42000e148, 0x0, 0x0) /usr/local/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc4203cc020, 0x458470, 0xc420453e30, 0xc420453e38) /usr/local/src/net/tcpsock_posix.go:136 +0x2e net.(*TCPListener).Accept(0xc4203cc020, 0x137a468, 0xc4204e0c00, 0x1411060, 0xc42000e148) /usr/local/src/net/tcpsock.go:259 +0x49 istio.io/istio/vendor/google.golang.org/grpc.(*Server).Serve(0xc4204e0c00, 0x1404ee0, 0xc4203cc020, 0x0, 0x0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:557 +0x21a istio.io/istio/mixer/test/client/env.(*MixerServer).Start.func1(0xc4203e2820, 0xc42004c6c0) /go/src/istio.io/istio/mixer/test/client/env/mixer_server.go:178 +0x45 created by istio.io/istio/mixer/test/client/env.(*MixerServer).Start /go/src/istio.io/istio/mixer/test/client/env/mixer_server.go:177 +0x6b goroutine 190 [select, 9 minutes]: istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc42003af80, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x133 istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc420446900, 0x0, 0x0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1a8 istio.io/istio/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc4204ca780) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xda created by istio.io/istio/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xe29 goroutine 191 [select, 9 minutes]: istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc4204ca780) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:921 +0x23b created by istio.io/istio/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xe4e goroutine 192 [IO wait, 9 minutes]: internal/poll.runtime_pollWait(0x2b860f256880, 0x72, 0xc42043bb50) /usr/local/src/runtime/netpoll.go:173 +0x57 internal/poll.(*pollDesc).wait(0xc42023eb18, 0x72, 0xffffffffffffff00, 0x13f1260, 0x1ace7a0) /usr/local/src/internal/poll/fd_poll_runtime.go:85 +0x9b internal/poll.(*pollDesc).waitRead(0xc42023eb18, 0xc420508000, 0x8000, 0x8000) /usr/local/src/internal/poll/fd_poll_runtime.go:90 +0x3d internal/poll.(*FD).Read(0xc42023eb00, 0xc420508000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/src/internal/poll/fd_unix.go:157 +0x17d net.(*netFD).Read(0xc42023eb00, 0xc420508000, 0x8000, 0x8000, 0x11, 0x0, 0x0) /usr/local/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc42000e148, 0xc420508000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/src/net/net.go:176 +0x6a bufio.(*Reader).Read(0xc4203d9740, 0xc42002e118, 0x9, 0x9, 0x20, 0x0, 0x0) /usr/local/src/bufio/bufio.go:216 +0x238 io.ReadAtLeast(0x13ede60, 0xc4203d9740, 0xc42002e118, 0x9, 0x9, 0x9, 0xc431d47581, 0x31d4758100401afe, 0x5d536964) /usr/local/src/io/io.go:309 +0x86 io.ReadFull(0x13ede60, 0xc4203d9740, 0xc42002e118, 0x9, 0x9, 0xbf4cf83931d47581, 0xfc59cf65, 0x1b35640) /usr/local/src/io/io.go:327 +0x58 istio.io/istio/vendor/golang.org/x/net/http2.readFrameHeader(0xc42002e118, 0x9, 0x9, 0x13ede60, 0xc4203d9740, 0x0, 0x0, 0x1f4ae, 0xc42044a928) /go/src/istio.io/istio/vendor/golang.org/x/net/http2/frame.go:237 +0x7b istio.io/istio/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc42002e0e0, 0xc4203d2bc0, 0xc4203d2bc0, 0x0, 0x0) /go/src/istio.io/istio/vendor/golang.org/x/net/http2/frame.go:492 +0xa4 istio.io/istio/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc4204ca780, 0xc420441920, 0x137a4c0) /go/src/istio.io/istio/vendor/google.golang.org/grpc/internal/transport/http2_server.go:429 +0x7c istio.io/istio/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc4204e0c00, 0x1411a20, 0xc4204ca780) /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:682 +0xeb istio.io/istio/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc4204e0c00, 0x1411a20, 0xc4204ca780) /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:644 +0x43 created by istio.io/istio/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/istio.io/istio/vendor/google.golang.org/grpc/server.go:643 +0x582 rax 0xca rbx 0x1b364a0 rcx 0xffffffffffffffff rdx 0x0 rdi 0x1b365e8 rsi 0x0 rbp 0x7ffdce45d180 rsp 0x7ffdce45d138 r8 0x0 r9 0x0 r10 0x0 r11 0x286 r12 0xffffffffffffffff r13 0xec r14 0xeb r15 0x200 rip 0x45d2e1 rflags 0x286 cs 0x33 fs 0x0 gs 0x0 [2019-08-14 02:02:32.607][46220][warning][config] [bazel-out/k8-dbg/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:86] gRPC config stream closed: 13, [2019-08-14 02:02:32.608][46220][warning][main] [external/envoy/source/server/server.cc:425] caught SIGTERM [2019-08-14 02:02:32.608][46220][info][main] [external/envoy/source/server/server.cc:529] shutting down server instance [2019-08-14 02:02:32.608][46220][info][main] [external/envoy/source/server/server.cc:483] main dispatch loop exited [2019-08-14 02:02:32.611][46220][info][main] [external/envoy/source/server/server.cc:522] exiting *** Test killed with quit: ran too long (10m0s). FAIL istio.io/istio/mixer/test/client/pilotplugin_mtls 600.015s