2022-01-18T21:07:16.410800Z info klog Config not found: /var/run/secrets/remote/config 2022-01-18T21:07:16.413501Z info klog Config not found: /var/run/secrets/remote/config 2022-01-18T21:07:16.427744Z info pkica Load signing key and cert from existing secret pingpong-smcp:istio-ca-secret 2022-01-18T21:07:16.428361Z info pkica Using existing public key: -----BEGIN CERTIFICATE----- MIIC/TCCAeWgAwIBAgIRAI1r+mCBGeIxQIkyjCjqvMcwDQYJKoZIhvcNAQELBQAw GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMjAxMTcxNzMxMzdaFw0zMjAx MTUxNzMxMzdaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3 DQEBAQUAA4IBDwAwggEKAoIBAQCuow2pOLOvp6cMxPn0pgx7wU7lWLwVm/eL11GJ NvVYU5KdlnGepikGo4v/8PzwWO5LuWrKm0bttrCwGK/e39AKEawtOQV//gJXMC2M CpWJ4ulXo5TYzgwK0Q8S89NMRyRGutFJ96Qpqaw3FEFrsl1Y4g96Y1GK8+j6V/W6 4Ew0i1F27Aa5AbEoPTOrZGNSEjj+C4rXr350FwCeA4dSzwkvr77FOdI7mPFZAk7v FIrjBX6K0tqN5KSGxfcZPpQLZ9mXfCQLPz+c8UXp34lXC9tpbmaKXdBfZg2JOfJg HT4v9qHKZ8el2UQWINZdE4IySezXMjM/wV0Zg9ePXeiPut/BAgMBAAGjQjBAMA4G A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRUdJV6Lj/6 4gjU/eFWniYT0LUm1DANBgkqhkiG9w0BAQsFAAOCAQEARYNu7YDZpUwzRiHBUABf NvBlX22/ltbz/uKcTEB3m1UpfDue9E4HyrOedG917ZFuIWTlVMmkENKZmLSuMbNt cNy7KDuqX+wHzOxtxe8Sv26f1vINuvBKijx2a7OONhuHRWJtdVu+fzax+WC3oWkT wEcpaFYcbrt/mtwPq/DIgj2srhFpJQ9mr/v0vjT6t+A1Jucmc73OMQJ9nGy7VH/q 2shayE2ocG/2s6WDTs+1L8IlAKXqHpHHH3IvZjST5XbVqTzYau1uzJR+dT6PQcQk mYlx/MZhncJ3w8psN96iV+ACiSLupHLz07kljdVMvOiRY8nLC2eh4hFmhO0tmLkn fA== -----END CERTIFICATE----- 2022-01-18T21:07:16.428401Z info rootcertrotator Set up back off time 2m53s to start rotator. 2022-01-18T21:07:16.428479Z info rootcertrotator Jitter is enabled, wait 2m53s before starting root cert rotator. 2022-01-18T21:07:16.831833Z info spiffe Added 1 certs to trust domain cluster.local in peer cert verifier 2022-01-18T21:07:16.832484Z warn federation ResyncPeriod not specified, defaulting to 1m0s component=federation-discovery-controller 2022-01-18T21:07:16.891011Z info klog Config not found: /var/run/secrets/remote/config 2022-01-18T21:07:16.896159Z info klog attempting to acquire leader lease pingpong-smcp/ior-leader... 2022-01-18T21:07:16.896254Z info federation Starting controller component=federation-discovery-controller 2022-01-18T21:07:16.896407Z info klog attempting to acquire leader lease pingpong-smcp/servicemesh-federation... 2022-01-18T21:07:16.896465Z info klog attempting to acquire leader lease pingpong-smcp/istio-namespace-controller-election... 2022-01-18T21:07:16.896658Z info federation Starting controller component=federation-exports-controller 2022-01-18T21:07:16.898375Z info federation Starting controller component=federation-imports-controller 2022-01-18T21:07:16.897842Z info klog Waiting for caches to sync for smmr 2022-01-18T21:07:16.898663Z info kube Starting Pilot K8S CRD controller 2022-01-18T21:07:16.899595Z info kube Pilot K8S CRD controller synced 3.476008ms 2022-01-18T21:07:16.996964Z info federation Controller synced in 100.306139ms component=federation-exports-controller 2022-01-18T21:07:16.997007Z info federation Starting workers component=federation-exports-controller 2022-01-18T21:07:16.996996Z info federation Controller synced in 100.758095ms component=federation-discovery-controller 2022-01-18T21:07:16.997024Z info federation Starting workers component=federation-discovery-controller 2022-01-18T21:07:16.998207Z info ads All caches have been synced up in 689.077413ms, marking server ready 2022-01-18T21:07:16.998332Z info federation starting federation service discovery at [::]:8188 component=federation-server 2022-01-18T21:07:17.000337Z info federation Controller synced in 103.642586ms component=federation-imports-controller 2022-01-18T21:07:17.000364Z info federation Starting workers component=federation-imports-controller 2022-01-18T21:07:17.000390Z info klog Caches are synced for smmr 2022-01-18T21:07:17.013492Z info smmr Cache synced for listener "dynamic-informers" 2022-01-18T21:07:17.013537Z info smmr Cache synced for listener "kubernetes-informers" 2022-01-18T21:07:17.013537Z info smmr Cache synced for listener "istio-infomrers" 2022-01-18T21:07:17.014139Z info smmr Cache synced for listener "service-apis-informers" 2022-01-18T21:07:17.014174Z info smmr Cache synced for listener "metadata-informers" 2022-01-18T21:07:17.014910Z info smmr Cache synced for listener "extensions-controller" 2022-01-18T21:07:17.021015Z info kube controller "networking.istio.io/v1alpha3/EnvoyFilter" is syncing... 2022-01-18T21:07:17.021531Z info ads Incremental push, service argocd-repo-server.pingpong.svc.cluster.local has no endpoints 2022-01-18T21:07:17.021570Z info ads Incremental push, service argocd-redis.pingpong.svc.cluster.local has no endpoints 2022-01-18T21:07:17.021604Z info ads Incremental push, service pong.pingpong.svc.cluster.local has no endpoints 2022-01-18T21:07:17.021626Z info ads Incremental push, service argocd-server-metrics.pingpong.svc.cluster.local has no endpoints 2022-01-18T21:07:17.021661Z info ads Incremental push, service argocd-server.pingpong.svc.cluster.local has no endpoints 2022-01-18T21:07:17.021678Z info ads Incremental push, service argocd-metrics.pingpong.svc.cluster.local has no endpoints 2022-01-18T21:07:17.021691Z info ads Incremental push, service argocd-dex-server.pingpong.svc.cluster.local has no endpoints 2022-01-18T21:07:17.092126Z info kube controller "security.istio.io/v1beta1/RequestAuthentication" is syncing... 2022-01-18T21:07:17.098667Z info kube controller "networking.istio.io/v1alpha3/VirtualService" is syncing... 2022-01-18T21:07:17.098728Z info kube controller "networking.istio.io/v1alpha3/VirtualService" is syncing... 2022-01-18T21:07:17.098743Z info kube controller "security.istio.io/v1beta1/PeerAuthentication" is syncing... 2022-01-18T21:07:17.098749Z info kube controller "networking.istio.io/v1alpha3/VirtualService" is syncing... 2022-01-18T21:07:17.098801Z info kube controller "security.istio.io/v1beta1/PeerAuthentication" is syncing... 2022-01-18T21:07:17.098815Z info kube controller "security.istio.io/v1beta1/PeerAuthentication" is syncing... 2022-01-18T21:07:17.194586Z info ads Incremental push, service jaeger-collector.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.194664Z info ads Incremental push, service prometheus.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.194690Z info ads Incremental push, service kiali.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.194723Z info ads Incremental push, service jaeger-agent.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.194767Z info ads Incremental push, service wasm-cacher-basic.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.194791Z info ads Incremental push, service istio-ingressgateway.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.194835Z info ads Incremental push, service jaeger-collector-headless.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.194875Z info ads Incremental push, service zipkin.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.194910Z info ads Incremental push, service istiod-basic.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.195109Z info ads Incremental push, service jaeger-query.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.195150Z info ads Incremental push, service grafana.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.195186Z info ads Incremental push, service istio-egressgateway.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.293148Z info kube controller "networking.istio.io/v1alpha3/EnvoyFilter" is syncing... 2022-01-18T21:07:17.293227Z info kube controller "networking.istio.io/v1alpha3/EnvoyFilter" is syncing... 2022-01-18T21:07:17.293257Z info kube controller "networking.istio.io/v1alpha3/EnvoyFilter" is syncing... 2022-01-18T21:07:17.296032Z info ads Push debounce stable[1] 40: 100.672974ms since last change, 277.360271ms since last push, full=true 2022-01-18T21:07:17.393501Z info ads Incremental push, service istiod-basic.pingpong-smcp.svc.cluster.local has no endpoints 2022-01-18T21:07:17.393543Z info ads XDS: Pushing:2022-01-18T21:07:17Z/1 Services:19 ConnectedEndpoints:0 Version:2022-01-18T21:07:17Z/1 2022-01-18T21:07:17.398735Z info ads Full push, new service pingpong-smcp/kiali.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.398856Z info ads Full push, new service pingpong-smcp/jaeger-collector.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.398913Z info ads Full push, new service pingpong-smcp/jaeger-agent.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.398982Z info ads Full push, new service pingpong-smcp/jaeger-collector-headless.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.399048Z info ads Full push, new service pingpong-smcp/zipkin.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.399118Z info ads Full push, new service pingpong-smcp/jaeger-query.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.399162Z info ads Full push, new service pingpong-smcp/prometheus.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.399221Z info ads Full push, new service pingpong-smcp/istiod-basic.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.399272Z info ads Full push, new service pingpong-smcp/grafana.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.399352Z info ads Full push, new service pingpong-smcp/wasm-cacher-basic.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.399407Z info ads Full push, new service pingpong-smcp/istio-egressgateway.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.399460Z info ads Full push, new service pingpong-smcp/istio-ingressgateway.pingpong-smcp.svc.cluster.local 2022-01-18T21:07:17.399742Z info ads Full push, new service pingpong/pong.pingpong.svc.cluster.local 2022-01-18T21:07:17.399860Z info ads Full push, new service pingpong/argocd-redis.pingpong.svc.cluster.local 2022-01-18T21:07:17.399911Z info ads Full push, new service pingpong/argocd-repo-server.pingpong.svc.cluster.local 2022-01-18T21:07:17.399975Z info ads Full push, new service pingpong/argocd-metrics.pingpong.svc.cluster.local 2022-01-18T21:07:17.400083Z info ads Full push, new service pingpong/argocd-dex-server.pingpong.svc.cluster.local 2022-01-18T21:07:17.400138Z info ads Full push, new service pingpong/argocd-server-metrics.pingpong.svc.cluster.local 2022-01-18T21:07:17.400201Z info ads Full push, new service pingpong/argocd-server.pingpong.svc.cluster.local 2022-01-18T21:07:17.709211Z info ads Push debounce stable[2] 105: 100.123707ms since last change, 315.636709ms since last push, full=true 2022-01-18T21:07:17.717995Z info ads XDS: Pushing:2022-01-18T21:07:17Z/2 Services:19 ConnectedEndpoints:0 Version:2022-01-18T21:07:17Z/2 2022-01-18T21:07:18.200975Z info ads Push debounce stable[3] 8: 100.251369ms since last change, 179.742261ms since last push, full=true 2022-01-18T21:07:18.201412Z info ads XDS: Pushing:2022-01-18T21:07:18Z/3 Services:19 ConnectedEndpoints:0 Version:2022-01-18T21:07:18Z/3 2022-01-18T21:07:18.394844Z info ads Push debounce stable[4] 3: 100.057217ms since last change, 100.611343ms since last push, full=true 2022-01-18T21:07:18.395052Z info ads XDS: Pushing:2022-01-18T21:07:18Z/4 Services:19 ConnectedEndpoints:0 Version:2022-01-18T21:07:18Z/4 2022-01-18T21:07:20.551879Z warn Insecure first-party-jwt option used to validate token; use third-party-jwt 2022-01-18T21:07:20.558153Z info ads ADS: new connection for node:sidecar~10.131.0.40~pong-v1-556df8f56f-979p2.pingpong~pingpong.svc.cluster.local-1 2022-01-18T21:07:20.560823Z info ads CDS: PUSH for node:pong-v1-556df8f56f-979p2.pingpong resources:46 size:39.6kB 2022-01-18T21:07:20.561784Z info ads EDS: PUSH for node:pong-v1-556df8f56f-979p2.pingpong resources:36 size:6.4kB empty:0 cached:0/36 2022-01-18T21:07:20.565789Z info ads LDS: PUSH for node:pong-v1-556df8f56f-979p2.pingpong resources:33 size:57.0kB 2022-01-18T21:07:20.566121Z info ads NDS: PUSH for node:pong-v1-556df8f56f-979p2.pingpong resources:1 size:1.9kB 2022-01-18T21:07:20.567119Z info ads RDS: PUSH for node:pong-v1-556df8f56f-979p2.pingpong resources:13 size:17.2kB 2022-01-18T21:07:21.061902Z warn Insecure first-party-jwt option used to validate token; use third-party-jwt 2022-01-18T21:07:21.061985Z warn Insecure first-party-jwt option used to validate token; use third-party-jwt 2022-01-18T21:07:21.061963Z warn Insecure first-party-jwt option used to validate token; use third-party-jwt 2022-01-18T21:07:21.062244Z warn Insecure first-party-jwt option used to validate token; use third-party-jwt 2022-01-18T21:07:21.067826Z info ads ADS: new connection for node:sidecar~10.131.0.41~pong-v2-5b45dc84d6-htfrn.pingpong~pingpong.svc.cluster.local-2 2022-01-18T21:07:21.068079Z info ads ADS: new connection for node:router~10.131.0.25~istio-ingressgateway-7d7f5596b7-jgw67.pingpong-smcp~pingpong-smcp.svc.cluster.local-5 2022-01-18T21:07:21.068176Z info ads ADS: new connection for node:router~10.131.0.4~istio-egressgateway-75bc54966d-ffnnw.pingpong-smcp~pingpong-smcp.svc.cluster.local-4 2022-01-18T21:07:21.068182Z info ads ADS: new connection for node:sidecar~10.131.0.42~ping-b6d78fd88-2r6dq.pingpong~pingpong.svc.cluster.local-3 2022-01-18T21:07:21.069599Z info ads CDS: PUSH for node:pong-v2-5b45dc84d6-htfrn.pingpong resources:46 size:39.6kB 2022-01-18T21:07:21.069760Z info ads CDS: PUSH for node:istio-egressgateway-75bc54966d-ffnnw.pingpong-smcp resources:42 size:39.3kB 2022-01-18T21:07:21.069780Z info ads EDS: PUSH for node:pong-v2-5b45dc84d6-htfrn.pingpong resources:36 size:6.4kB empty:0 cached:36/36 2022-01-18T21:07:21.069804Z info ads CDS: PUSH for node:istio-ingressgateway-7d7f5596b7-jgw67.pingpong-smcp resources:42 size:39.3kB 2022-01-18T21:07:21.069847Z info ads LDS: PUSH for node:istio-egressgateway-75bc54966d-ffnnw.pingpong-smcp resources:0 size:0B 2022-01-18T21:07:21.070013Z info ads CDS: PUSH for node:ping-b6d78fd88-2r6dq.pingpong resources:45 size:39.4kB 2022-01-18T21:07:21.070130Z info ads LDS: PUSH for node:istio-ingressgateway-7d7f5596b7-jgw67.pingpong-smcp resources:1 size:1.9kB 2022-01-18T21:07:21.071318Z info ads EDS: PUSH for node:ping-b6d78fd88-2r6dq.pingpong resources:36 size:6.4kB empty:0 cached:19/36 2022-01-18T21:07:21.071617Z info ads EDS: PUSH for node:istio-egressgateway-75bc54966d-ffnnw.pingpong-smcp resources:41 size:7.2kB empty:0 cached:24/41 2022-01-18T21:07:21.071678Z info ads EDS: PUSH for node:istio-ingressgateway-7d7f5596b7-jgw67.pingpong-smcp resources:41 size:7.2kB empty:0 cached:32/41 2022-01-18T21:07:21.071903Z info ads RDS: PUSH for node:istio-ingressgateway-7d7f5596b7-jgw67.pingpong-smcp resources:1 size:990B 2022-01-18T21:07:21.072078Z info ads LDS: PUSH for node:pong-v2-5b45dc84d6-htfrn.pingpong resources:33 size:57.0kB 2022-01-18T21:07:21.072314Z info ads NDS: PUSH for node:pong-v2-5b45dc84d6-htfrn.pingpong resources:1 size:1.9kB 2022-01-18T21:07:21.073240Z info ads RDS: PUSH for node:pong-v2-5b45dc84d6-htfrn.pingpong resources:13 size:17.2kB 2022-01-18T21:07:21.074164Z info ads LDS: PUSH for node:ping-b6d78fd88-2r6dq.pingpong resources:33 size:52.1kB 2022-01-18T21:07:21.074359Z info ads NDS: PUSH for node:ping-b6d78fd88-2r6dq.pingpong resources:1 size:1.9kB 2022-01-18T21:07:21.075061Z info ads RDS: PUSH for node:ping-b6d78fd88-2r6dq.pingpong resources:13 size:17.2kB 2022-01-18T21:07:21.110937Z warn Insecure first-party-jwt option used to validate token; use third-party-jwt 2022-01-18T21:07:21.116810Z info ads ADS: new connection for node:sidecar~10.128.2.30~pong-v1-556df8f56f-f84qd.pingpong~pingpong.svc.cluster.local-6 2022-01-18T21:07:21.118423Z info ads CDS: PUSH for node:pong-v1-556df8f56f-f84qd.pingpong resources:46 size:39.6kB 2022-01-18T21:07:21.119205Z info ads EDS: PUSH for node:pong-v1-556df8f56f-f84qd.pingpong resources:36 size:6.4kB empty:0 cached:0/36 2022-01-18T21:07:21.120948Z info ads LDS: PUSH for node:pong-v1-556df8f56f-f84qd.pingpong resources:33 size:57.0kB 2022-01-18T21:07:21.121187Z info ads NDS: PUSH for node:pong-v1-556df8f56f-f84qd.pingpong resources:1 size:1.9kB 2022-01-18T21:07:21.121854Z info ads RDS: PUSH for node:pong-v1-556df8f56f-f84qd.pingpong resources:13 size:17.2kB 2022-01-18T21:07:21.621547Z warn Insecure first-party-jwt option used to validate token; use third-party-jwt 2022-01-18T21:07:21.626436Z info ads ADS: new connection for node:sidecar~10.128.2.29~ping-b6d78fd88-zdps8.pingpong~pingpong.svc.cluster.local-7 2022-01-18T21:07:21.628169Z info ads CDS: PUSH for node:ping-b6d78fd88-zdps8.pingpong resources:45 size:39.4kB 2022-01-18T21:07:21.628362Z info ads EDS: PUSH for node:ping-b6d78fd88-zdps8.pingpong resources:36 size:6.4kB empty:0 cached:36/36 2022-01-18T21:07:21.630053Z info ads LDS: PUSH for node:ping-b6d78fd88-zdps8.pingpong resources:33 size:52.1kB 2022-01-18T21:07:21.630396Z info ads NDS: PUSH for node:ping-b6d78fd88-zdps8.pingpong resources:1 size:1.9kB 2022-01-18T21:07:21.641492Z info ads RDS: PUSH for node:ping-b6d78fd88-zdps8.pingpong resources:13 size:17.2kB 2022-01-18T21:07:54.340814Z info klog successfully acquired lease pingpong-smcp/servicemesh-federation 2022-01-18T21:07:56.255829Z info klog successfully acquired lease pingpong-smcp/ior-leader 2022-01-18T21:07:56.255994Z info ior Registering IOR component 2022-01-18T21:07:56.294395Z info smmr Cache synced for listener "ior" 2022-01-18T21:07:56.294503Z info klog Waiting for caches to sync for Gateways 2022-01-18T21:07:56.294533Z info klog Caches are synced for Gateways 2022-01-18T21:07:56.364257Z info ior Created route pingpong-smcp/pingpong-pingpong-36dcec7d7cbfa353 for hostname pingpong.evanstoner.net (gateway: pingpong/pingpong) 2022-01-18T21:07:56.413703Z info ior Created route pingpong-smcp/pingpong-pingpong-bd4c9b8af0356fe9 for hostname pingpong.apps.cluster-fvz84.fvz84.sandbox302.opentlc.com (gateway: pingpong/pingpong) 2022-01-18T21:07:56.848461Z info klog successfully acquired lease pingpong-smcp/istio-namespace-controller-election 2022-01-18T21:07:56.848713Z info smmr Cache synced for listener "namespace-controller" 2022-01-18T21:10:09.428973Z info rootcertrotator Jitter complete, start rotator.