Mar 8 12:06:17 dnsmasq[9]: started, version 2.66 cachesize 1000 Mar 8 12:06:17 dnsmasq[9]: compile time options: IPv6 GNU-getopt DBus no-i18n IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth Mar 8 12:06:17 dnsmasq[9]: using nameserver 9 #0 Mar 8 12:06:17 dnsmasq[9]: reading /etc/resolv.conf Mar 8 12:06:17 dnsmasq[9]: using nameserver 10.8.40.12#53 Mar 8 12:06:17 dnsmasq[9]: using nameserver 10.8.40.12#53 Mar 8 12:06:17 dnsmasq[9]: using nameserver 10.8.40.12#53 Mar 8 12:06:17 dnsmasq[9]: cleared cache 2017/03/08 12:06:18 [debug] 1#1: [lua] resolver.lua:59: parse_nameservers(): /etc/resolv.conf: search 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com nameserver 10.8.40.12 nameserver 10.8.40.12 options ndots:5 2017/03/08 12:06:18 [debug] 1#1: [lua] resolver.lua:66: parse_nameservers(): search 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:06:18 [debug] 1#1: [lua] resolver.lua:68: parse_nameservers(): search domain: 3scale.svc.cluster.local 2017/03/08 12:06:18 [debug] 1#1: [lua] resolver.lua:68: parse_nameservers(): search domain: svc.cluster.local 2017/03/08 12:06:18 [debug] 1#1: [lua] resolver.lua:68: parse_nameservers(): search domain: cluster.local 2017/03/08 12:06:18 [debug] 1#1: [lua] resolver.lua:68: parse_nameservers(): search domain: mw.lab.eng.rdu2.redhat.com 2017/03/08 12:06:18 [debug] 1#1: [lua] resolver.lua:68: parse_nameservers(): search domain: api-qe.eng.rdu2.redhat.com 2017/03/08 12:06:18 [info] 1#1: [lua] resolver.lua:93: init_nameservers(): adding 127.0.0.1:5353 as default nameserver 2017/03/08 12:06:18 [info] 1#1: [lua] resolver.lua:98: init_nameservers(): adding as search domain 2017/03/08 12:06:18 [info] 1#1: [lua] resolver.lua:98: init_nameservers(): adding 3scale.svc.cluster.local as search domain 2017/03/08 12:06:18 [info] 1#1: [lua] resolver.lua:98: init_nameservers(): adding svc.cluster.local as search domain 2017/03/08 12:06:18 [info] 1#1: [lua] resolver.lua:98: init_nameservers(): adding cluster.local as search domain 2017/03/08 12:06:18 [info] 1#1: [lua] resolver.lua:98: init_nameservers(): adding mw.lab.eng.rdu2.redhat.com as search domain 2017/03/08 12:06:18 [info] 1#1: [lua] resolver.lua:98: init_nameservers(): adding api-qe.eng.rdu2.redhat.com as search domain 2017/03/08 12:06:18 [info] 1#1: [lua] configuration_loader.lua:150: new(): using lazy configuration loader 2017/03/08 12:06:18 [notice] 1#1: using the "epoll" event method 2017/03/08 12:06:18 [notice] 1#1: openresty/1.11.2.2 2017/03/08 12:06:18 [notice] 1#1: built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) 2017/03/08 12:06:18 [notice] 1#1: OS: Linux 3.10.0-514.6.1.el7.x86_64 2017/03/08 12:06:18 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2017/03/08 12:06:18 [notice] 1#1: start worker processes 2017/03/08 12:06:18 [notice] 1#1: start worker process 20 2017/03/08 12:06:34 [debug] 20#20: *2 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:06:36 [debug] 20#20: *3 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:06:42 [info] 20#20: *4 [lua] configuration_loader.lua:130: rewrite(): lazy loading configuration for: api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com" 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:118: fetch_answers(): resolver cache miss system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:184: get(): resolver cache miss: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:194: lookup(): resolver query: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: 3scale.svc.cluster.local query: system-3scale.3scale.svc.cluster.local 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:213: lookup(): resolver query: system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:70: store(): resolver cache write system-3scale.3scale.svc.cluster.local with TLL 30 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:260: get_servers(): query for system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] http.lua:568: send_request(): GET /admin/api/services.json HTTP/1.1 Host: system-3scale:3000 User-Agent: APIcast/3.0.0-pre (Linux; x64; env:sandbox) APIcast/3.0.0-pre Authorization: Basic dzh4d3h3bmI6 Connection: Keep-Alive 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:118: fetch_answers(): resolver cache miss system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:184: get(): resolver cache miss: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:194: lookup(): resolver query: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: 3scale.svc.cluster.local query: system-3scale.3scale.svc.cluster.local 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:213: lookup(): resolver query: system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:70: store(): resolver cache write system-3scale.3scale.svc.cluster.local with TLL 30 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:260: get_servers(): query for system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] http.lua:568: send_request(): GET /admin/api/services/2/proxy/configs/sandbox/latest.json HTTP/1.1 Host: system-3scale:3000 User-Agent: APIcast/3.0.0-pre (Linux; x64; env:sandbox) APIcast/3.0.0-pre Authorization: Basic dzh4d3h3bmI6 Connection: Keep-Alive 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:118: fetch_answers(): resolver cache miss system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:184: get(): resolver cache miss: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:194: lookup(): resolver query: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: 3scale.svc.cluster.local query: system-3scale.3scale.svc.cluster.local 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:213: lookup(): resolver query: system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:70: store(): resolver cache write system-3scale.3scale.svc.cluster.local with TLL 30 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:260: get_servers(): query for system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] http.lua:568: send_request(): GET /admin/api/services/3/proxy/configs/sandbox/latest.json HTTP/1.1 Host: system-3scale:3000 User-Agent: APIcast/3.0.0-pre (Linux; x64; env:sandbox) APIcast/3.0.0-pre Authorization: Basic dzh4d3h3bmI6 Connection: Keep-Alive 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:118: fetch_answers(): resolver cache miss system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:184: get(): resolver cache miss: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:194: lookup(): resolver query: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: 3scale.svc.cluster.local query: system-3scale.3scale.svc.cluster.local 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:213: lookup(): resolver query: system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:70: store(): resolver cache write system-3scale.3scale.svc.cluster.local with TLL 30 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:260: get_servers(): query for system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] http.lua:568: send_request(): GET /admin/api/services/4/proxy/configs/sandbox/latest.json HTTP/1.1 Host: system-3scale:3000 User-Agent: APIcast/3.0.0-pre (Linux; x64; env:sandbox) APIcast/3.0.0-pre Authorization: Basic dzh4d3h3bmI6 Connection: Keep-Alive 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:118: fetch_answers(): resolver cache miss system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:184: get(): resolver cache miss: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:194: lookup(): resolver query: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: 3scale.svc.cluster.local query: system-3scale.3scale.svc.cluster.local 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:213: lookup(): resolver query: system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:70: store(): resolver cache write system-3scale.3scale.svc.cluster.local with TLL 30 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:260: get_servers(): query for system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] http.lua:568: send_request(): GET /admin/api/services/5/proxy/configs/sandbox/latest.json HTTP/1.1 Host: system-3scale:3000 User-Agent: APIcast/3.0.0-pre (Linux; x64; env:sandbox) APIcast/3.0.0-pre Authorization: Basic dzh4d3h3bmI6 Connection: Keep-Alive 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:118: fetch_answers(): resolver cache miss system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:184: get(): resolver cache miss: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:194: lookup(): resolver query: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: 3scale.svc.cluster.local query: system-3scale.3scale.svc.cluster.local 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:213: lookup(): resolver query: system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:70: store(): resolver cache write system-3scale.3scale.svc.cluster.local with TLL 30 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:260: get_servers(): query for system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] http.lua:568: send_request(): GET /admin/api/services/6/proxy/configs/sandbox/latest.json HTTP/1.1 Host: system-3scale:3000 User-Agent: APIcast/3.0.0-pre (Linux; x64; env:sandbox) APIcast/3.0.0-pre Authorization: Basic dzh4d3h3bmI6 Connection: Keep-Alive 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:118: fetch_answers(): resolver cache miss system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:184: get(): resolver cache miss: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:194: lookup(): resolver query: system-3scale 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: query: system-3scale. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: system-3scale search: 3scale.svc.cluster.local query: system-3scale.3scale.svc.cluster.local 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: system-3scale.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:213: lookup(): resolver query: system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:70: store(): resolver cache write system-3scale.3scale.svc.cluster.local with TLL 30 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:260: get_servers(): query for system-3scale finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] http.lua:568: send_request(): GET /admin/api/services/7/proxy/configs/sandbox/latest.json HTTP/1.1 Host: system-3scale:3000 User-Agent: APIcast/3.0.0-pre (Linux; x64; env:sandbox) APIcast/3.0.0-pre Authorization: Basic dzh4d3h3bmI6 Connection: Keep-Alive 2017/03/08 12:06:42 [info] 20#20: *4 [lua] configuration_store.lua:92: configure(): added service 2 configuration with hosts: api-2-3scale.cloudapps.api-qe.eng.rdu2.redhat.com, api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com ttl: 0, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] configuration_store.lua:92: configure(): added service 3 configuration with hosts: auth-app-key.cloudapps.api-qe.eng.rdu2.redhat.com, auth-app-key-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com ttl: 0, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] configuration_store.lua:92: configure(): added service 4 configuration with hosts: auth-oauth-prod.cloudapps.api-qe.eng.rdu2.redhat.com, auth-oauth-stage.cloudapps.api-qe.eng.rdu2.redhat.com ttl: 0, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] configuration_store.lua:92: configure(): added service 5 configuration with hosts: api-hit-limit.cloudapps.api-qe.eng.rdu2.redhat.com, api-hit-limit-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com ttl: 0, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] configuration_store.lua:92: configure(): added service 6 configuration with hosts: auth-user-key.cloudapps.api-qe.eng.rdu2.redhat.com, auth-user-key-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com ttl: 0, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] configuration_store.lua:92: configure(): added service 7 configuration with hosts: headers-auth.cloudapps.api-qe.eng.rdu2.redhat.com, headers-auth-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com ttl: 0, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com" 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:118: fetch_answers(): resolver cache miss echo-api.3scale.net 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:184: get(): resolver cache miss: echo-api.3scale.net 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:194: lookup(): resolver query: echo-api.3scale.net 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: echo-api.3scale.net search: query: echo-api.3scale.net. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: echo-api.3scale.net. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:213: lookup(): resolver query: echo-api.3scale.net finished with 2 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:70: store(): resolver cache write echo-api.3scale.net with TLL 363 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:70: store(): resolver cache write gateway.3scale.net with TLL 60 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:260: get_servers(): query for echo-api.3scale.net finished with 2 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 30 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] configuration.lua:175: extract_usage(): [mapping] service 2 has 1 rules 2017/03/08 12:06:42 [info] 20#20: *4 [lua] proxy.lua:368: access(): usage: usage%5Bhits%5D=1 credentials: user_key=fd11d39381cc90f25e0e2986ae30b297, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] proxy.lua:222: authrep(): apicast cache miss key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] proxy.lua:168: get(): [http] requesting /threescale_authrep, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] proxy.lua:171: get(): [ctx] copying backend_upstream of size: 1 2017/03/08 12:06:42 [info] 20#20: *4 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", subrequest: "/threescale_authrep", host: "echo-api.3scale.net" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] proxy.lua:177: get(): [http] http://backend_upstream/transactions/authrep.xml?service_token=2726f6e57378a4b83bd2fb5538fa90f2ecd5a767bb3ef595543d43b426627a61&service_id=2&usage%5Bhits%5D=1&user_key=fd11d39381cc90f25e0e2986ae30b297 (200) while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:06:42 [debug] 20#20: *4 [lua] proxy.lua:225: authrep(): [backend] response status: 200 body: trueBasic 2017/03/08 12:06:42 [info] 20#20: *4 [lua] proxy.lua:229: authrep(): apicast cache write key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] balancer.lua:110: set_peer(): balancer set peer 52.54.210.35:443 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] proxy.lua:436: post_action(): [async] skipping after action, no cached key while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:06:42 [info] 20#20: *4 [lua] log_by_lua(apicast.conf:48):3: [authrep] /?user_key=fd11d39381cc90f25e0e2986ae30b297 200 while logging request, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" [08/Mar/2017:12:06:42 +0000] api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:43748 "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1" 200 755 (0.229) 0 2017/03/08 12:06:42 [info] 20#20: *4 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:06:44 [debug] 20#20: *16 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:06:44 [debug] 20#20: *15 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] cache.lua:111: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] cache.lua:111: fetch_answers(): resolver cache read gateway.3scale.net 1 entries 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] cache.lua:186: get(): resolver cache hit: echo-api.3scale.net 52.54.210.35 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:260: get_servers(): query for echo-api.3scale.net finished with 2 answers 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 27 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] configuration.lua:175: extract_usage(): [mapping] service 2 has 1 rules 2017/03/08 12:06:45 [info] 20#20: *17 [lua] proxy.lua:368: access(): usage: usage%5Bhits%5D=1 credentials: user_key=fd11d39381cc90f25e0e2986ae30b297, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] proxy.lua:219: authrep(): apicast cache hit key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 2017/03/08 12:06:45 [info] 20#20: *17 [lua] balancer.lua:110: set_peer(): balancer set peer 52.54.210.35:443 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:06:45 [info] 20#20: *17 [lua] proxy.lua:416: post_action(): [async] reporting to backend asynchronously, cached_key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 27 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:06:45 [info] 20#20: *17 [lua] proxy.lua:168: get(): [http] requesting /threescale_authrep?log= while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:06:45 [debug] 20#20: *17 [lua] proxy.lua:171: get(): [ctx] copying backend_upstream of size: 1 2017/03/08 12:06:45 [info] 20#20: *17 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", subrequest: "/threescale_authrep", host: "echo-api.3scale.net" 2017/03/08 12:06:45 [info] 20#20: *17 [lua] proxy.lua:177: get(): [http] http://backend_upstream/transactions/authrep.xml?service_token=2726f6e57378a4b83bd2fb5538fa90f2ecd5a767bb3ef595543d43b426627a61&service_id=2&usage%5Bhits%5D=1&user_key=fd11d39381cc90f25e0e2986ae30b297& (200) while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:06:45 [info] 20#20: *17 [lua] log_by_lua(apicast.conf:48):3: [authrep] /?user_key=fd11d39381cc90f25e0e2986ae30b297 200 while logging request, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" [08/Mar/2017:12:06:45 +0000] api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:43768 "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1" 200 755 (0.029) 0.003 2017/03/08 12:06:45 [info] 20#20: *17 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:06:54 [debug] 20#20: *20 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] cache.lua:111: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] cache.lua:111: fetch_answers(): resolver cache read gateway.3scale.net 1 entries 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] cache.lua:186: get(): resolver cache hit: echo-api.3scale.net 52.54.210.35 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:260: get_servers(): query for echo-api.3scale.net finished with 2 answers 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 12 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] configuration.lua:175: extract_usage(): [mapping] service 2 has 1 rules 2017/03/08 12:07:00 [info] 20#20: *21 [lua] proxy.lua:368: access(): usage: usage%5Bhits%5D=1 credentials: user_key=fd11d39381cc90f25e0e2986ae30b297, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] proxy.lua:219: authrep(): apicast cache hit key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 2017/03/08 12:07:00 [info] 20#20: *21 [lua] balancer.lua:110: set_peer(): balancer set peer 52.54.210.35:443 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:00 [info] 20#20: *21 [lua] proxy.lua:416: post_action(): [async] reporting to backend asynchronously, cached_key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 12 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:00 [info] 20#20: *21 [lua] proxy.lua:168: get(): [http] requesting /threescale_authrep?log= while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:00 [debug] 20#20: *21 [lua] proxy.lua:171: get(): [ctx] copying backend_upstream of size: 1 2017/03/08 12:07:00 [info] 20#20: *21 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", subrequest: "/threescale_authrep", host: "echo-api.3scale.net" 2017/03/08 12:07:00 [info] 20#20: *21 [lua] proxy.lua:177: get(): [http] http://backend_upstream/transactions/authrep.xml?service_token=2726f6e57378a4b83bd2fb5538fa90f2ecd5a767bb3ef595543d43b426627a61&service_id=2&usage%5Bhits%5D=1&user_key=fd11d39381cc90f25e0e2986ae30b297& (200) while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:00 [info] 20#20: *21 [lua] log_by_lua(apicast.conf:48):3: [authrep] /?user_key=fd11d39381cc90f25e0e2986ae30b297 200 while logging request, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" [08/Mar/2017:12:07:00 +0000] api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:43886 "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1" 200 755 (0.034) 0.004 2017/03/08 12:07:00 [info] 20#20: *21 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:07:04 [debug] 20#20: *24 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] cache.lua:111: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] cache.lua:111: fetch_answers(): resolver cache read gateway.3scale.net 1 entries 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] cache.lua:186: get(): resolver cache hit: echo-api.3scale.net 52.54.210.35 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:260: get_servers(): query for echo-api.3scale.net finished with 2 answers 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 30 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] configuration.lua:175: extract_usage(): [mapping] service 2 has 1 rules 2017/03/08 12:07:12 [info] 20#20: *25 [lua] proxy.lua:368: access(): usage: usage%5Bhits%5D=1 credentials: user_key=fd11d39381cc90f25e0e2986ae30b297, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] proxy.lua:219: authrep(): apicast cache hit key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 2017/03/08 12:07:12 [info] 20#20: *25 [lua] balancer.lua:110: set_peer(): balancer set peer 52.54.210.35:443 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:12 [info] 20#20: *25 [lua] proxy.lua:416: post_action(): [async] reporting to backend asynchronously, cached_key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 30 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:12 [info] 20#20: *25 [lua] proxy.lua:168: get(): [http] requesting /threescale_authrep?log= while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:12 [debug] 20#20: *25 [lua] proxy.lua:171: get(): [ctx] copying backend_upstream of size: 1 2017/03/08 12:07:12 [info] 20#20: *25 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", subrequest: "/threescale_authrep", host: "echo-api.3scale.net" 2017/03/08 12:07:12 [info] 20#20: *25 [lua] proxy.lua:177: get(): [http] http://backend_upstream/transactions/authrep.xml?service_token=2726f6e57378a4b83bd2fb5538fa90f2ecd5a767bb3ef595543d43b426627a61&service_id=2&usage%5Bhits%5D=1&user_key=fd11d39381cc90f25e0e2986ae30b297& (200) while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:12 [info] 20#20: *25 [lua] log_by_lua(apicast.conf:48):3: [authrep] /?user_key=fd11d39381cc90f25e0e2986ae30b297 200 while logging request, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" [08/Mar/2017:12:07:12 +0000] api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:43982 "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1" 200 884 (0.033) 0.003 2017/03/08 12:07:14 [debug] 20#20: *29 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:07:14 [debug] 20#20: *28 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:07:22 [info] 20#20: *25 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:07:24 [debug] 20#20: *30 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:07:34 [debug] 20#20: *31 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] cache.lua:111: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] cache.lua:111: fetch_answers(): resolver cache read gateway.3scale.net 1 entries 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] cache.lua:186: get(): resolver cache hit: echo-api.3scale.net 52.54.210.35 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:260: get_servers(): query for echo-api.3scale.net finished with 2 answers 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 5 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] configuration.lua:175: extract_usage(): [mapping] service 2 has 1 rules 2017/03/08 12:07:37 [info] 20#20: *32 [lua] proxy.lua:368: access(): usage: usage%5Bhits%5D=1 credentials: user_key=fd11d39381cc90f25e0e2986ae30b297, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] proxy.lua:219: authrep(): apicast cache hit key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 2017/03/08 12:07:37 [info] 20#20: *32 [lua] balancer.lua:110: set_peer(): balancer set peer 52.54.210.35:443 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:37 [info] 20#20: *32 [lua] proxy.lua:416: post_action(): [async] reporting to backend asynchronously, cached_key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 5 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:37 [info] 20#20: *32 [lua] proxy.lua:168: get(): [http] requesting /threescale_authrep?log= while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:37 [debug] 20#20: *32 [lua] proxy.lua:171: get(): [ctx] copying backend_upstream of size: 1 2017/03/08 12:07:37 [info] 20#20: *32 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", subrequest: "/threescale_authrep", host: "echo-api.3scale.net" 2017/03/08 12:07:37 [info] 20#20: *32 [lua] proxy.lua:177: get(): [http] http://backend_upstream/transactions/authrep.xml?service_token=2726f6e57378a4b83bd2fb5538fa90f2ecd5a767bb3ef595543d43b426627a61&service_id=2&usage%5Bhits%5D=1&user_key=fd11d39381cc90f25e0e2986ae30b297& (200) while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:37 [info] 20#20: *32 [lua] log_by_lua(apicast.conf:48):3: [authrep] /?user_key=fd11d39381cc90f25e0e2986ae30b297 200 while logging request, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" [08/Mar/2017:12:07:37 +0000] api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:44160 "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1" 200 755 (0.031) 0.004 2017/03/08 12:07:37 [info] 20#20: *32 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] cache.lua:111: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] cache.lua:111: fetch_answers(): resolver cache read gateway.3scale.net 1 entries 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] cache.lua:186: get(): resolver cache hit: echo-api.3scale.net 52.54.210.35 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:260: get_servers(): query for echo-api.3scale.net finished with 2 answers 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 4 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] configuration.lua:175: extract_usage(): [mapping] service 2 has 1 rules 2017/03/08 12:07:38 [info] 20#20: *36 [lua] proxy.lua:368: access(): usage: usage%5Bhits%5D=1 credentials: user_key=fd11d39381cc90f25e0e2986ae30b297, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] proxy.lua:219: authrep(): apicast cache hit key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 2017/03/08 12:07:38 [info] 20#20: *36 [lua] balancer.lua:110: set_peer(): balancer set peer 52.54.210.35:443 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:38 [info] 20#20: *36 [lua] proxy.lua:416: post_action(): [async] reporting to backend asynchronously, cached_key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 4 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:38 [info] 20#20: *36 [lua] proxy.lua:168: get(): [http] requesting /threescale_authrep?log= while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:38 [debug] 20#20: *36 [lua] proxy.lua:171: get(): [ctx] copying backend_upstream of size: 1 2017/03/08 12:07:38 [info] 20#20: *36 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", subrequest: "/threescale_authrep", host: "echo-api.3scale.net" 2017/03/08 12:07:38 [info] 20#20: *36 [lua] proxy.lua:177: get(): [http] http://backend_upstream/transactions/authrep.xml?service_token=2726f6e57378a4b83bd2fb5538fa90f2ecd5a767bb3ef595543d43b426627a61&service_id=2&usage%5Bhits%5D=1&user_key=fd11d39381cc90f25e0e2986ae30b297& (200) while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:38 [info] 20#20: *36 [lua] log_by_lua(apicast.conf:48):3: [authrep] /?user_key=fd11d39381cc90f25e0e2986ae30b297 200 while logging request, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" [08/Mar/2017:12:07:38 +0000] api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:44178 "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1" 200 755 (0.029) 0.002 2017/03/08 12:07:39 [info] 20#20: *36 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] cache.lua:111: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] cache.lua:111: fetch_answers(): resolver cache read gateway.3scale.net 1 entries 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] cache.lua:186: get(): resolver cache hit: echo-api.3scale.net 52.54.210.35 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:260: get_servers(): query for echo-api.3scale.net finished with 2 answers 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 2 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] configuration.lua:175: extract_usage(): [mapping] service 2 has 1 rules 2017/03/08 12:07:40 [info] 20#20: *39 [lua] proxy.lua:368: access(): usage: usage%5Bhits%5D=1 credentials: user_key=fd11d39381cc90f25e0e2986ae30b297, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] proxy.lua:219: authrep(): apicast cache hit key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 2017/03/08 12:07:40 [info] 20#20: *39 [lua] balancer.lua:110: set_peer(): balancer set peer 52.54.210.35:443 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:40 [info] 20#20: *39 [lua] proxy.lua:416: post_action(): [async] reporting to backend asynchronously, cached_key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 2 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:40 [info] 20#20: *39 [lua] proxy.lua:168: get(): [http] requesting /threescale_authrep?log= while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:40 [debug] 20#20: *39 [lua] proxy.lua:171: get(): [ctx] copying backend_upstream of size: 1 2017/03/08 12:07:40 [info] 20#20: *39 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", subrequest: "/threescale_authrep", host: "echo-api.3scale.net" 2017/03/08 12:07:40 [info] 20#20: *39 [lua] proxy.lua:177: get(): [http] http://backend_upstream/transactions/authrep.xml?service_token=2726f6e57378a4b83bd2fb5538fa90f2ecd5a767bb3ef595543d43b426627a61&service_id=2&usage%5Bhits%5D=1&user_key=fd11d39381cc90f25e0e2986ae30b297& (200) while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:40 [info] 20#20: *39 [lua] log_by_lua(apicast.conf:48):3: [authrep] /?user_key=fd11d39381cc90f25e0e2986ae30b297 200 while logging request, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.54.210.35:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" [08/Mar/2017:12:07:40 +0000] api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:44192 "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1" 200 755 (0.029) 0.002 2017/03/08 12:07:40 [info] 20#20: *39 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:07:44 [debug] 20#20: *42 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:07:44 [debug] 20#20: *43 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:111: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:118: fetch_answers(): resolver cache miss gateway.3scale.net 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:186: get(): resolver cache hit: echo-api.3scale.net 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:194: lookup(): resolver query: echo-api.3scale.net 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:204: lookup(): resolver query: echo-api.3scale.net search: query: echo-api.3scale.net. 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:61: resolver query: echo-api.3scale.net. nameserver: 127.0.0.1:5353 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:213: lookup(): resolver query: echo-api.3scale.net finished with 2 answers 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:70: store(): resolver cache write echo-api.3scale.net with TLL 294 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:70: store(): resolver cache write gateway.3scale.net with TLL 60 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:260: get_servers(): query for echo-api.3scale.net finished with 2 answers 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 30 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] configuration.lua:175: extract_usage(): [mapping] service 2 has 1 rules 2017/03/08 12:07:51 [info] 20#20: *44 [lua] proxy.lua:368: access(): usage: usage%5Bhits%5D=1 credentials: user_key=fd11d39381cc90f25e0e2986ae30b297, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] proxy.lua:219: authrep(): apicast cache hit key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 2017/03/08 12:07:51 [info] 20#20: *44 [lua] balancer.lua:110: set_peer(): balancer set peer 52.5.224.156:443 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:51 [info] 20#20: *44 [lua] proxy.lua:416: post_action(): [async] reporting to backend asynchronously, cached_key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 30 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:51 [info] 20#20: *44 [lua] proxy.lua:168: get(): [http] requesting /threescale_authrep?log= while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:51 [debug] 20#20: *44 [lua] proxy.lua:171: get(): [ctx] copying backend_upstream of size: 1 2017/03/08 12:07:51 [info] 20#20: *44 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", subrequest: "/threescale_authrep", host: "echo-api.3scale.net" 2017/03/08 12:07:51 [info] 20#20: *44 [lua] proxy.lua:177: get(): [http] http://backend_upstream/transactions/authrep.xml?service_token=2726f6e57378a4b83bd2fb5538fa90f2ecd5a767bb3ef595543d43b426627a61&service_id=2&usage%5Bhits%5D=1&user_key=fd11d39381cc90f25e0e2986ae30b297& (200) while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:51 [info] 20#20: *44 [lua] log_by_lua(apicast.conf:48):3: [authrep] /?user_key=fd11d39381cc90f25e0e2986ae30b297 200 while logging request, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" [08/Mar/2017:12:07:51 +0000] api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:44290 "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1" 200 756 (0.113) 0.005 2017/03/08 12:07:51 [info] 20#20: *44 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:07:54 [debug] 20#20: *48 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] cache.lua:111: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] cache.lua:111: fetch_answers(): resolver cache read gateway.3scale.net 1 entries 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] cache.lua:186: get(): resolver cache hit: echo-api.3scale.net 52.5.224.156 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:260: get_servers(): query for echo-api.3scale.net finished with 2 answers 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 24 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] configuration.lua:175: extract_usage(): [mapping] service 2 has 1 rules 2017/03/08 12:07:57 [info] 20#20: *49 [lua] proxy.lua:368: access(): usage: usage%5Bhits%5D=1 credentials: user_key=fd11d39381cc90f25e0e2986ae30b297, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] proxy.lua:219: authrep(): apicast cache hit key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 2017/03/08 12:07:57 [info] 20#20: *49 [lua] balancer.lua:110: set_peer(): balancer set peer 52.5.224.156:443 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:57 [info] 20#20: *49 [lua] proxy.lua:416: post_action(): [async] reporting to backend asynchronously, cached_key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 24 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:57 [info] 20#20: *49 [lua] proxy.lua:168: get(): [http] requesting /threescale_authrep?log= while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:57 [debug] 20#20: *49 [lua] proxy.lua:171: get(): [ctx] copying backend_upstream of size: 1 2017/03/08 12:07:57 [info] 20#20: *49 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", subrequest: "/threescale_authrep", host: "echo-api.3scale.net" 2017/03/08 12:07:57 [info] 20#20: *49 [lua] proxy.lua:177: get(): [http] http://backend_upstream/transactions/authrep.xml?service_token=2726f6e57378a4b83bd2fb5538fa90f2ecd5a767bb3ef595543d43b426627a61&service_id=2&usage%5Bhits%5D=1&user_key=fd11d39381cc90f25e0e2986ae30b297& (200) while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:57 [info] 20#20: *49 [lua] log_by_lua(apicast.conf:48):3: [authrep] /?user_key=fd11d39381cc90f25e0e2986ae30b297 200 while logging request, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" [08/Mar/2017:12:07:57 +0000] api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:44334 "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1" 200 756 (0.031) 0.003 2017/03/08 12:07:58 [info] 20#20: *49 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] cache.lua:111: fetch_answers(): resolver cache read echo-api.3scale.net 1 entries 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] cache.lua:111: fetch_answers(): resolver cache read gateway.3scale.net 1 entries 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] cache.lua:186: get(): resolver cache hit: echo-api.3scale.net 52.5.224.156 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:260: get_servers(): query for echo-api.3scale.net finished with 2 answers 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 23 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] configuration.lua:175: extract_usage(): [mapping] service 2 has 1 rules 2017/03/08 12:07:58 [info] 20#20: *52 [lua] proxy.lua:368: access(): usage: usage%5Bhits%5D=1 credentials: user_key=fd11d39381cc90f25e0e2986ae30b297, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] proxy.lua:219: authrep(): apicast cache hit key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 2017/03/08 12:07:58 [info] 20#20: *52 [lua] balancer.lua:110: set_peer(): balancer set peer 52.5.224.156:443 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", host: "echo-api.3scale.net" 2017/03/08 12:07:58 [info] 20#20: *52 [lua] proxy.lua:416: post_action(): [async] reporting to backend asynchronously, cached_key: 2:fd11d39381cc90f25e0e2986ae30b297:usage%5Bhits%5D=1 while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 23 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:07:58 [info] 20#20: *52 [lua] proxy.lua:168: get(): [http] requesting /threescale_authrep?log= while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:58 [debug] 20#20: *52 [lua] proxy.lua:171: get(): [ctx] copying backend_upstream of size: 1 2017/03/08 12:07:58 [info] 20#20: *52 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", subrequest: "/threescale_authrep", host: "echo-api.3scale.net" 2017/03/08 12:07:58 [info] 20#20: *52 [lua] proxy.lua:177: get(): [http] http://backend_upstream/transactions/authrep.xml?service_token=2726f6e57378a4b83bd2fb5538fa90f2ecd5a767bb3ef595543d43b426627a61&service_id=2&usage%5Bhits%5D=1&user_key=fd11d39381cc90f25e0e2986ae30b297& (200) while sending to client, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" 2017/03/08 12:07:58 [info] 20#20: *52 [lua] log_by_lua(apicast.conf:48):3: [authrep] /?user_key=fd11d39381cc90f25e0e2986ae30b297 200 while logging request, client: 10.128.0.1, server: _, request: "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1", upstream: "https://52.5.224.156:443/?user_key=fd11d39381cc90f25e0e2986ae30b297", host: "echo-api.3scale.net" [08/Mar/2017:12:07:58 +0000] api-2-3scale-staging.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:44348 "GET /?user_key=fd11d39381cc90f25e0e2986ae30b297 HTTP/1.1" 200 756 (0.031) 0.002 2017/03/08 12:07:59 [info] 20#20: *52 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:08:04 [debug] 20#20: *55 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:08:14 [debug] 20#20: *56 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:08:14 [debug] 20#20: *57 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:08:24 [debug] 20#20: *58 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:118: fetch_answers(): resolver cache miss httpbin.org 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:184: get(): resolver cache miss: httpbin.org 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:194: lookup(): resolver query: httpbin.org 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:204: lookup(): resolver query: httpbin.org search: query: httpbin.org. 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:61: resolver query: httpbin.org. nameserver: 127.0.0.1:5353 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:213: lookup(): resolver query: httpbin.org finished with 1 answers 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:70: store(): resolver cache write httpbin.org with TLL 279 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:260: get_servers(): query for httpbin.org finished with 1 answers 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 30 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] proxy.lua:320: call(): apicast oauth flow 2017/03/08 12:08:30 [info] 20#20: *59 [lua] balancer.lua:110: set_peer(): balancer set peer 172.30.61.83:3000 while connecting to upstream, client: 10.128.0.1, server: _, request: "GET /authorize?state=68a76266-365c-46ef-9b33-c058b4387d40&response_type=code&client_id=auth-oauth123456789&redirect_uri=http%3A%2F%2Flocalhost%3A3001%2Fcallback&scope= HTTP/1.1", subrequest: "/_threescale/check_credentials", host: "httpbin.org", referrer: "http://localhost:3001/" 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] authorize.lua:120: access(): oauth params valid 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:118: fetch_answers(): resolver cache miss 127.0.0.1 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:184: get(): resolver cache miss: 127.0.0.1 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:194: lookup(): resolver query: 127.0.0.1 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:199: lookup(): host is ip address: 127.0.0.1 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:213: lookup(): resolver query: 127.0.0.1 finished with 1 answers 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:70: store(): resolver cache write 127.0.0.1 with TLL -1 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:260: get_servers(): query for 127.0.0.1 finished with 1 answers 2017/03/08 12:08:30 [error] 20#20: *59 connect() failed (111: Connection refused) while sending to client, client: 10.128.0.1, server: _, request: "GET /authorize?state=68a76266-365c-46ef-9b33-c058b4387d40&response_type=code&client_id=auth-oauth123456789&redirect_uri=http%3A%2F%2Flocalhost%3A3001%2Fcallback&scope= HTTP/1.1", host: "httpbin.org", referrer: "http://localhost:3001/" 2017/03/08 12:08:30 [info] 20#20: *59 [lua] proxy.lua:436: post_action(): [async] skipping after action, no cached key while sending to client, client: 10.128.0.1, server: _, request: "GET /authorize?state=68a76266-365c-46ef-9b33-c058b4387d40&response_type=code&client_id=auth-oauth123456789&redirect_uri=http%3A%2F%2Flocalhost%3A3001%2Fcallback&scope= HTTP/1.1", host: "httpbin.org", referrer: "http://localhost:3001/" 2017/03/08 12:08:30 [info] 20#20: *59 [lua] log_by_lua(apicast.conf:48):3: [authrep] /authorize?state=68a76266-365c-46ef-9b33-c058b4387d40&response_type=code&client_id=auth-oauth123456789&redirect_uri=http%3A%2F%2Flocalhost%3A3001%2Fcallback&scope= 500 while logging request, client: 10.128.0.1, server: _, request: "GET /authorize?state=68a76266-365c-46ef-9b33-c058b4387d40&response_type=code&client_id=auth-oauth123456789&redirect_uri=http%3A%2F%2Flocalhost%3A3001%2Fcallback&scope= HTTP/1.1", host: "httpbin.org", referrer: "http://localhost:3001/" [08/Mar/2017:12:08:30 +0000] auth-oauth-stage.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:44566 "GET /authorize?state=68a76266-365c-46ef-9b33-c058b4387d40&response_type=code&client_id=auth-oauth123456789&redirect_uri=http%3A%2F%2Flocalhost%3A3001%2Fcallback&scope= HTTP/1.1" 500 75 (0.011) 0 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:126: new(): resolver search domains: 3scale.svc.cluster.local svc.cluster.local cluster.local mw.lab.eng.rdu2.redhat.com api-qe.eng.rdu2.redhat.com 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:111: fetch_answers(): resolver cache read httpbin.org 1 entries 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:186: get(): resolver cache hit: httpbin.org 54.175.219.8 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:260: get_servers(): query for httpbin.org finished with 1 answers 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:118: fetch_answers(): resolver cache miss backend-listener 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:184: get(): resolver cache miss: backend-listener 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:194: lookup(): resolver query: backend-listener 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:43: init_resolvers(): initializing 1 nameservers 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:52: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: query: backend-listener. 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:61: resolver query: backend-listener. nameserver: 127.0.0.1:5353 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:204: lookup(): resolver query: backend-listener search: 3scale.svc.cluster.local query: backend-listener.3scale.svc.cluster.local 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] dns_client.lua:61: resolver query: backend-listener.3scale.svc.cluster.local nameserver: 127.0.0.1:5353 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:213: lookup(): resolver query: backend-listener finished with 1 answers 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] cache.lua:70: store(): resolver cache write backend-listener.3scale.svc.cluster.local with TLL 30 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] resolver.lua:260: get_servers(): query for backend-listener finished with 1 answers 2017/03/08 12:08:30 [debug] 20#20: *59 [lua] proxy.lua:303: set_backend_upstream(): [resolver] resolved backend upstream: 1 2017/03/08 12:08:30 [info] 20#20: *59 [lua] proxy.lua:73: access(): no credentials provided for service 4, client: 10.128.0.1, server: _, request: "GET /favicon.ico HTTP/1.1", host: "httpbin.org", referrer: "http://auth-oauth-stage.cloudapps.api-qe.eng.rdu2.redhat.com/authorize?state=68a76266-365c-46ef-9b33-c058b4387d40&response_type=code&client_id=auth-oauth123456789&redirect_uri=http%3A%2F%2Flocalhost%3A3001%2Fcallback&scope=" 2017/03/08 12:08:30 [info] 20#20: *59 [lua] proxy.lua:436: post_action(): [async] skipping after action, no cached key, client: 10.128.0.1, server: _, request: "GET /favicon.ico HTTP/1.1", host: "httpbin.org", referrer: "http://auth-oauth-stage.cloudapps.api-qe.eng.rdu2.redhat.com/authorize?state=68a76266-365c-46ef-9b33-c058b4387d40&response_type=code&client_id=auth-oauth123456789&redirect_uri=http%3A%2F%2Flocalhost%3A3001%2Fcallback&scope=" 2017/03/08 12:08:30 [info] 20#20: *59 [lua] log_by_lua(apicast.conf:48):3: [authrep] /favicon.ico 403 while logging request, client: 10.128.0.1, server: _, request: "GET /favicon.ico HTTP/1.1", host: "httpbin.org", referrer: "http://auth-oauth-stage.cloudapps.api-qe.eng.rdu2.redhat.com/authorize?state=68a76266-365c-46ef-9b33-c058b4387d40&response_type=code&client_id=auth-oauth123456789&redirect_uri=http%3A%2F%2Flocalhost%3A3001%2Fcallback&scope=" [08/Mar/2017:12:08:30 +0000] auth-oauth-stage.cloudapps.api-qe.eng.rdu2.redhat.com:8080 10.128.0.1:44566 "GET /favicon.ico HTTP/1.1" 403 44 (0.001) 0 2017/03/08 12:08:34 [debug] 20#20: *64 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:08:44 [debug] 20#20: *65 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:08:44 [debug] 20#20: *66 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:08:45 [info] 20#20: *59 client 10.128.0.1 closed keepalive connection (104: Connection reset by peer) 2017/03/08 12:08:54 [debug] 20#20: *67 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:09:04 [debug] 20#20: *68 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:09:14 [debug] 20#20: *70 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:09:14 [debug] 20#20: *69 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:09:24 [debug] 20#20: *71 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:09:34 [debug] 20#20: *72 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:09:44 [debug] 20#20: *74 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:09:44 [debug] 20#20: *73 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:09:54 [debug] 20#20: *75 [lua] management.lua:146: router(): management api mode: debug 2017/03/08 12:10:04 [debug] 20#20: *76 [lua] management.lua:146: router(): management api mode: debug