Details from the tests below: Tinyproxy: tinyproxy.internal:8888 APIcast: apicast.internal:8080 Backend API: 3scale Echo API (echo-api.3scale.net) APIcast parameters: # docker container run --rm --name apicast -e THREESCALE_PORTAL_ENDPOINT=https://@ekonecsn-admin.3scale.net -e APICAST_LOG_LEVEL=debug -e APICAST_CONFIGURATION_LOADER=lazy -e APICAST_CONFIGURATION_CACHE=0 -e http_proxy=http://tinyproxy.internal:8888 -e https_proxy=http://tinyproxy.internal:8888 -e no_proxy=ekonecsn-admin.3scale.net registry.access.redhat.com/3scale-amp24/apicast-gateway:latest # 1. HTTP GET using http://echo-api.3scale.net:80 (NOK) # 1.1. 'curl' using Tinyproxy directly: # export http_proxy="http://tinyproxy.internal:8888" ; export https_proxy=$http_proxy # curl -X GET http://echo-api.3scale.net:80 { "method": "GET", "path": "/", "args": "", "body": "", "headers": { "HTTP_VERSION": "HTTP/1.1", "HTTP_HOST": "echo-api.3scale.net", "HTTP_ACCEPT": "*/*", "HTTP_USER_AGENT": "curl/7.29.0", "HTTP_VIA": "1.1 tinyproxy (tinyproxy/1.8.3)", "HTTP_X_FORWARDED_FOR": "201.95.85.39, 10.0.103.54", "HTTP_X_FORWARDED_HOST": "echo-api.3scale.net", "HTTP_X_FORWARDED_PORT": "80", "HTTP_X_FORWARDED_PROTO": "http", "HTTP_FORWARDED": "for=10.0.103.54;host=echo-api.3scale.net;proto=http" }, "uuid": "70a7c75b-d0c1-442d-9ea8-a227362c5b8e" } CONNECT Apr 25 20:25:18 [10]: Connect (file descriptor 9): 64d56e9cb45b.internal [172.18.0.5] CONNECT Apr 25 20:25:18 [10]: Request (file descriptor 9): GET http://echo-api.3scale.net:80/ HTTP/1.1 INFO Apr 25 20:25:18 [10]: No upstream proxy for echo-api.3scale.net CONNECT Apr 25 20:25:18 [10]: Established connection to host "echo-api.3scale.net" using file descriptor 10. INFO Apr 25 20:25:19 [10]: Closed connection between local client (fd:9) and remote client (fd:10) 1.2. 'curl' using APIcast: # unset http_proxy https_proxy # curl -v -X GET "http://apicast.internal:8080/?user_key=" * About to connect() to apicast.internal port 8080 (#0) * Trying 172.18.0.2... * Connected to apicast.internal (172.18.0.2) port 8080 (#0) > GET /?user_key= HTTP/1.1 > User-Agent: curl/7.29.0 > Host: apicast.internal:8080 > Accept: */* > < HTTP/1.1 504 Gateway Time-out < Server: openresty/1.13.6.2 < Date: Thu, 25 Apr 2019 20:31:37 GMT < Content-Type: text/html < Content-Length: 189 < Connection: keep-alive < 504 Gateway Time-out

504 Gateway Time-out

openresty/1.13.6.2 * Connection #0 to host apicast.internal left intact 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] executor.lua:25: rewrite(): executor phase: rewrite 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Load Configuration, i: 1 2019/04/25 20:30:34 [info] 28#28: *14 [lua] configuration_loader.lua:213: rewrite(): lazy loading configuration for: apicast.internal, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] env.lua:86: fetch(): env: APICAST_CONFIGURATION = nil 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] env.lua:86: fetch(): env: THREESCALE_CONFIG_FILE = nil 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] env.lua:86: fetch(): env: APICAST_SERVICES_LIST = nil 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] env.lua:86: fetch(): env: APICAST_SERVICES = nil 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] resolver.lua:197: new(): resolver search domains: 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] cache.lua:122: fetch_answers(): resolver cache miss ekonecsn-admin.3scale.net 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] cache.lua:188: get(): resolver cache miss: ekonecsn-admin.3scale.net 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] resolver.lua:294: search_dns(): resolver query: ekonecsn-admin.3scale.net search: query: ekonecsn-admin.3scale.net. 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] cache.lua:122: fetch_answers(): resolver cache miss ekonecsn-admin.3scale.net. 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] cache.lua:188: get(): resolver cache miss: ekonecsn-admin.3scale.net. 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] dns_client.lua:43: init_resolvers(): initializing 4 nameservers 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] dns_client.lua:56: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] dns_client.lua:56: init_resolvers(): nameserver 8.8.8.8:53 initialized 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] dns_client.lua:56: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] dns_client.lua:56: init_resolvers(): nameserver 8.8.8.8:53 initialized 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] dns_client.lua:68: query(): resolver query: ekonecsn-admin.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] cache.lua:74: store(): resolver cache write ekonecsn-admin.3scale.net with TLL 574 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] cache.lua:74: store(): resolver cache write multitenant.3scale.net with TLL 274 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] cache.lua:74: store(): resolver cache write 3scale-20120424-299787090.us-east-1.elb.amazonaws.com with TLL 34 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] resolver.lua:328: lookup(): resolver query: 23.23.89.189 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] resolver.lua:333: lookup(): host is ip address: 23.23.89.189 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] resolver.lua:348: lookup(): resolver query: 23.23.89.189 finished with 1 answers 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] resolver.lua:388: get_servers(): query for 23.23.89.189 finished with 1 answers 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] http.lua:50: connect(): connected to ip:23.23.89.189 host: 23.23.89.189 port: 443 ok: 1 err: nil 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 0 2019/04/25 20:30:34 [debug] 28#28: *14 [lua] http.lua:633: send_request(): GET /admin/api/services.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] remote_v2.lua:234: services(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services.json 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] env.lua:86: fetch(): env: APICAST_SERVICE_2555417761253_CONFIGURATION_VERSION = nil 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] cache.lua:115: fetch_answers(): resolver cache read ekonecsn-admin.3scale.net 1 entries 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] cache.lua:115: fetch_answers(): resolver cache read multitenant.3scale.net 1 entries 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] cache.lua:115: fetch_answers(): resolver cache read 3scale-20120424-299787090.us-east-1.elb.amazonaws.com 2 entries 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] cache.lua:190: get(): resolver cache hit: ekonecsn-admin.3scale.net 50.19.254.27, 23.23.89.189 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:328: lookup(): resolver query: 50.19.254.27 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:333: lookup(): host is ip address: 50.19.254.27 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:348: lookup(): resolver query: 50.19.254.27 finished with 1 answers 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:388: get_servers(): query for 50.19.254.27 finished with 1 answers 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] http.lua:50: connect(): connected to ip:50.19.254.27 host: 50.19.254.27 port: 443 ok: 1 err: nil 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 0 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] http.lua:633: send_request(): GET /admin/api/services/2555417761253/proxy/configs/production/latest.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] remote_v2.lua:278: config(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services/2555417761253/proxy/configs/production/latest.json body: {"proxy_config":{"id":122306,"version":83,"environment":"production","content":{"id":2555417761253,"account_id":2445582593288,"name":"Echo API Hosted","oneline_description":null,"description":"","txt_api":null,"txt_support":null,"txt_features":null,"created_at":"2018-10-17T17:49:41-03:00","updated_at":"2019-04-25T17:21:51-03:00","logo_file_name":null,"logo_content_type":null,"logo_file_size":null,"state":"incomplete","intentions_required":false,"draft_name":"","infobar":null,"terms":null,"display_provider_keys":false,"credit_card_support_email":null,"buyers_manage_apps":true,"buyers_manage_keys":true,"custom_keys_enabled":true,"buyer_plan_change_permission":"request","buyer_can_select_plan":false,"notification_settings":{"web_provider":[0],"email_provider":[0],"web_buyer":[0,90,100],"email_buyer":[0,90,100]},"default_application_plan_id":2357355960517,"default_service_plan_id":2357355960516,"default_end_user_plan_id":null,"end_user_registration_required":true,"tenant_id":2445582593288,"system_name":"echo_api_hosted","backend_version":"1","mandatory_app_key":true,"buyer_key_regenerate_enabled":true,"support_email":"ekonecsn@redhat.com","referrer_filters_required":false,"deployment_option":"self_managed","kubernetes_service_link":null,"tech_support_email":null,"admin_support_email":null,"proxiable?":true,"backend_authentication_type":"service_token","backend_authentication_value":"b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c","proxy":{"id":107499,"tenant_id":2445582593288,"service_id":2555417761253,"endpoint":"http://apicast.internal:8080","deployed_at":null,"api_backend":"http://echo-api.3scale.net:80","auth_app_key":"app_key","auth_app_id":"app_id","auth_user_key":"user_key","credentials_location":"query","error_auth_failed":"Authentication failed","error_auth_missing":"Authentication parameters missing","created_at":"2018-10-17T17:49:41-03:00","updated_at":"2019-04-25T17:21:51-03:00","error_status_auth_failed":403,"error_headers_auth_failed":"text/plain; charset=us-ascii","error_status_auth_missing":403,"error_headers_auth_missing":"text/plain; charset=us-ascii","error_no_match":"No Mapping Rule matched","error_status_no_match":404,"error_headers_no_match":"text/plain; charset=us-ascii","secret_token":"Shared_secret_sent_from_proxy_to_API_backend_ec120e006f910d45","hostname_rewrite":"","oauth_login_url":null,"sandbox_endpoint":"http://apicast.internal:8080","api_test_path":"/","api_test_success":null,"apicast_configuration_driven":true,"oidc_issuer_endpoint":null,"lock_version":88,"authentication_method":"1","hostname_rewrite_for_sandbox":"echo-api.3scale.net","endpoint_port":8080,"valid?":true,"service_backend_version":"1","hosts":["apicast.internal"],"backend":{"endpoint":"https://su1.3scale.net","host":"su1.3scale.net"},"policy_chain":[{"name":"apicast","version":"builtin","configuration":{}}],"proxy_rules":[{"id":434579,"proxy_id":107499,"http_method":"GET","pattern":"/","metric_id":2555418124738,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-04-16T13:42:31-03:00","updated_at":"2019-04-25T16:53:28-03:00","redirect_url":null,"position":1,"last":false,"parameters":[],"querystring_parameters":{}},{"id":442138,"proxy_id":107499,"http_method":"POST","pattern":"/","metric_id":2555418124738,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-04-25T14:42:49-03:00","updated_at":"2019-04-25T16:53:28-03:00","redirect_url":null,"position":2,"last":false,"parameters":[],"querystring_parameters":{}}]}}}} 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] env.lua:86: fetch(): env: APICAST_SERVICE_2555417768857_CONFIGURATION_VERSION = nil 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] cache.lua:115: fetch_answers(): resolver cache read ekonecsn-admin.3scale.net 1 entries 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] cache.lua:115: fetch_answers(): resolver cache read multitenant.3scale.net 1 entries 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] cache.lua:115: fetch_answers(): resolver cache read 3scale-20120424-299787090.us-east-1.elb.amazonaws.com 2 entries 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] cache.lua:190: get(): resolver cache hit: ekonecsn-admin.3scale.net 50.19.254.27, 23.23.89.189 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:328: lookup(): resolver query: 23.23.89.189 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:333: lookup(): host is ip address: 23.23.89.189 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:348: lookup(): resolver query: 23.23.89.189 finished with 1 answers 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] resolver.lua:388: get_servers(): query for 23.23.89.189 finished with 1 answers 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] http.lua:50: connect(): connected to ip:23.23.89.189 host: 23.23.89.189 port: 443 ok: 1 err: nil 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 1 2019/04/25 20:30:36 [debug] 28#28: *14 [lua] http.lua:633: send_request(): GET /admin/api/services/2555417768857/proxy/configs/production/latest.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] remote_v2.lua:278: config(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services/2555417768857/proxy/configs/production/latest.json body: {"proxy_config":{"id":115033,"version":32,"environment":"production","content":{"id":2555417768857,"account_id":2445582593288,"name":"Test API","oneline_description":null,"description":"Test API Using Service Create","txt_api":null,"txt_support":null,"txt_features":null,"created_at":"2019-02-05T19:19:36-02:00","updated_at":"2019-04-11T16:23:03-03:00","logo_file_name":null,"logo_content_type":null,"logo_file_size":null,"state":"incomplete","intentions_required":false,"draft_name":"","infobar":null,"terms":null,"display_provider_keys":false,"credit_card_support_email":null,"buyers_manage_apps":true,"buyers_manage_keys":true,"custom_keys_enabled":true,"buyer_plan_change_permission":"request","buyer_can_select_plan":false,"notification_settings":{"web_provider":[0],"email_provider":[0],"web_buyer":[0],"email_buyer":[0]},"default_application_plan_id":null,"default_service_plan_id":null,"default_end_user_plan_id":null,"end_user_registration_required":true,"tenant_id":2445582593288,"system_name":"test_api","backend_version":"1","mandatory_app_key":true,"buyer_key_regenerate_enabled":true,"support_email":"ekonecsn@redhat.com","referrer_filters_required":true,"deployment_option":"self_managed","kubernetes_service_link":null,"tech_support_email":null,"admin_support_email":null,"proxiable?":true,"backend_authentication_type":"service_token","backend_authentication_value":"fc80b523e672cc8ff5df22b0f3914f87f60a82c168d1ce34c52df3d25cf4e9be","proxy":{"id":115061,"tenant_id":2445582593288,"service_id":2555417768857,"endpoint":"http://localhost:8080","deployed_at":null,"api_backend":"https://echo-api.3scale.net:443","auth_app_key":"app_key","auth_app_id":"app_id","auth_user_key":"user_key","credentials_location":"query","error_auth_failed":"Authentication failed","error_auth_missing":"Authentication parameters missing","created_at":"2019-02-05T19:19:37-02:00","updated_at":"2019-04-11T16:23:03-03:00","error_status_auth_failed":403,"error_headers_auth_failed":"text/plain; charset=us-ascii","error_status_auth_missing":403,"error_headers_auth_missing":"text/plain; charset=us-ascii","error_no_match":"No Mapping Rule matched","error_status_no_match":404,"error_headers_no_match":"text/plain; charset=us-ascii","secret_token":"Shared_secret_sent_from_proxy_to_API_backend_a5fb94644b900716","hostname_rewrite":"","oauth_login_url":null,"sandbox_endpoint":"http://localhost:8080","api_test_path":"/","api_test_success":null,"apicast_configuration_driven":true,"oidc_issuer_endpoint":null,"lock_version":35,"authentication_method":"1","hostname_rewrite_for_sandbox":"echo-api.3scale.net","endpoint_port":8080,"valid?":true,"service_backend_version":"1","hosts":["localhost"],"backend":{"endpoint":"https://su1.3scale.net","host":"su1.3scale.net"},"policy_chain":[{"name":"upstream","version":"builtin","configuration":{"rules":[{"url":"http://apicast-url2.internal","regex":"^/apicast.internal/.*"}]}},{"name":"apicast","version":"builtin","configuration":{}}],"proxy_rules":[{"id":339975,"proxy_id":115061,"http_method":"GET","pattern":"/testapi","metric_id":2555418152044,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-02-18T17:29:40-03:00","updated_at":"2019-04-11T16:23:03-03:00","redirect_url":null,"position":1,"last":false,"parameters":[],"querystring_parameters":{}}]}}}} 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_loader.lua:150: loading policy: upstream version: builtin 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_loader.lua:150: loading policy: upstream version: builtin 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:30:37 [info] 28#28: *14 [lua] configuration_store.lua:124: store(): added service 2555417761253 configuration with hosts: apicast.internal ttl: 0, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:30:37 [info] 28#28: *14 [lua] configuration_store.lua:124: store(): added service 2555417768857 configuration with hosts: localhost ttl: 0, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Find Service Policy, i: 2 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Local Policy Chain, i: 3 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] env.lua:86: fetch(): env: APICAST_BACKEND_CACHE_HANDLER = nil 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache_handler.lua:23: new(): backend cache handler: strict 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: APIcast, i: 1 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] service.lua:227: get_usage(): [mapping] service 2555417761253 has 2 rules 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Metrics, i: 4 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] executor.lua:25: access(): executor phase: access 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Load Configuration, i: 1 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Find Service Policy, i: 2 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Local Policy Chain, i: 3 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: policy chain execute phase: access, policy: APIcast, i: 1 2019/04/25 20:30:37 [info] 28#28: *14 [lua] proxy.lua:81: output_debug_headers(): usage: usage%5Bhits%5D=1 credentials: user_key=, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] proxy.lua:145: apicast cache hit key: 2555417761253::usage%5Bhits%5D=1 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Metrics, i: 4 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] executor.lua:25: content(): executor phase: content 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Load Configuration, i: 1 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Find Service Policy, i: 2 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Local Policy Chain, i: 3 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: policy chain execute phase: content, policy: APIcast, i: 1 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] upstream.lua:181: call(): using proxy: http://tinyproxy.internal:8888 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] resolver.lua:328: lookup(): resolver query: tinyproxy.internal 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:122: fetch_answers(): resolver cache miss tinyproxy.internal 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:188: get(): resolver cache miss: tinyproxy.internal 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] resolver.lua:294: search_dns(): resolver query: tinyproxy.internal search: query: tinyproxy.internal. 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:122: fetch_answers(): resolver cache miss tinyproxy.internal. 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:188: get(): resolver cache miss: tinyproxy.internal. 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] dns_client.lua:68: query(): resolver query: tinyproxy.internal. nameserver: 127.0.0.1:5353 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:74: store(): resolver cache write tinyproxy.internal with TLL 586 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] resolver.lua:348: lookup(): resolver query: tinyproxy.internal finished with 1 answers 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] resolver.lua:388: resolve_servers(): query for tinyproxy.internal finished with 1 answers 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] resolver.lua:328: lookup(): resolver query: echo-api.3scale.net 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:122: fetch_answers(): resolver cache miss echo-api.3scale.net 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:188: get(): resolver cache miss: echo-api.3scale.net 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] resolver.lua:294: search_dns(): resolver query: echo-api.3scale.net search: query: echo-api.3scale.net. 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:122: fetch_answers(): resolver cache miss echo-api.3scale.net. 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:188: get(): resolver cache miss: echo-api.3scale.net. 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] dns_client.lua:68: query(): resolver query: echo-api.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:74: store(): resolver cache write echo-api.3scale.net with TLL 286 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] cache.lua:74: store(): resolver cache write tf-lb-0081dad066b2afef5328e0256a-2081992367.us-east-1.elb.amazonaws.com with TLL 46 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] resolver.lua:348: lookup(): resolver query: echo-api.3scale.net finished with 3 answers 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] resolver.lua:388: resolve_servers(): query for echo-api.3scale.net finished with 3 answers 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] executor.lua:25: balancer(): executor phase: balancer 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: balancer(): policy chain execute phase: balancer, policy: Load Configuration, i: 1 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: balancer(): policy chain execute phase: balancer, policy: Find Service Policy, i: 2 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: balancer(): policy chain execute phase: balancer, policy: Local Policy Chain, i: 3 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: policy chain execute phase: balancer, policy: APIcast, i: 1 2019/04/25 20:30:37 [info] 28#28: *14 [lua] balancer.lua:108: set_current_peer(): balancer set peer 172.18.0.3:8888 ok: true err: nil while connecting to upstream, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "echo-api.3scale.net" 2019/04/25 20:30:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: balancer(): policy chain execute phase: balancer, policy: Metrics, i: 4 2019/04/25 20:31:37 [error] 28#28: *14 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", upstream: "http://172.18.0.3:8888http://18.213.6.196:80?user_key=", host: "echo-api.3scale.net" 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] executor.lua:25: header_filter(): executor phase: header_filter 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Load Configuration, i: 1 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Find Service Policy, i: 2 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Local Policy Chain, i: 3 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: policy chain execute phase: header_filter, policy: APIcast, i: 1 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Metrics, i: 4 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] executor.lua:25: body_filter(): executor phase: body_filter 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Load Configuration, i: 1 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Find Service Policy, i: 2 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Local Policy Chain, i: 3 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: policy chain execute phase: body_filter, policy: APIcast, i: 1 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Metrics, i: 4 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] executor.lua:25: post_action(): executor phase: post_action 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Load Configuration, i: 1 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Find Service Policy, i: 2 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Local Policy Chain, i: 3 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] policy_chain.lua:162: policy chain execute phase: post_action, policy: APIcast, i: 1 2019/04/25 20:31:37 [info] 28#28: *14 [lua] proxy.lua:335: [async] reporting to backend asynchronously, cached_key: 2555417761253::usage%5Bhits%5D=1 while sending to client, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", upstream: "http://172.18.0.3:8888http://18.213.6.196:80?user_key=", host: "echo-api.3scale.net" 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] resolver.lua:328: lookup(): resolver query: su1.3scale.net 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] cache.lua:122: fetch_answers(): resolver cache miss su1.3scale.net 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] cache.lua:188: get(): resolver cache miss: su1.3scale.net 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] resolver.lua:294: search_dns(): resolver query: su1.3scale.net search: query: su1.3scale.net. 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] cache.lua:122: fetch_answers(): resolver cache miss su1.3scale.net. 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] cache.lua:188: get(): resolver cache miss: su1.3scale.net. 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] dns_client.lua:68: query(): resolver query: su1.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] cache.lua:74: store(): resolver cache write su1.3scale.net with TLL 26 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] resolver.lua:348: lookup(): resolver query: su1.3scale.net finished with 1 answers 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] resolver.lua:388: get_servers(): query for su1.3scale.net finished with 1 answers 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] resolver.lua:328: lookup(): resolver query: tinyproxy.internal 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] cache.lua:115: fetch_answers(): resolver cache read tinyproxy.internal 1 entries 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] cache.lua:190: get(): resolver cache hit: tinyproxy.internal 172.18.0.3 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] resolver.lua:348: lookup(): resolver query: tinyproxy.internal finished with 1 answers 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] resolver.lua:388: get_servers(): query for tinyproxy.internal finished with 1 answers 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] http.lua:50: connect(): connected to ip:172.18.0.3 host: tinyproxy.internal port: 8888 ok: 1 err: nil 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] proxy.lua:82: new(): connection to tinyproxy.internal:8888 established, pool: tinyproxy.internal:8888:su1.3scale.net:443 reused times: 0 2019/04/25 20:31:37 [debug] 28#28: *14 [lua] http.lua:633: send_request(): CONNECT 54.83.62.94:443 HTTP/1.1 User-Agent: lua-resty-http/0.12 (Lua) ngx_lua/10013 Host: su1.3scale.net 2019/04/25 20:31:38 [debug] 28#28: *14 [lua] http.lua:633: send_request(): GET /transactions/authrep.xml?service_token=b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c&service_id=2555417761253&usage%5Bhits%5D=1&user_key= HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 3scale-Options: rejection_reason_header=1&limit_headers=1&no_body=1 Connection: Keep-Alive Host: su1.3scale.net 2019/04/25 20:31:38 [info] 28#28: *14 [lua] backend_client.lua:139: call_backend_transaction(): backend client uri: https://su1.3scale.net/transactions/authrep.xml?service_token=b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c&service_id=2555417761253&usage%5Bhits%5D=1&user_key= ok: true status: 200 body: error: nil while sending to client, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", upstream: "http://172.18.0.3:8888http://18.213.6.196:80?user_key=", host: "echo-api.3scale.net" 2019/04/25 20:31:38 [debug] 28#28: *14 [lua] proxy.lua:368: handle_backend_response(): [backend] response status: 200 body: 2019/04/25 20:31:38 [debug] 28#28: *14 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Metrics, i: 4 2019/04/25 20:31:38 [debug] 28#28: *14 [lua] executor.lua:25: log(): executor phase: log 2019/04/25 20:31:38 [debug] 28#28: *14 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Load Configuration, i: 1 2019/04/25 20:31:38 [debug] 28#28: *14 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Find Service Policy, i: 2 2019/04/25 20:31:38 [debug] 28#28: *14 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Local Policy Chain, i: 3 2019/04/25 20:31:38 [debug] 28#28: *14 [lua] policy_chain.lua:162: policy chain execute phase: log, policy: APIcast, i: 1 2019/04/25 20:31:38 [debug] 28#28: *14 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Metrics, i: 4 [25/Apr/2019:20:31:38 +0000] apicast.internal:8080 172.18.0.5:38092 "GET /?user_key= HTTP/1.1" 504 189 (64.100) 0.89999999999999 2019/04/25 20:31:38 [info] 28#28: *14 client 172.18.0.5 closed keepalive connection CONNECT Apr 25 20:30:37 [13]: Connect (file descriptor 9): apicast.internal [172.18.0.2] CONNECT Apr 25 20:30:37 [13]: Request (file descriptor 9): GET http://18.213.6.196:80?user_key= HTTP/1.1 INFO Apr 25 20:30:37 [13]: No upstream proxy for 18.213.6.196 CONNECT Apr 25 20:30:37 [13]: Established connection to host "18.213.6.196" using file descriptor 10. CONNECT Apr 25 20:31:37 [14]: Connect (file descriptor 9): apicast.internal [172.18.0.2] CONNECT Apr 25 20:31:37 [14]: Request (file descriptor 9): CONNECT 54.83.62.94:443 HTTP/1.1 INFO Apr 25 20:31:37 [14]: No upstream proxy for 54.83.62.94 CONNECT Apr 25 20:31:37 [14]: Established connection to host "54.83.62.94" using file descriptor 10. INFO Apr 25 20:31:37 [14]: Not sending client headers to remote machine INFO Apr 25 20:31:38 [14]: Closed connection between local client (fd:9) and remote client (fd:10) # 2. HTTP GET using https://echo-api.3scale.net:443 (OK) # 2.1. 'curl' using Tinyproxy directly: # export http_proxy="http://tinyproxy.internal:8888" ; export https_proxy=$http_proxy # curl -X GET https://echo-api.3scale.net:443 { "method": "GET", "path": "/", "args": "", "body": "", "headers": { "HTTP_VERSION": "HTTP/1.1", "HTTP_HOST": "echo-api.3scale.net", "HTTP_ACCEPT": "*/*", "HTTP_USER_AGENT": "curl/7.29.0", "HTTP_X_FORWARDED_FOR": "201.95.85.39, 10.0.103.54", "HTTP_X_FORWARDED_HOST": "echo-api.3scale.net", "HTTP_X_FORWARDED_PORT": "443", "HTTP_X_FORWARDED_PROTO": "https", "HTTP_FORWARDED": "for=10.0.103.54;host=echo-api.3scale.net;proto=https" }, "uuid": "b63f7c70-0a37-4895-940a-69d0dfaedc9b" } CONNECT Apr 25 20:35:41 [17]: Connect (file descriptor 9): 64d56e9cb45b.internal [172.18.0.5] CONNECT Apr 25 20:35:41 [17]: Request (file descriptor 9): CONNECT echo-api.3scale.net:443 HTTP/1.1 INFO Apr 25 20:35:41 [17]: No upstream proxy for echo-api.3scale.net CONNECT Apr 25 20:35:42 [17]: Established connection to host "echo-api.3scale.net" using file descriptor 10. INFO Apr 25 20:35:42 [17]: Not sending client headers to remote machine INFO Apr 25 20:35:42 [17]: Closed connection between local client (fd:9) and remote client (fd:10) 2.2. 'curl' using APIcast: # unset http_proxy https_proxy # curl -v -X GET "http://apicast.internal:8080/?user_key=" * About to connect() to apicast.internal port 8080 (#0) * Trying 172.18.0.2... * Connected to apicast.internal (172.18.0.2) port 8080 (#0) > GET /?user_key= HTTP/1.1 > User-Agent: curl/7.29.0 > Host: apicast.internal:8080 > Accept: */* > < HTTP/1.1 200 OK < Server: openresty/1.13.6.2 < Date: Thu, 25 Apr 2019 20:37:41 GMT < Content-Type: application/json < Transfer-Encoding: chunked < Connection: keep-alive < Vary: Origin < Set-Cookie: d8c1dd0e39ac4456ed39ce5889b9a5a5=abe93c04c357bfdab7978dd760ca80c0; path=/; HttpOnly < Cache-control: private < X-Content-Type-Options: nosniff < { "method": "GET", "path": "/", "args": "user_key=", "body": "", "headers": { "HTTP_VERSION": "HTTP/1.1", "HTTP_HOST": "echo-api.3scale.net", "HTTP_ACCEPT": "*/*", "HTTP_USER_AGENT": "curl/7.29.0", "HTTP_X_FORWARDED_FOR": "201.95.85.39, 10.0.103.54", "HTTP_X_FORWARDED_HOST": "echo-api.3scale.net", "HTTP_X_FORWARDED_PORT": "443", "HTTP_X_FORWARDED_PROTO": "https", "HTTP_FORWARDED": "for=10.0.103.54;host=echo-api.3scale.net;proto=https" }, "uuid": "a830827d-1a77-493b-97d0-040aa213604d" * Connection #0 to host apicast.internal left intact } 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] executor.lua:25: rewrite(): executor phase: rewrite 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Load Configuration, i: 1 2019/04/25 20:37:38 [info] 28#28: *32 [lua] configuration_loader.lua:213: rewrite(): lazy loading configuration for: apicast.internal, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] resolver.lua:197: new(): resolver search domains: 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] cache.lua:115: fetch_answers(): resolver cache read ekonecsn-admin.3scale.net 1 entries 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] cache.lua:122: fetch_answers(): resolver cache miss multitenant.3scale.net 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] cache.lua:190: get(): resolver cache hit: ekonecsn-admin.3scale.net 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] resolver.lua:294: search_dns(): resolver query: ekonecsn-admin.3scale.net search: query: ekonecsn-admin.3scale.net. 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] cache.lua:122: fetch_answers(): resolver cache miss ekonecsn-admin.3scale.net. 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] cache.lua:188: get(): resolver cache miss: ekonecsn-admin.3scale.net. 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] dns_client.lua:43: init_resolvers(): initializing 4 nameservers 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] dns_client.lua:56: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] dns_client.lua:56: init_resolvers(): nameserver 8.8.8.8:53 initialized 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] dns_client.lua:56: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] dns_client.lua:56: init_resolvers(): nameserver 8.8.8.8:53 initialized 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] dns_client.lua:68: query(): resolver query: ekonecsn-admin.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] cache.lua:74: store(): resolver cache write ekonecsn-admin.3scale.net with TLL 587 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] cache.lua:74: store(): resolver cache write multitenant.3scale.net with TLL 287 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] cache.lua:74: store(): resolver cache write 3scale-20120424-299787090.us-east-1.elb.amazonaws.com with TLL 47 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] resolver.lua:328: lookup(): resolver query: 50.19.254.27 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] resolver.lua:333: lookup(): host is ip address: 50.19.254.27 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] resolver.lua:348: lookup(): resolver query: 50.19.254.27 finished with 1 answers 2019/04/25 20:37:38 [debug] 28#28: *32 [lua] resolver.lua:388: get_servers(): query for 50.19.254.27 finished with 1 answers 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] http.lua:50: connect(): connected to ip:50.19.254.27 host: 50.19.254.27 port: 443 ok: 1 err: nil 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 0 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] http.lua:633: send_request(): GET /admin/api/services.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] remote_v2.lua:234: services(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services.json 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] cache.lua:115: fetch_answers(): resolver cache read ekonecsn-admin.3scale.net 1 entries 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] cache.lua:115: fetch_answers(): resolver cache read multitenant.3scale.net 1 entries 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] cache.lua:115: fetch_answers(): resolver cache read 3scale-20120424-299787090.us-east-1.elb.amazonaws.com 2 entries 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] cache.lua:190: get(): resolver cache hit: ekonecsn-admin.3scale.net 50.19.254.27, 23.23.89.189 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] resolver.lua:328: lookup(): resolver query: 23.23.89.189 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] resolver.lua:333: lookup(): host is ip address: 23.23.89.189 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] resolver.lua:348: lookup(): resolver query: 23.23.89.189 finished with 1 answers 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] resolver.lua:388: get_servers(): query for 23.23.89.189 finished with 1 answers 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] http.lua:50: connect(): connected to ip:23.23.89.189 host: 23.23.89.189 port: 443 ok: 1 err: nil 2019/04/25 20:37:39 [debug] 28#28: *32 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 0 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] http.lua:633: send_request(): GET /admin/api/services/2555417761253/proxy/configs/production/latest.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] remote_v2.lua:278: config(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services/2555417761253/proxy/configs/production/latest.json body: {"proxy_config":{"id":122308,"version":84,"environment":"production","content":{"id":2555417761253,"account_id":2445582593288,"name":"Echo API Hosted","oneline_description":null,"description":"","txt_api":null,"txt_support":null,"txt_features":null,"created_at":"2018-10-17T17:49:41-03:00","updated_at":"2019-04-25T17:34:00-03:00","logo_file_name":null,"logo_content_type":null,"logo_file_size":null,"state":"incomplete","intentions_required":false,"draft_name":"","infobar":null,"terms":null,"display_provider_keys":false,"credit_card_support_email":null,"buyers_manage_apps":true,"buyers_manage_keys":true,"custom_keys_enabled":true,"buyer_plan_change_permission":"request","buyer_can_select_plan":false,"notification_settings":{"web_provider":[0],"email_provider":[0],"web_buyer":[0,90,100],"email_buyer":[0,90,100]},"default_application_plan_id":2357355960517,"default_service_plan_id":2357355960516,"default_end_user_plan_id":null,"end_user_registration_required":true,"tenant_id":2445582593288,"system_name":"echo_api_hosted","backend_version":"1","mandatory_app_key":true,"buyer_key_regenerate_enabled":true,"support_email":"ekonecsn@redhat.com","referrer_filters_required":false,"deployment_option":"self_managed","kubernetes_service_link":null,"tech_support_email":null,"admin_support_email":null,"proxiable?":true,"backend_authentication_type":"service_token","backend_authentication_value":"b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c","proxy":{"id":107499,"tenant_id":2445582593288,"service_id":2555417761253,"endpoint":"http://apicast.internal:8080","deployed_at":null,"api_backend":"https://echo-api.3scale.net:443","auth_app_key":"app_key","auth_app_id":"app_id","auth_user_key":"user_key","credentials_location":"query","error_auth_failed":"Authentication failed","error_auth_missing":"Authentication parameters missing","created_at":"2018-10-17T17:49:41-03:00","updated_at":"2019-04-25T17:34:00-03:00","error_status_auth_failed":403,"error_headers_auth_failed":"text/plain; charset=us-ascii","error_status_auth_missing":403,"error_headers_auth_missing":"text/plain; charset=us-ascii","error_no_match":"No Mapping Rule matched","error_status_no_match":404,"error_headers_no_match":"text/plain; charset=us-ascii","secret_token":"Shared_secret_sent_from_proxy_to_API_backend_ec120e006f910d45","hostname_rewrite":"","oauth_login_url":null,"sandbox_endpoint":"http://apicast.internal:8080","api_test_path":"/","api_test_success":null,"apicast_configuration_driven":true,"oidc_issuer_endpoint":null,"lock_version":89,"authentication_method":"1","hostname_rewrite_for_sandbox":"echo-api.3scale.net","endpoint_port":8080,"valid?":true,"service_backend_version":"1","hosts":["apicast.internal"],"backend":{"endpoint":"https://su1.3scale.net","host":"su1.3scale.net"},"policy_chain":[{"name":"apicast","version":"builtin","configuration":{}}],"proxy_rules":[{"id":434579,"proxy_id":107499,"http_method":"GET","pattern":"/","metric_id":2555418124738,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-04-16T13:42:31-03:00","updated_at":"2019-04-25T16:53:28-03:00","redirect_url":null,"position":1,"last":false,"parameters":[],"querystring_parameters":{}},{"id":442138,"proxy_id":107499,"http_method":"POST","pattern":"/","metric_id":2555418124738,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-04-25T14:42:49-03:00","updated_at":"2019-04-25T16:53:28-03:00","redirect_url":null,"position":2,"last":false,"parameters":[],"querystring_parameters":{}}]}}}} 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:115: fetch_answers(): resolver cache read ekonecsn-admin.3scale.net 1 entries 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:115: fetch_answers(): resolver cache read multitenant.3scale.net 1 entries 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:115: fetch_answers(): resolver cache read 3scale-20120424-299787090.us-east-1.elb.amazonaws.com 2 entries 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:190: get(): resolver cache hit: ekonecsn-admin.3scale.net 50.19.254.27, 23.23.89.189 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:328: lookup(): resolver query: 50.19.254.27 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:333: lookup(): host is ip address: 50.19.254.27 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:348: lookup(): resolver query: 50.19.254.27 finished with 1 answers 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:388: get_servers(): query for 50.19.254.27 finished with 1 answers 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] http.lua:50: connect(): connected to ip:50.19.254.27 host: 50.19.254.27 port: 443 ok: 1 err: nil 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 1 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] http.lua:633: send_request(): GET /admin/api/services/2555417768857/proxy/configs/production/latest.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] remote_v2.lua:278: config(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services/2555417768857/proxy/configs/production/latest.json body: {"proxy_config":{"id":115033,"version":32,"environment":"production","content":{"id":2555417768857,"account_id":2445582593288,"name":"Test API","oneline_description":null,"description":"Test API Using Service Create","txt_api":null,"txt_support":null,"txt_features":null,"created_at":"2019-02-05T19:19:36-02:00","updated_at":"2019-04-11T16:23:03-03:00","logo_file_name":null,"logo_content_type":null,"logo_file_size":null,"state":"incomplete","intentions_required":false,"draft_name":"","infobar":null,"terms":null,"display_provider_keys":false,"credit_card_support_email":null,"buyers_manage_apps":true,"buyers_manage_keys":true,"custom_keys_enabled":true,"buyer_plan_change_permission":"request","buyer_can_select_plan":false,"notification_settings":{"web_provider":[0],"email_provider":[0],"web_buyer":[0],"email_buyer":[0]},"default_application_plan_id":null,"default_service_plan_id":null,"default_end_user_plan_id":null,"end_user_registration_required":true,"tenant_id":2445582593288,"system_name":"test_api","backend_version":"1","mandatory_app_key":true,"buyer_key_regenerate_enabled":true,"support_email":"ekonecsn@redhat.com","referrer_filters_required":true,"deployment_option":"self_managed","kubernetes_service_link":null,"tech_support_email":null,"admin_support_email":null,"proxiable?":true,"backend_authentication_type":"service_token","backend_authentication_value":"fc80b523e672cc8ff5df22b0f3914f87f60a82c168d1ce34c52df3d25cf4e9be","proxy":{"id":115061,"tenant_id":2445582593288,"service_id":2555417768857,"endpoint":"http://localhost:8080","deployed_at":null,"api_backend":"https://echo-api.3scale.net:443","auth_app_key":"app_key","auth_app_id":"app_id","auth_user_key":"user_key","credentials_location":"query","error_auth_failed":"Authentication failed","error_auth_missing":"Authentication parameters missing","created_at":"2019-02-05T19:19:37-02:00","updated_at":"2019-04-11T16:23:03-03:00","error_status_auth_failed":403,"error_headers_auth_failed":"text/plain; charset=us-ascii","error_status_auth_missing":403,"error_headers_auth_missing":"text/plain; charset=us-ascii","error_no_match":"No Mapping Rule matched","error_status_no_match":404,"error_headers_no_match":"text/plain; charset=us-ascii","secret_token":"Shared_secret_sent_from_proxy_to_API_backend_a5fb94644b900716","hostname_rewrite":"","oauth_login_url":null,"sandbox_endpoint":"http://localhost:8080","api_test_path":"/","api_test_success":null,"apicast_configuration_driven":true,"oidc_issuer_endpoint":null,"lock_version":35,"authentication_method":"1","hostname_rewrite_for_sandbox":"echo-api.3scale.net","endpoint_port":8080,"valid?":true,"service_backend_version":"1","hosts":["localhost"],"backend":{"endpoint":"https://su1.3scale.net","host":"su1.3scale.net"},"policy_chain":[{"name":"upstream","version":"builtin","configuration":{"rules":[{"url":"http://apicast-url2.internal","regex":"^/apicast.internal/.*"}]}},{"name":"apicast","version":"builtin","configuration":{}}],"proxy_rules":[{"id":339975,"proxy_id":115061,"http_method":"GET","pattern":"/testapi","metric_id":2555418152044,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-02-18T17:29:40-03:00","updated_at":"2019-04-11T16:23:03-03:00","redirect_url":null,"position":1,"last":false,"parameters":[],"querystring_parameters":{}}]}}}} 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_loader.lua:150: loading policy: upstream version: builtin 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_loader.lua:150: loading policy: upstream version: builtin 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:37:40 [info] 28#28: *32 [lua] configuration_store.lua:124: store(): added service 2555417761253 configuration with hosts: apicast.internal ttl: 0, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:37:40 [info] 28#28: *32 [lua] configuration_store.lua:124: store(): added service 2555417768857 configuration with hosts: localhost ttl: 0, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Find Service Policy, i: 2 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Local Policy Chain, i: 3 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache_handler.lua:23: new(): backend cache handler: strict 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: APIcast, i: 1 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] service.lua:227: get_usage(): [mapping] service 2555417761253 has 2 rules 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Metrics, i: 4 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] executor.lua:25: access(): executor phase: access 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Load Configuration, i: 1 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Find Service Policy, i: 2 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Local Policy Chain, i: 3 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: policy chain execute phase: access, policy: APIcast, i: 1 2019/04/25 20:37:40 [info] 28#28: *32 [lua] proxy.lua:81: output_debug_headers(): usage: usage%5Bhits%5D=1 credentials: user_key=, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] proxy.lua:145: apicast cache hit key: 2555417761253::usage%5Bhits%5D=1 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Metrics, i: 4 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] executor.lua:25: content(): executor phase: content 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Load Configuration, i: 1 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Find Service Policy, i: 2 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Local Policy Chain, i: 3 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] policy_chain.lua:162: policy chain execute phase: content, policy: APIcast, i: 1 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] upstream.lua:181: call(): using proxy: http://tinyproxy.internal:8888 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:328: lookup(): resolver query: echo-api.3scale.net 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:122: fetch_answers(): resolver cache miss echo-api.3scale.net 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:188: get(): resolver cache miss: echo-api.3scale.net 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:294: search_dns(): resolver query: echo-api.3scale.net search: query: echo-api.3scale.net. 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:122: fetch_answers(): resolver cache miss echo-api.3scale.net. 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:188: get(): resolver cache miss: echo-api.3scale.net. 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] dns_client.lua:68: query(): resolver query: echo-api.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:74: store(): resolver cache write echo-api.3scale.net with TLL 288 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:74: store(): resolver cache write tf-lb-0081dad066b2afef5328e0256a-2081992367.us-east-1.elb.amazonaws.com with TLL 48 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:348: lookup(): resolver query: echo-api.3scale.net finished with 3 answers 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:388: get_servers(): query for echo-api.3scale.net finished with 3 answers 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:328: lookup(): resolver query: tinyproxy.internal 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:115: fetch_answers(): resolver cache read tinyproxy.internal 1 entries 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] cache.lua:190: get(): resolver cache hit: tinyproxy.internal 172.18.0.3 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:348: lookup(): resolver query: tinyproxy.internal finished with 1 answers 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] resolver.lua:388: get_servers(): query for tinyproxy.internal finished with 1 answers 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] http.lua:50: connect(): connected to ip:172.18.0.3 host: tinyproxy.internal port: 8888 ok: 1 err: nil 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] proxy.lua:82: new(): connection to tinyproxy.internal:8888 established, pool: tinyproxy.internal:8888:echo-api.3scale.net:443 reused times: 0 2019/04/25 20:37:40 [debug] 28#28: *32 [lua] http.lua:633: send_request(): CONNECT 18.213.6.196:443 HTTP/1.1 User-Agent: lua-resty-http/0.12 (Lua) ngx_lua/10013 Host: echo-api.3scale.net:443 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] http.lua:633: send_request(): GET /?user_key= HTTP/1.1 User-Agent: curl/7.29.0 Accept: */* Host: echo-api.3scale.net 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] executor.lua:25: header_filter(): executor phase: header_filter 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Load Configuration, i: 1 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Find Service Policy, i: 2 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Local Policy Chain, i: 3 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: policy chain execute phase: header_filter, policy: APIcast, i: 1 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Metrics, i: 4 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] executor.lua:25: body_filter(): executor phase: body_filter 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Load Configuration, i: 1 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Find Service Policy, i: 2 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Local Policy Chain, i: 3 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: policy chain execute phase: body_filter, policy: APIcast, i: 1 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Metrics, i: 4 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] executor.lua:25: body_filter(): executor phase: body_filter 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Load Configuration, i: 1 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Find Service Policy, i: 2 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Local Policy Chain, i: 3 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: policy chain execute phase: body_filter, policy: APIcast, i: 1 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Metrics, i: 4 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] executor.lua:25: post_action(): executor phase: post_action 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Load Configuration, i: 1 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Find Service Policy, i: 2 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Local Policy Chain, i: 3 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] policy_chain.lua:162: policy chain execute phase: post_action, policy: APIcast, i: 1 2019/04/25 20:37:41 [info] 28#28: *32 [lua] proxy.lua:335: [async] reporting to backend asynchronously, cached_key: 2555417761253::usage%5Bhits%5D=1, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "echo-api.3scale.net" 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] resolver.lua:328: lookup(): resolver query: su1.3scale.net 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] cache.lua:122: fetch_answers(): resolver cache miss su1.3scale.net 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] cache.lua:188: get(): resolver cache miss: su1.3scale.net 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] resolver.lua:294: search_dns(): resolver query: su1.3scale.net search: query: su1.3scale.net. 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] cache.lua:122: fetch_answers(): resolver cache miss su1.3scale.net. 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] cache.lua:188: get(): resolver cache miss: su1.3scale.net. 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] dns_client.lua:68: query(): resolver query: su1.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] cache.lua:74: store(): resolver cache write su1.3scale.net with TLL 16 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] resolver.lua:348: lookup(): resolver query: su1.3scale.net finished with 1 answers 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] resolver.lua:388: get_servers(): query for su1.3scale.net finished with 1 answers 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] resolver.lua:328: lookup(): resolver query: tinyproxy.internal 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] cache.lua:115: fetch_answers(): resolver cache read tinyproxy.internal 1 entries 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] cache.lua:190: get(): resolver cache hit: tinyproxy.internal 172.18.0.3 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] resolver.lua:348: lookup(): resolver query: tinyproxy.internal finished with 1 answers 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] resolver.lua:388: get_servers(): query for tinyproxy.internal finished with 1 answers 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] http.lua:50: connect(): connected to ip:172.18.0.3 host: tinyproxy.internal port: 8888 ok: 1 err: nil 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] proxy.lua:82: new(): connection to tinyproxy.internal:8888 established, pool: tinyproxy.internal:8888:su1.3scale.net:443 reused times: 0 2019/04/25 20:37:41 [debug] 28#28: *32 [lua] http.lua:633: send_request(): CONNECT 54.83.62.186:443 HTTP/1.1 User-Agent: lua-resty-http/0.12 (Lua) ngx_lua/10013 Host: su1.3scale.net 2019/04/25 20:37:42 [debug] 28#28: *32 [lua] http.lua:633: send_request(): GET /transactions/authrep.xml?service_token=b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c&service_id=2555417761253&usage%5Bhits%5D=1&user_key= HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 3scale-Options: rejection_reason_header=1&limit_headers=1&no_body=1 Connection: Keep-Alive Host: su1.3scale.net 2019/04/25 20:37:42 [info] 28#28: *32 [lua] backend_client.lua:139: call_backend_transaction(): backend client uri: https://su1.3scale.net/transactions/authrep.xml?service_token=b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c&service_id=2555417761253&usage%5Bhits%5D=1&user_key= ok: true status: 200 body: error: nil, client: 172.18.0.5, server: _, request: "GET /?user_key= HTTP/1.1", host: "echo-api.3scale.net" 2019/04/25 20:37:42 [debug] 28#28: *32 [lua] proxy.lua:368: handle_backend_response(): [backend] response status: 200 body: 2019/04/25 20:37:42 [debug] 28#28: *32 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Metrics, i: 4 2019/04/25 20:37:42 [debug] 28#28: *32 [lua] executor.lua:25: log(): executor phase: log 2019/04/25 20:37:42 [debug] 28#28: *32 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Load Configuration, i: 1 2019/04/25 20:37:42 [debug] 28#28: *32 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Find Service Policy, i: 2 2019/04/25 20:37:42 [debug] 28#28: *32 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Local Policy Chain, i: 3 2019/04/25 20:37:42 [debug] 28#28: *32 [lua] policy_chain.lua:162: policy chain execute phase: log, policy: APIcast, i: 1 2019/04/25 20:37:42 [debug] 28#28: *32 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Metrics, i: 4 2019/04/25 20:37:42 [info] 28#28: *32 client 172.18.0.5 closed keepalive connection [25/Apr/2019:20:37:42 +0000] apicast.internal:8080 172.18.0.5:38182 "GET /?user_key= HTTP/1.1" 200 584 (3.600) 0.9 CONNECT Apr 25 20:37:40 [10]: Connect (file descriptor 9): apicast.internal [172.18.0.2] CONNECT Apr 25 20:37:40 [10]: Request (file descriptor 9): CONNECT 18.213.6.196:443 HTTP/1.1 INFO Apr 25 20:37:40 [10]: No upstream proxy for 18.213.6.196 CONNECT Apr 25 20:37:41 [10]: Established connection to host "18.213.6.196" using file descriptor 10. INFO Apr 25 20:37:41 [10]: Not sending client headers to remote machine INFO Apr 25 20:37:41 [10]: Closed connection between local client (fd:9) and remote client (fd:10) CONNECT Apr 25 20:37:41 [11]: Connect (file descriptor 9): apicast.internal [172.18.0.2] CONNECT Apr 25 20:37:41 [11]: Request (file descriptor 9): CONNECT 54.83.62.186:443 HTTP/1.1 INFO Apr 25 20:37:41 [11]: No upstream proxy for 54.83.62.186 CONNECT Apr 25 20:37:41 [11]: Established connection to host "54.83.62.186" using file descriptor 10. INFO Apr 25 20:37:41 [11]: Not sending client headers to remote machine INFO Apr 25 20:37:42 [11]: Closed connection between local client (fd:9) and remote client (fd:10) # 3. HTTP POST using http://echo-api.3scale.net:80 (NOK) # 3.1. 'curl' using Tinyproxy directly: # export http_proxy="http://tinyproxy.internal:8888" ; export https_proxy=$http_proxy # curl -v -X POST --data "some data" http://echo-api.3scale.net:80 * About to connect() to proxy tinyproxy.internal port 8888 (#0) * Trying 172.18.0.3... * Connected to tinyproxy.internal (172.18.0.3) port 8888 (#0) > POST http://echo-api.3scale.net:80/ HTTP/1.1 > User-Agent: curl/7.29.0 > Host: echo-api.3scale.net > Accept: */* > Proxy-Connection: Keep-Alive > Content-Length: 9 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 9 out of 9 bytes < HTTP/1.1 200 OK < Via: 1.1 tinyproxy (tinyproxy/1.8.3) < X-Content-Type-Options: nosniff < Content-Length: 654 < Content-Type: application/json < Vary: Origin < Set-Cookie: d8c1dd0e39ac4456ed39ce5889b9a5a5=497798f4606acdfb2787fcfc67ab0b88; path=/; HttpOnly < { "method": "POST", "path": "/", "args": "", "body": "some data", "headers": { "HTTP_VERSION": "HTTP/1.1", "HTTP_HOST": "echo-api.3scale.net", "HTTP_ACCEPT": "*/*", "HTTP_USER_AGENT": "curl/7.29.0", "HTTP_VIA": "1.1 tinyproxy (tinyproxy/1.8.3)", "HTTP_X_FORWARDED_FOR": "201.95.85.39, 10.0.103.54", "HTTP_X_FORWARDED_HOST": "echo-api.3scale.net", "HTTP_X_FORWARDED_PORT": "80", "HTTP_X_FORWARDED_PROTO": "http", "HTTP_FORWARDED": "for=10.0.103.54;host=echo-api.3scale.net;proto=http" }, "uuid": "9206cb29-b4f5-4515-a238-9b0e5adc75e2", "bodySha1": "uvNFUf7LSKzD2oaOuF4bbayd41Y=", "bodyLength": 9 * Connection #0 to host tinyproxy.internal left intact } 3.2. 'curl' using APIcast: # unset http_proxy https_proxy # curl -v -X POST --data "some data" "http://apicast.internal:8080/?user_key=" * About to connect() to apicast.internal port 8080 (#0) * Trying 172.18.0.2... * Connected to apicast.internal (172.18.0.2) port 8080 (#0) > POST /?user_key= HTTP/1.1 > User-Agent: curl/7.29.0 > Host: apicast.internal:8080 > Accept: */* > Content-Length: 9 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 9 out of 9 bytes < HTTP/1.1 504 Gateway Time-out < Server: openresty/1.13.6.2 < Date: Thu, 25 Apr 2019 20:44:24 GMT < Content-Type: text/html < Content-Length: 189 < Connection: keep-alive < 504 Gateway Time-out

504 Gateway Time-out

openresty/1.13.6.2 * Connection #0 to host apicast.internal left intact 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] executor.lua:25: rewrite(): executor phase: rewrite 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Load Configuration, i: 1 2019/04/25 20:43:21 [info] 27#27: *41 [lua] configuration_loader.lua:213: rewrite(): lazy loading configuration for: apicast.internal, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] env.lua:86: fetch(): env: APICAST_CONFIGURATION = nil 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] env.lua:86: fetch(): env: THREESCALE_CONFIG_FILE = nil 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] env.lua:86: fetch(): env: APICAST_SERVICES_LIST = nil 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] env.lua:86: fetch(): env: APICAST_SERVICES = nil 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] resolver.lua:197: new(): resolver search domains: 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] cache.lua:122: fetch_answers(): resolver cache miss ekonecsn-admin.3scale.net 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] cache.lua:188: get(): resolver cache miss: ekonecsn-admin.3scale.net 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] resolver.lua:294: search_dns(): resolver query: ekonecsn-admin.3scale.net search: query: ekonecsn-admin.3scale.net. 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] cache.lua:122: fetch_answers(): resolver cache miss ekonecsn-admin.3scale.net. 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] cache.lua:188: get(): resolver cache miss: ekonecsn-admin.3scale.net. 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] dns_client.lua:43: init_resolvers(): initializing 4 nameservers 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] dns_client.lua:56: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] dns_client.lua:56: init_resolvers(): nameserver 8.8.8.8:53 initialized 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] dns_client.lua:56: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] dns_client.lua:56: init_resolvers(): nameserver 8.8.8.8:53 initialized 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] dns_client.lua:68: query(): resolver query: ekonecsn-admin.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] cache.lua:74: store(): resolver cache write ekonecsn-admin.3scale.net with TLL 599 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] cache.lua:74: store(): resolver cache write multitenant.3scale.net with TLL 299 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] cache.lua:74: store(): resolver cache write 3scale-20120424-299787090.us-east-1.elb.amazonaws.com with TLL 59 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] resolver.lua:328: lookup(): resolver query: 50.19.254.27 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] resolver.lua:333: lookup(): host is ip address: 50.19.254.27 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] resolver.lua:348: lookup(): resolver query: 50.19.254.27 finished with 1 answers 2019/04/25 20:43:21 [debug] 27#27: *41 [lua] resolver.lua:388: get_servers(): query for 50.19.254.27 finished with 1 answers 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] http.lua:50: connect(): connected to ip:50.19.254.27 host: 50.19.254.27 port: 443 ok: 1 err: nil 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 0 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] http.lua:633: send_request(): GET /admin/api/services.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] remote_v2.lua:234: services(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services.json 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] env.lua:86: fetch(): env: APICAST_SERVICE_2555417761253_CONFIGURATION_VERSION = nil 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] cache.lua:115: fetch_answers(): resolver cache read ekonecsn-admin.3scale.net 1 entries 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] cache.lua:115: fetch_answers(): resolver cache read multitenant.3scale.net 1 entries 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] cache.lua:115: fetch_answers(): resolver cache read 3scale-20120424-299787090.us-east-1.elb.amazonaws.com 2 entries 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] cache.lua:190: get(): resolver cache hit: ekonecsn-admin.3scale.net 50.19.254.27, 23.23.89.189 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] resolver.lua:328: lookup(): resolver query: 23.23.89.189 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] resolver.lua:333: lookup(): host is ip address: 23.23.89.189 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] resolver.lua:348: lookup(): resolver query: 23.23.89.189 finished with 1 answers 2019/04/25 20:43:22 [debug] 27#27: *41 [lua] resolver.lua:388: get_servers(): query for 23.23.89.189 finished with 1 answers 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] http.lua:50: connect(): connected to ip:23.23.89.189 host: 23.23.89.189 port: 443 ok: 1 err: nil 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 0 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] http.lua:633: send_request(): GET /admin/api/services/2555417761253/proxy/configs/production/latest.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] remote_v2.lua:278: config(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services/2555417761253/proxy/configs/production/latest.json body: {"proxy_config":{"id":122310,"version":85,"environment":"production","content":{"id":2555417761253,"account_id":2445582593288,"name":"Echo API Hosted","oneline_description":null,"description":"","txt_api":null,"txt_support":null,"txt_features":null,"created_at":"2018-10-17T17:49:41-03:00","updated_at":"2019-04-25T17:42:32-03:00","logo_file_name":null,"logo_content_type":null,"logo_file_size":null,"state":"incomplete","intentions_required":false,"draft_name":"","infobar":null,"terms":null,"display_provider_keys":false,"credit_card_support_email":null,"buyers_manage_apps":true,"buyers_manage_keys":true,"custom_keys_enabled":true,"buyer_plan_change_permission":"request","buyer_can_select_plan":false,"notification_settings":{"web_provider":[0],"email_provider":[0],"web_buyer":[0,90,100],"email_buyer":[0,90,100]},"default_application_plan_id":2357355960517,"default_service_plan_id":2357355960516,"default_end_user_plan_id":null,"end_user_registration_required":true,"tenant_id":2445582593288,"system_name":"echo_api_hosted","backend_version":"1","mandatory_app_key":true,"buyer_key_regenerate_enabled":true,"support_email":"ekonecsn@redhat.com","referrer_filters_required":false,"deployment_option":"self_managed","kubernetes_service_link":null,"tech_support_email":null,"admin_support_email":null,"proxiable?":true,"backend_authentication_type":"service_token","backend_authentication_value":"b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c","proxy":{"id":107499,"tenant_id":2445582593288,"service_id":2555417761253,"endpoint":"http://apicast.internal:8080","deployed_at":null,"api_backend":"http://echo-api.3scale.net:80","auth_app_key":"app_key","auth_app_id":"app_id","auth_user_key":"user_key","credentials_location":"query","error_auth_failed":"Authentication failed","error_auth_missing":"Authentication parameters missing","created_at":"2018-10-17T17:49:41-03:00","updated_at":"2019-04-25T17:42:32-03:00","error_status_auth_failed":403,"error_headers_auth_failed":"text/plain; charset=us-ascii","error_status_auth_missing":403,"error_headers_auth_missing":"text/plain; charset=us-ascii","error_no_match":"No Mapping Rule matched","error_status_no_match":404,"error_headers_no_match":"text/plain; charset=us-ascii","secret_token":"Shared_secret_sent_from_proxy_to_API_backend_ec120e006f910d45","hostname_rewrite":"","oauth_login_url":null,"sandbox_endpoint":"http://apicast.internal:8080","api_test_path":"/","api_test_success":null,"apicast_configuration_driven":true,"oidc_issuer_endpoint":null,"lock_version":90,"authentication_method":"1","hostname_rewrite_for_sandbox":"echo-api.3scale.net","endpoint_port":8080,"valid?":true,"service_backend_version":"1","hosts":["apicast.internal"],"backend":{"endpoint":"https://su1.3scale.net","host":"su1.3scale.net"},"policy_chain":[{"name":"apicast","version":"builtin","configuration":{}}],"proxy_rules":[{"id":434579,"proxy_id":107499,"http_method":"GET","pattern":"/","metric_id":2555418124738,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-04-16T13:42:31-03:00","updated_at":"2019-04-25T16:53:28-03:00","redirect_url":null,"position":1,"last":false,"parameters":[],"querystring_parameters":{}},{"id":442138,"proxy_id":107499,"http_method":"POST","pattern":"/","metric_id":2555418124738,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-04-25T14:42:49-03:00","updated_at":"2019-04-25T16:53:28-03:00","redirect_url":null,"position":2,"last":false,"parameters":[],"querystring_parameters":{}}]}}}} 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] env.lua:86: fetch(): env: APICAST_SERVICE_2555417768857_CONFIGURATION_VERSION = nil 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:115: fetch_answers(): resolver cache read ekonecsn-admin.3scale.net 1 entries 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:115: fetch_answers(): resolver cache read multitenant.3scale.net 1 entries 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:115: fetch_answers(): resolver cache read 3scale-20120424-299787090.us-east-1.elb.amazonaws.com 2 entries 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:190: get(): resolver cache hit: ekonecsn-admin.3scale.net 50.19.254.27, 23.23.89.189 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:328: lookup(): resolver query: 50.19.254.27 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:333: lookup(): host is ip address: 50.19.254.27 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:348: lookup(): resolver query: 50.19.254.27 finished with 1 answers 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:388: get_servers(): query for 50.19.254.27 finished with 1 answers 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] http.lua:50: connect(): connected to ip:50.19.254.27 host: 50.19.254.27 port: 443 ok: 1 err: nil 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 1 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] http.lua:633: send_request(): GET /admin/api/services/2555417768857/proxy/configs/production/latest.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] remote_v2.lua:278: config(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services/2555417768857/proxy/configs/production/latest.json body: {"proxy_config":{"id":115033,"version":32,"environment":"production","content":{"id":2555417768857,"account_id":2445582593288,"name":"Test API","oneline_description":null,"description":"Test API Using Service Create","txt_api":null,"txt_support":null,"txt_features":null,"created_at":"2019-02-05T19:19:36-02:00","updated_at":"2019-04-11T16:23:03-03:00","logo_file_name":null,"logo_content_type":null,"logo_file_size":null,"state":"incomplete","intentions_required":false,"draft_name":"","infobar":null,"terms":null,"display_provider_keys":false,"credit_card_support_email":null,"buyers_manage_apps":true,"buyers_manage_keys":true,"custom_keys_enabled":true,"buyer_plan_change_permission":"request","buyer_can_select_plan":false,"notification_settings":{"web_provider":[0],"email_provider":[0],"web_buyer":[0],"email_buyer":[0]},"default_application_plan_id":null,"default_service_plan_id":null,"default_end_user_plan_id":null,"end_user_registration_required":true,"tenant_id":2445582593288,"system_name":"test_api","backend_version":"1","mandatory_app_key":true,"buyer_key_regenerate_enabled":true,"support_email":"ekonecsn@redhat.com","referrer_filters_required":true,"deployment_option":"self_managed","kubernetes_service_link":null,"tech_support_email":null,"admin_support_email":null,"proxiable?":true,"backend_authentication_type":"service_token","backend_authentication_value":"fc80b523e672cc8ff5df22b0f3914f87f60a82c168d1ce34c52df3d25cf4e9be","proxy":{"id":115061,"tenant_id":2445582593288,"service_id":2555417768857,"endpoint":"http://localhost:8080","deployed_at":null,"api_backend":"https://echo-api.3scale.net:443","auth_app_key":"app_key","auth_app_id":"app_id","auth_user_key":"user_key","credentials_location":"query","error_auth_failed":"Authentication failed","error_auth_missing":"Authentication parameters missing","created_at":"2019-02-05T19:19:37-02:00","updated_at":"2019-04-11T16:23:03-03:00","error_status_auth_failed":403,"error_headers_auth_failed":"text/plain; charset=us-ascii","error_status_auth_missing":403,"error_headers_auth_missing":"text/plain; charset=us-ascii","error_no_match":"No Mapping Rule matched","error_status_no_match":404,"error_headers_no_match":"text/plain; charset=us-ascii","secret_token":"Shared_secret_sent_from_proxy_to_API_backend_a5fb94644b900716","hostname_rewrite":"","oauth_login_url":null,"sandbox_endpoint":"http://localhost:8080","api_test_path":"/","api_test_success":null,"apicast_configuration_driven":true,"oidc_issuer_endpoint":null,"lock_version":35,"authentication_method":"1","hostname_rewrite_for_sandbox":"echo-api.3scale.net","endpoint_port":8080,"valid?":true,"service_backend_version":"1","hosts":["localhost"],"backend":{"endpoint":"https://su1.3scale.net","host":"su1.3scale.net"},"policy_chain":[{"name":"upstream","version":"builtin","configuration":{"rules":[{"url":"http://apicast-url2.internal","regex":"^/apicast.internal/.*"}]}},{"name":"apicast","version":"builtin","configuration":{}}],"proxy_rules":[{"id":339975,"proxy_id":115061,"http_method":"GET","pattern":"/testapi","metric_id":2555418152044,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-02-18T17:29:40-03:00","updated_at":"2019-04-11T16:23:03-03:00","redirect_url":null,"position":1,"last":false,"parameters":[],"querystring_parameters":{}}]}}}} 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_loader.lua:150: loading policy: upstream version: builtin 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_loader.lua:150: loading policy: upstream version: builtin 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:43:23 [info] 27#27: *41 [lua] configuration_store.lua:124: store(): added service 2555417761253 configuration with hosts: apicast.internal ttl: 0, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:43:23 [info] 27#27: *41 [lua] configuration_store.lua:124: store(): added service 2555417768857 configuration with hosts: localhost ttl: 0, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Find Service Policy, i: 2 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Local Policy Chain, i: 3 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] env.lua:86: fetch(): env: APICAST_BACKEND_CACHE_HANDLER = nil 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache_handler.lua:23: new(): backend cache handler: strict 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: APIcast, i: 1 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] service.lua:227: get_usage(): [mapping] service 2555417761253 has 2 rules 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Metrics, i: 4 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] executor.lua:25: access(): executor phase: access 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Load Configuration, i: 1 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Find Service Policy, i: 2 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Local Policy Chain, i: 3 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: policy chain execute phase: access, policy: APIcast, i: 1 2019/04/25 20:43:23 [info] 27#27: *41 [lua] proxy.lua:81: output_debug_headers(): usage: usage%5Bhits%5D=1 credentials: user_key=, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] proxy.lua:145: apicast cache hit key: 2555417761253::usage%5Bhits%5D=1 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Metrics, i: 4 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] executor.lua:25: content(): executor phase: content 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Load Configuration, i: 1 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Find Service Policy, i: 2 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Local Policy Chain, i: 3 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] policy_chain.lua:162: policy chain execute phase: content, policy: APIcast, i: 1 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] upstream.lua:181: call(): using proxy: http://tinyproxy.internal:8888 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:328: lookup(): resolver query: tinyproxy.internal 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:122: fetch_answers(): resolver cache miss tinyproxy.internal 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:188: get(): resolver cache miss: tinyproxy.internal 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:294: search_dns(): resolver query: tinyproxy.internal search: query: tinyproxy.internal. 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:122: fetch_answers(): resolver cache miss tinyproxy.internal. 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:188: get(): resolver cache miss: tinyproxy.internal. 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] dns_client.lua:68: query(): resolver query: tinyproxy.internal. nameserver: 127.0.0.1:5353 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:74: store(): resolver cache write tinyproxy.internal with TLL 600 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:348: lookup(): resolver query: tinyproxy.internal finished with 1 answers 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:388: resolve_servers(): query for tinyproxy.internal finished with 1 answers 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:328: lookup(): resolver query: echo-api.3scale.net 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:122: fetch_answers(): resolver cache miss echo-api.3scale.net 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:188: get(): resolver cache miss: echo-api.3scale.net 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] resolver.lua:294: search_dns(): resolver query: echo-api.3scale.net search: query: echo-api.3scale.net. 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:122: fetch_answers(): resolver cache miss echo-api.3scale.net. 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] cache.lua:188: get(): resolver cache miss: echo-api.3scale.net. 2019/04/25 20:43:23 [debug] 27#27: *41 [lua] dns_client.lua:68: query(): resolver query: echo-api.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:43:24 [debug] 27#27: *41 [lua] cache.lua:74: store(): resolver cache write echo-api.3scale.net with TLL 299 2019/04/25 20:43:24 [debug] 27#27: *41 [lua] cache.lua:74: store(): resolver cache write tf-lb-0081dad066b2afef5328e0256a-2081992367.us-east-1.elb.amazonaws.com with TLL 59 2019/04/25 20:43:24 [debug] 27#27: *41 [lua] resolver.lua:348: lookup(): resolver query: echo-api.3scale.net finished with 3 answers 2019/04/25 20:43:24 [debug] 27#27: *41 [lua] resolver.lua:388: resolve_servers(): query for echo-api.3scale.net finished with 3 answers 2019/04/25 20:43:24 [debug] 27#27: *41 [lua] executor.lua:25: balancer(): executor phase: balancer 2019/04/25 20:43:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: balancer(): policy chain execute phase: balancer, policy: Load Configuration, i: 1 2019/04/25 20:43:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: balancer(): policy chain execute phase: balancer, policy: Find Service Policy, i: 2 2019/04/25 20:43:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: balancer(): policy chain execute phase: balancer, policy: Local Policy Chain, i: 3 2019/04/25 20:43:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: policy chain execute phase: balancer, policy: APIcast, i: 1 2019/04/25 20:43:24 [info] 27#27: *41 [lua] balancer.lua:108: set_current_peer(): balancer set peer 172.18.0.3:8888 ok: true err: nil while connecting to upstream, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "echo-api.3scale.net" 2019/04/25 20:43:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: balancer(): policy chain execute phase: balancer, policy: Metrics, i: 4 2019/04/25 20:44:24 [error] 27#27: *41 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", upstream: "http://172.18.0.3:8888http://18.214.132.12:80?user_key=", host: "echo-api.3scale.net" 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] executor.lua:25: header_filter(): executor phase: header_filter 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Load Configuration, i: 1 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Find Service Policy, i: 2 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Local Policy Chain, i: 3 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: policy chain execute phase: header_filter, policy: APIcast, i: 1 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Metrics, i: 4 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] executor.lua:25: body_filter(): executor phase: body_filter 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Load Configuration, i: 1 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Find Service Policy, i: 2 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Local Policy Chain, i: 3 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: policy chain execute phase: body_filter, policy: APIcast, i: 1 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Metrics, i: 4 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] executor.lua:25: post_action(): executor phase: post_action 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Load Configuration, i: 1 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Find Service Policy, i: 2 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Local Policy Chain, i: 3 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] policy_chain.lua:162: policy chain execute phase: post_action, policy: APIcast, i: 1 2019/04/25 20:44:24 [info] 27#27: *41 [lua] proxy.lua:335: [async] reporting to backend asynchronously, cached_key: 2555417761253::usage%5Bhits%5D=1 while sending to client, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", upstream: "http://172.18.0.3:8888http://18.214.132.12:80?user_key=", host: "echo-api.3scale.net" 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] resolver.lua:328: lookup(): resolver query: su1.3scale.net 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] cache.lua:122: fetch_answers(): resolver cache miss su1.3scale.net 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] cache.lua:188: get(): resolver cache miss: su1.3scale.net 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] resolver.lua:294: search_dns(): resolver query: su1.3scale.net search: query: su1.3scale.net. 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] cache.lua:122: fetch_answers(): resolver cache miss su1.3scale.net. 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] cache.lua:188: get(): resolver cache miss: su1.3scale.net. 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] dns_client.lua:68: query(): resolver query: su1.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] cache.lua:74: store(): resolver cache write su1.3scale.net with TLL 59 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] resolver.lua:348: lookup(): resolver query: su1.3scale.net finished with 1 answers 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] resolver.lua:388: get_servers(): query for su1.3scale.net finished with 1 answers 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] resolver.lua:328: lookup(): resolver query: tinyproxy.internal 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] cache.lua:115: fetch_answers(): resolver cache read tinyproxy.internal 1 entries 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] cache.lua:190: get(): resolver cache hit: tinyproxy.internal 172.18.0.3 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] resolver.lua:348: lookup(): resolver query: tinyproxy.internal finished with 1 answers 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] resolver.lua:388: get_servers(): query for tinyproxy.internal finished with 1 answers 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] http.lua:50: connect(): connected to ip:172.18.0.3 host: tinyproxy.internal port: 8888 ok: 1 err: nil 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] proxy.lua:82: new(): connection to tinyproxy.internal:8888 established, pool: tinyproxy.internal:8888:su1.3scale.net:443 reused times: 0 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] http.lua:633: send_request(): CONNECT 54.83.63.187:443 HTTP/1.1 User-Agent: lua-resty-http/0.12 (Lua) ngx_lua/10013 Host: su1.3scale.net 2019/04/25 20:44:24 [debug] 27#27: *41 [lua] http.lua:633: send_request(): GET /transactions/authrep.xml?service_token=b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c&service_id=2555417761253&usage%5Bhits%5D=1&user_key= HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 3scale-Options: rejection_reason_header=1&limit_headers=1&no_body=1 Connection: Keep-Alive Host: su1.3scale.net 2019/04/25 20:44:25 [info] 27#27: *41 [lua] backend_client.lua:139: call_backend_transaction(): backend client uri: https://su1.3scale.net/transactions/authrep.xml?service_token=b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c&service_id=2555417761253&usage%5Bhits%5D=1&user_key= ok: true status: 200 body: error: nil while sending to client, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", upstream: "http://172.18.0.3:8888http://18.214.132.12:80?user_key=", host: "echo-api.3scale.net" 2019/04/25 20:44:25 [debug] 27#27: *41 [lua] proxy.lua:368: handle_backend_response(): [backend] response status: 200 body: 2019/04/25 20:44:25 [debug] 27#27: *41 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Metrics, i: 4 2019/04/25 20:44:25 [debug] 27#27: *41 [lua] executor.lua:25: log(): executor phase: log 2019/04/25 20:44:25 [debug] 27#27: *41 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Load Configuration, i: 1 2019/04/25 20:44:25 [debug] 27#27: *41 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Find Service Policy, i: 2 2019/04/25 20:44:25 [debug] 27#27: *41 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Local Policy Chain, i: 3 2019/04/25 20:44:25 [debug] 27#27: *41 [lua] policy_chain.lua:162: policy chain execute phase: log, policy: APIcast, i: 1 2019/04/25 20:44:25 [debug] 27#27: *41 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Metrics, i: 4 2019/04/25 20:44:25 [info] 27#27: *41 client 172.18.0.5 closed keepalive connection [25/Apr/2019:20:44:25 +0000] apicast.internal:8080 172.18.0.5:38320 "POST /?user_key= HTTP/1.1" 504 189 (63.400) 1 CONNECT Apr 25 20:43:24 [16]: Connect (file descriptor 9): apicast.internal [172.18.0.2] CONNECT Apr 25 20:43:24 [16]: Request (file descriptor 9): POST http://18.214.132.12:80?user_key= HTTP/1.1 INFO Apr 25 20:43:24 [16]: No upstream proxy for 18.214.132.12 CONNECT Apr 25 20:43:24 [16]: Established connection to host "18.214.132.12" using file descriptor 10. CONNECT Apr 25 20:44:24 [17]: Connect (file descriptor 9): apicast.internal [172.18.0.2] CONNECT Apr 25 20:44:24 [17]: Request (file descriptor 9): CONNECT 54.83.63.187:443 HTTP/1.1 INFO Apr 25 20:44:24 [17]: No upstream proxy for 54.83.63.187 CONNECT Apr 25 20:44:24 [17]: Established connection to host "54.83.63.187" using file descriptor 10. INFO Apr 25 20:44:24 [17]: Not sending client headers to remote machine INFO Apr 25 20:44:25 [17]: Closed connection between local client (fd:9) and remote client (fd:10) # 4. HTTP POST using https://echo-api.3scale.net:443 (NOK) # 4.1. 'curl' using Tinyproxy directly: # export http_proxy="http://tinyproxy.internal:8888" ; export https_proxy=$http_proxy # curl -v -X POST --data "some data" https://echo-api.3scale.net:443 * About to connect() to proxy tinyproxy.internal port 8888 (#0) * Trying 172.18.0.3... * Connected to tinyproxy.internal (172.18.0.3) port 8888 (#0) * Establish HTTP proxy tunnel to echo-api.3scale.net:443 > CONNECT echo-api.3scale.net:443 HTTP/1.1 > Host: echo-api.3scale.net:443 > User-Agent: curl/7.29.0 > Proxy-Connection: Keep-Alive > < HTTP/1.0 200 Connection established < Proxy-agent: tinyproxy/1.8.3 < * Proxy replied OK to CONNECT request * Initializing NSS with certpath: sql:/etc/pki/nssdb * CAfile: /etc/pki/tls/certs/ca-bundle.crt CApath: none * SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 * Server certificate: * subject: CN=*.3scale.net * start date: Apr 04 00:00:00 2019 GMT * expire date: May 04 12:00:00 2020 GMT * common name: *.3scale.net * issuer: CN=Amazon,OU=Server CA 1B,O=Amazon,C=US > POST / HTTP/1.1 > User-Agent: curl/7.29.0 > Host: echo-api.3scale.net > Accept: */* > Content-Length: 9 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 9 out of 9 bytes < HTTP/1.1 200 OK < Content-Type: application/json < Set-Cookie: d8c1dd0e39ac4456ed39ce5889b9a5a5=1f3ae150fd257186107c0631ee0eaaa3; path=/; HttpOnly < Vary: Origin < X-Content-Type-Options: nosniff < Content-Length: 606 < Connection: keep-alive < { "method": "POST", "path": "/", "args": "", "body": "some data", "headers": { "HTTP_VERSION": "HTTP/1.1", "HTTP_HOST": "echo-api.3scale.net", "HTTP_ACCEPT": "*/*", "HTTP_USER_AGENT": "curl/7.29.0", "HTTP_X_FORWARDED_FOR": "201.95.85.39, 10.0.103.54", "HTTP_X_FORWARDED_HOST": "echo-api.3scale.net", "HTTP_X_FORWARDED_PORT": "443", "HTTP_X_FORWARDED_PROTO": "https", "HTTP_FORWARDED": "for=10.0.103.54;host=echo-api.3scale.net;proto=https" }, "uuid": "7403b013-d41e-4437-b7a4-356c3c22d7bf", "bodySha1": "uvNFUf7LSKzD2oaOuF4bbayd41Y=", "bodyLength": 9 * Connection #0 to host tinyproxy.internal left intact } 4.2. 'curl' using APIcast: # unset http_proxy https_proxy # curl -v -X POST --data "some data" "http://apicast.internal:8080/?user_key=" * About to connect() to apicast.internal port 8080 (#0) * Trying 172.18.0.2... * Connected to apicast.internal (172.18.0.2) port 8080 (#0) > POST /?user_key= HTTP/1.1 > User-Agent: curl/7.29.0 > Host: apicast.internal:8080 > Accept: */* > Content-Length: 9 > Content-Type: application/x-www-form-urlencoded > * upload completely sent off: 9 out of 9 bytes < HTTP/1.1 504 Gateway Time-out < Server: openresty/1.13.6.2 < Date: Thu, 25 Apr 2019 20:49:53 GMT < Content-Type: text/html < Transfer-Encoding: chunked < Connection: keep-alive < Cache-Control: no-cache <

504 Gateway Time-out

The server didn't respond in time. * Connection #0 to host apicast.internal left intact 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] executor.lua:25: rewrite(): executor phase: rewrite 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Load Configuration, i: 1 2019/04/25 20:49:17 [info] 29#29: *50 [lua] configuration_loader.lua:213: rewrite(): lazy loading configuration for: apicast.internal, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] resolver.lua:197: new(): resolver search domains: 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] cache.lua:122: fetch_answers(): resolver cache miss ekonecsn-admin.3scale.net 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] cache.lua:188: get(): resolver cache miss: ekonecsn-admin.3scale.net 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] resolver.lua:294: search_dns(): resolver query: ekonecsn-admin.3scale.net search: query: ekonecsn-admin.3scale.net. 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] cache.lua:122: fetch_answers(): resolver cache miss ekonecsn-admin.3scale.net. 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] cache.lua:188: get(): resolver cache miss: ekonecsn-admin.3scale.net. 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] dns_client.lua:43: init_resolvers(): initializing 4 nameservers 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] dns_client.lua:56: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] dns_client.lua:56: init_resolvers(): nameserver 8.8.8.8:53 initialized 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] dns_client.lua:56: init_resolvers(): nameserver 127.0.0.1:5353 initialized 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] dns_client.lua:56: init_resolvers(): nameserver 8.8.8.8:53 initialized 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] dns_client.lua:68: query(): resolver query: ekonecsn-admin.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] cache.lua:74: store(): resolver cache write ekonecsn-admin.3scale.net with TLL 599 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] cache.lua:74: store(): resolver cache write multitenant.3scale.net with TLL 299 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] cache.lua:74: store(): resolver cache write 3scale-20120424-299787090.us-east-1.elb.amazonaws.com with TLL 59 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] resolver.lua:328: lookup(): resolver query: 23.23.89.189 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] resolver.lua:333: lookup(): host is ip address: 23.23.89.189 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] resolver.lua:348: lookup(): resolver query: 23.23.89.189 finished with 1 answers 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] resolver.lua:388: get_servers(): query for 23.23.89.189 finished with 1 answers 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] http.lua:50: connect(): connected to ip:23.23.89.189 host: 23.23.89.189 port: 443 ok: 1 err: nil 2019/04/25 20:49:17 [debug] 29#29: *50 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 0 2019/04/25 20:49:18 [debug] 29#29: *50 [lua] http.lua:633: send_request(): GET /admin/api/services.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] remote_v2.lua:234: services(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services.json 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] cache.lua:115: fetch_answers(): resolver cache read ekonecsn-admin.3scale.net 1 entries 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] cache.lua:115: fetch_answers(): resolver cache read multitenant.3scale.net 1 entries 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] cache.lua:115: fetch_answers(): resolver cache read 3scale-20120424-299787090.us-east-1.elb.amazonaws.com 2 entries 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] cache.lua:190: get(): resolver cache hit: ekonecsn-admin.3scale.net 23.23.89.189, 50.19.254.27 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] resolver.lua:328: lookup(): resolver query: 50.19.254.27 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] resolver.lua:333: lookup(): host is ip address: 50.19.254.27 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] resolver.lua:348: lookup(): resolver query: 50.19.254.27 finished with 1 answers 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] resolver.lua:388: get_servers(): query for 50.19.254.27 finished with 1 answers 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] http.lua:50: connect(): connected to ip:50.19.254.27 host: 50.19.254.27 port: 443 ok: 1 err: nil 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 0 2019/04/25 20:49:20 [debug] 29#29: *50 [lua] http.lua:633: send_request(): GET /admin/api/services/2555417761253/proxy/configs/production/latest.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] remote_v2.lua:278: config(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services/2555417761253/proxy/configs/production/latest.json body: {"proxy_config":{"id":122312,"version":86,"environment":"production","content":{"id":2555417761253,"account_id":2445582593288,"name":"Echo API Hosted","oneline_description":null,"description":"","txt_api":null,"txt_support":null,"txt_features":null,"created_at":"2018-10-17T17:49:41-03:00","updated_at":"2019-04-25T17:48:58-03:00","logo_file_name":null,"logo_content_type":null,"logo_file_size":null,"state":"incomplete","intentions_required":false,"draft_name":"","infobar":null,"terms":null,"display_provider_keys":false,"credit_card_support_email":null,"buyers_manage_apps":true,"buyers_manage_keys":true,"custom_keys_enabled":true,"buyer_plan_change_permission":"request","buyer_can_select_plan":false,"notification_settings":{"web_provider":[0],"email_provider":[0],"web_buyer":[0,90,100],"email_buyer":[0,90,100]},"default_application_plan_id":2357355960517,"default_service_plan_id":2357355960516,"default_end_user_plan_id":null,"end_user_registration_required":true,"tenant_id":2445582593288,"system_name":"echo_api_hosted","backend_version":"1","mandatory_app_key":true,"buyer_key_regenerate_enabled":true,"support_email":"ekonecsn@redhat.com","referrer_filters_required":false,"deployment_option":"self_managed","kubernetes_service_link":null,"tech_support_email":null,"admin_support_email":null,"proxiable?":true,"backend_authentication_type":"service_token","backend_authentication_value":"b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c","proxy":{"id":107499,"tenant_id":2445582593288,"service_id":2555417761253,"endpoint":"http://apicast.internal:8080","deployed_at":null,"api_backend":"https://echo-api.3scale.net:443","auth_app_key":"app_key","auth_app_id":"app_id","auth_user_key":"user_key","credentials_location":"query","error_auth_failed":"Authentication failed","error_auth_missing":"Authentication parameters missing","created_at":"2018-10-17T17:49:41-03:00","updated_at":"2019-04-25T17:48:58-03:00","error_status_auth_failed":403,"error_headers_auth_failed":"text/plain; charset=us-ascii","error_status_auth_missing":403,"error_headers_auth_missing":"text/plain; charset=us-ascii","error_no_match":"No Mapping Rule matched","error_status_no_match":404,"error_headers_no_match":"text/plain; charset=us-ascii","secret_token":"Shared_secret_sent_from_proxy_to_API_backend_ec120e006f910d45","hostname_rewrite":"","oauth_login_url":null,"sandbox_endpoint":"http://apicast.internal:8080","api_test_path":"/","api_test_success":null,"apicast_configuration_driven":true,"oidc_issuer_endpoint":null,"lock_version":91,"authentication_method":"1","hostname_rewrite_for_sandbox":"echo-api.3scale.net","endpoint_port":8080,"valid?":true,"service_backend_version":"1","hosts":["apicast.internal"],"backend":{"endpoint":"https://su1.3scale.net","host":"su1.3scale.net"},"policy_chain":[{"name":"apicast","version":"builtin","configuration":{}}],"proxy_rules":[{"id":434579,"proxy_id":107499,"http_method":"GET","pattern":"/","metric_id":2555418124738,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-04-16T13:42:31-03:00","updated_at":"2019-04-25T16:53:28-03:00","redirect_url":null,"position":1,"last":false,"parameters":[],"querystring_parameters":{}},{"id":442138,"proxy_id":107499,"http_method":"POST","pattern":"/","metric_id":2555418124738,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-04-25T14:42:49-03:00","updated_at":"2019-04-25T16:53:28-03:00","redirect_url":null,"position":2,"last":false,"parameters":[],"querystring_parameters":{}}]}}}} 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] resolver.lua:328: lookup(): resolver query: ekonecsn-admin.3scale.net 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] cache.lua:115: fetch_answers(): resolver cache read ekonecsn-admin.3scale.net 1 entries 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] cache.lua:115: fetch_answers(): resolver cache read multitenant.3scale.net 1 entries 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] cache.lua:115: fetch_answers(): resolver cache read 3scale-20120424-299787090.us-east-1.elb.amazonaws.com 2 entries 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] cache.lua:190: get(): resolver cache hit: ekonecsn-admin.3scale.net 23.23.89.189, 50.19.254.27 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] resolver.lua:348: lookup(): resolver query: ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] resolver.lua:388: get_servers(): query for ekonecsn-admin.3scale.net finished with 4 answers 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] resolver.lua:328: lookup(): resolver query: 23.23.89.189 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] resolver.lua:333: lookup(): host is ip address: 23.23.89.189 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] resolver.lua:348: lookup(): resolver query: 23.23.89.189 finished with 1 answers 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] resolver.lua:388: get_servers(): query for 23.23.89.189 finished with 1 answers 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] http.lua:50: connect(): connected to ip:23.23.89.189 host: 23.23.89.189 port: 443 ok: 1 err: nil 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] proxy.lua:24: new(): connection to ekonecsn-admin.3scale.net:443 established, reused times: 1 2019/04/25 20:49:21 [debug] 29#29: *50 [lua] http.lua:633: send_request(): GET /admin/api/services/2555417768857/proxy/configs/production/latest.json HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 Host: ekonecsn-admin.3scale.net Connection: Keep-Alive Authorization: Basic ZWNmY2E3YmNhNDU4MWNiYjgzMzcwMTdmZmIzYThjMzE6 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] remote_v2.lua:278: config(): services get status: 200 url: https://@ekonecsn-admin.3scale.net/admin/api/services/2555417768857/proxy/configs/production/latest.json body: {"proxy_config":{"id":115033,"version":32,"environment":"production","content":{"id":2555417768857,"account_id":2445582593288,"name":"Test API","oneline_description":null,"description":"Test API Using Service Create","txt_api":null,"txt_support":null,"txt_features":null,"created_at":"2019-02-05T19:19:36-02:00","updated_at":"2019-04-11T16:23:03-03:00","logo_file_name":null,"logo_content_type":null,"logo_file_size":null,"state":"incomplete","intentions_required":false,"draft_name":"","infobar":null,"terms":null,"display_provider_keys":false,"credit_card_support_email":null,"buyers_manage_apps":true,"buyers_manage_keys":true,"custom_keys_enabled":true,"buyer_plan_change_permission":"request","buyer_can_select_plan":false,"notification_settings":{"web_provider":[0],"email_provider":[0],"web_buyer":[0],"email_buyer":[0]},"default_application_plan_id":null,"default_service_plan_id":null,"default_end_user_plan_id":null,"end_user_registration_required":true,"tenant_id":2445582593288,"system_name":"test_api","backend_version":"1","mandatory_app_key":true,"buyer_key_regenerate_enabled":true,"support_email":"ekonecsn@redhat.com","referrer_filters_required":true,"deployment_option":"self_managed","kubernetes_service_link":null,"tech_support_email":null,"admin_support_email":null,"proxiable?":true,"backend_authentication_type":"service_token","backend_authentication_value":"fc80b523e672cc8ff5df22b0f3914f87f60a82c168d1ce34c52df3d25cf4e9be","proxy":{"id":115061,"tenant_id":2445582593288,"service_id":2555417768857,"endpoint":"http://localhost:8080","deployed_at":null,"api_backend":"https://echo-api.3scale.net:443","auth_app_key":"app_key","auth_app_id":"app_id","auth_user_key":"user_key","credentials_location":"query","error_auth_failed":"Authentication failed","error_auth_missing":"Authentication parameters missing","created_at":"2019-02-05T19:19:37-02:00","updated_at":"2019-04-11T16:23:03-03:00","error_status_auth_failed":403,"error_headers_auth_failed":"text/plain; charset=us-ascii","error_status_auth_missing":403,"error_headers_auth_missing":"text/plain; charset=us-ascii","error_no_match":"No Mapping Rule matched","error_status_no_match":404,"error_headers_no_match":"text/plain; charset=us-ascii","secret_token":"Shared_secret_sent_from_proxy_to_API_backend_a5fb94644b900716","hostname_rewrite":"","oauth_login_url":null,"sandbox_endpoint":"http://localhost:8080","api_test_path":"/","api_test_success":null,"apicast_configuration_driven":true,"oidc_issuer_endpoint":null,"lock_version":35,"authentication_method":"1","hostname_rewrite_for_sandbox":"echo-api.3scale.net","endpoint_port":8080,"valid?":true,"service_backend_version":"1","hosts":["localhost"],"backend":{"endpoint":"https://su1.3scale.net","host":"su1.3scale.net"},"policy_chain":[{"name":"upstream","version":"builtin","configuration":{"rules":[{"url":"http://apicast-url2.internal","regex":"^/apicast.internal/.*"}]}},{"name":"apicast","version":"builtin","configuration":{}}],"proxy_rules":[{"id":339975,"proxy_id":115061,"http_method":"GET","pattern":"/testapi","metric_id":2555418152044,"metric_system_name":"hits","delta":1,"tenant_id":2445582593288,"created_at":"2019-02-18T17:29:40-03:00","updated_at":"2019-04-11T16:23:03-03:00","redirect_url":null,"position":1,"last":false,"parameters":[],"querystring_parameters":{}}]}}}} 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_loader.lua:150: loading policy: upstream version: builtin 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_loader.lua:150: loading policy: upstream version: builtin 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_loader.lua:150: loading policy: apicast version: builtin 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] sandbox.lua:84: loader(): sandbox require: init 2019/04/25 20:49:22 [info] 29#29: *50 [lua] configuration_store.lua:124: store(): added service 2555417761253 configuration with hosts: apicast.internal ttl: 0, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:49:22 [info] 29#29: *50 [lua] configuration_store.lua:124: store(): added service 2555417768857 configuration with hosts: localhost ttl: 0, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Find Service Policy, i: 2 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Local Policy Chain, i: 3 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache_handler.lua:23: new(): backend cache handler: strict 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: APIcast, i: 1 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] service.lua:227: get_usage(): [mapping] service 2555417761253 has 2 rules 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: rewrite(): policy chain execute phase: rewrite, policy: Metrics, i: 4 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] executor.lua:25: access(): executor phase: access 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Load Configuration, i: 1 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Find Service Policy, i: 2 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Local Policy Chain, i: 3 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: policy chain execute phase: access, policy: APIcast, i: 1 2019/04/25 20:49:22 [info] 29#29: *50 [lua] proxy.lua:81: output_debug_headers(): usage: usage%5Bhits%5D=1 credentials: user_key=, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "apicast.internal:8080" 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] proxy.lua:145: apicast cache hit key: 2555417761253::usage%5Bhits%5D=1 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: access(): policy chain execute phase: access, policy: Metrics, i: 4 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] executor.lua:25: content(): executor phase: content 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Load Configuration, i: 1 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Find Service Policy, i: 2 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: content(): policy chain execute phase: content, policy: Local Policy Chain, i: 3 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] policy_chain.lua:162: policy chain execute phase: content, policy: APIcast, i: 1 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] upstream.lua:181: call(): using proxy: http://tinyproxy.internal:8888 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] resolver.lua:328: lookup(): resolver query: echo-api.3scale.net 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:122: fetch_answers(): resolver cache miss echo-api.3scale.net 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:188: get(): resolver cache miss: echo-api.3scale.net 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] resolver.lua:294: search_dns(): resolver query: echo-api.3scale.net search: query: echo-api.3scale.net. 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:122: fetch_answers(): resolver cache miss echo-api.3scale.net. 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:188: get(): resolver cache miss: echo-api.3scale.net. 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] dns_client.lua:68: query(): resolver query: echo-api.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:74: store(): resolver cache write echo-api.3scale.net with TLL 207 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:74: store(): resolver cache write tf-lb-0081dad066b2afef5328e0256a-2081992367.us-east-1.elb.amazonaws.com with TLL 59 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] resolver.lua:348: lookup(): resolver query: echo-api.3scale.net finished with 3 answers 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] resolver.lua:388: get_servers(): query for echo-api.3scale.net finished with 3 answers 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] resolver.lua:328: lookup(): resolver query: tinyproxy.internal 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:122: fetch_answers(): resolver cache miss tinyproxy.internal 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:188: get(): resolver cache miss: tinyproxy.internal 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] resolver.lua:294: search_dns(): resolver query: tinyproxy.internal search: query: tinyproxy.internal. 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:122: fetch_answers(): resolver cache miss tinyproxy.internal. 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:188: get(): resolver cache miss: tinyproxy.internal. 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] dns_client.lua:68: query(): resolver query: tinyproxy.internal. nameserver: 127.0.0.1:5353 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] cache.lua:74: store(): resolver cache write tinyproxy.internal with TLL 242 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] resolver.lua:348: lookup(): resolver query: tinyproxy.internal finished with 1 answers 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] resolver.lua:388: get_servers(): query for tinyproxy.internal finished with 1 answers 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] http.lua:50: connect(): connected to ip:172.18.0.3 host: tinyproxy.internal port: 8888 ok: 1 err: nil 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] proxy.lua:82: new(): connection to tinyproxy.internal:8888 established, pool: tinyproxy.internal:8888:echo-api.3scale.net:443 reused times: 0 2019/04/25 20:49:22 [debug] 29#29: *50 [lua] http.lua:633: send_request(): CONNECT 18.214.132.12:443 HTTP/1.1 User-Agent: lua-resty-http/0.12 (Lua) ngx_lua/10013 Host: echo-api.3scale.net:443 2019/04/25 20:49:23 [debug] 29#29: *50 [lua] http.lua:633: send_request(): POST /?user_key= HTTP/1.1 User-Agent: curl/7.29.0 Accept: */* Content-Type: application/x-www-form-urlencoded Content-Length: 9 Host: echo-api.3scale.net 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] executor.lua:25: header_filter(): executor phase: header_filter 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Load Configuration, i: 1 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Find Service Policy, i: 2 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Local Policy Chain, i: 3 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: policy chain execute phase: header_filter, policy: APIcast, i: 1 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: header_filter(): policy chain execute phase: header_filter, policy: Metrics, i: 4 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] executor.lua:25: body_filter(): executor phase: body_filter 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Load Configuration, i: 1 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Find Service Policy, i: 2 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Local Policy Chain, i: 3 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: policy chain execute phase: body_filter, policy: APIcast, i: 1 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Metrics, i: 4 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] executor.lua:25: body_filter(): executor phase: body_filter 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Load Configuration, i: 1 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Find Service Policy, i: 2 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Local Policy Chain, i: 3 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: policy chain execute phase: body_filter, policy: APIcast, i: 1 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: body_filter(): policy chain execute phase: body_filter, policy: Metrics, i: 4 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] executor.lua:25: post_action(): executor phase: post_action 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Load Configuration, i: 1 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Find Service Policy, i: 2 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Local Policy Chain, i: 3 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] policy_chain.lua:162: policy chain execute phase: post_action, policy: APIcast, i: 1 2019/04/25 20:49:53 [info] 29#29: *50 [lua] proxy.lua:335: [async] reporting to backend asynchronously, cached_key: 2555417761253::usage%5Bhits%5D=1, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "echo-api.3scale.net" 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] resolver.lua:328: lookup(): resolver query: su1.3scale.net 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] cache.lua:122: fetch_answers(): resolver cache miss su1.3scale.net 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] cache.lua:188: get(): resolver cache miss: su1.3scale.net 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] resolver.lua:294: search_dns(): resolver query: su1.3scale.net search: query: su1.3scale.net. 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] cache.lua:122: fetch_answers(): resolver cache miss su1.3scale.net. 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] cache.lua:188: get(): resolver cache miss: su1.3scale.net. 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] dns_client.lua:68: query(): resolver query: su1.3scale.net. nameserver: 127.0.0.1:5353 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] cache.lua:74: store(): resolver cache write su1.3scale.net with TLL 59 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] resolver.lua:348: lookup(): resolver query: su1.3scale.net finished with 1 answers 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] resolver.lua:388: get_servers(): query for su1.3scale.net finished with 1 answers 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] resolver.lua:328: lookup(): resolver query: tinyproxy.internal 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] cache.lua:115: fetch_answers(): resolver cache read tinyproxy.internal 1 entries 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] cache.lua:190: get(): resolver cache hit: tinyproxy.internal 172.18.0.3 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] resolver.lua:348: lookup(): resolver query: tinyproxy.internal finished with 1 answers 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] resolver.lua:388: get_servers(): query for tinyproxy.internal finished with 1 answers 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] http.lua:50: connect(): connected to ip:172.18.0.3 host: tinyproxy.internal port: 8888 ok: 1 err: nil 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] proxy.lua:82: new(): connection to tinyproxy.internal:8888 established, pool: tinyproxy.internal:8888:su1.3scale.net:443 reused times: 0 2019/04/25 20:49:53 [debug] 29#29: *50 [lua] http.lua:633: send_request(): CONNECT 54.83.63.187:443 HTTP/1.1 User-Agent: lua-resty-http/0.12 (Lua) ngx_lua/10013 Host: su1.3scale.net 2019/04/25 20:49:54 [debug] 29#29: *50 [lua] http.lua:633: send_request(): GET /transactions/authrep.xml?service_token=b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c&service_id=2555417761253&usage%5Bhits%5D=1&user_key= HTTP/1.1 User-Agent: APIcast/3.4.0 (Linux; x64; env:production) APIcast/3.4.0 3scale-Options: rejection_reason_header=1&limit_headers=1&no_body=1 Connection: Keep-Alive Host: su1.3scale.net 2019/04/25 20:49:54 [info] 29#29: *50 [lua] backend_client.lua:139: call_backend_transaction(): backend client uri: https://su1.3scale.net/transactions/authrep.xml?service_token=b545171d73b0f41fb11353516272084f75ef751727703d8587bf89cefc45b02c&service_id=2555417761253&usage%5Bhits%5D=1&user_key= ok: true status: 200 body: error: nil, client: 172.18.0.5, server: _, request: "POST /?user_key= HTTP/1.1", host: "echo-api.3scale.net" 2019/04/25 20:49:54 [debug] 29#29: *50 [lua] proxy.lua:368: handle_backend_response(): [backend] response status: 200 body: 2019/04/25 20:49:54 [debug] 29#29: *50 [lua] policy_chain.lua:162: post_action(): policy chain execute phase: post_action, policy: Metrics, i: 4 2019/04/25 20:49:54 [debug] 29#29: *50 [lua] executor.lua:25: log(): executor phase: log 2019/04/25 20:49:54 [debug] 29#29: *50 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Load Configuration, i: 1 2019/04/25 20:49:54 [debug] 29#29: *50 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Find Service Policy, i: 2 2019/04/25 20:49:54 [debug] 29#29: *50 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Local Policy Chain, i: 3 2019/04/25 20:49:54 [debug] 29#29: *50 [lua] policy_chain.lua:162: policy chain execute phase: log, policy: APIcast, i: 1 2019/04/25 20:49:54 [debug] 29#29: *50 [lua] policy_chain.lua:162: log(): policy chain execute phase: log, policy: Metrics, i: 4 2019/04/25 20:49:54 [info] 29#29: *50 client 172.18.0.5 closed keepalive connection [25/Apr/2019:20:49:54 +0000] apicast.internal:8080 172.18.0.5:38400 "POST /?user_key= HTTP/1.1" 504 103 (37.100) 1.1 CONNECT Apr 25 20:49:22 [19]: Connect (file descriptor 9): apicast.internal [172.18.0.2] CONNECT Apr 25 20:49:22 [19]: Request (file descriptor 9): CONNECT 18.214.132.12:443 HTTP/1.1 INFO Apr 25 20:49:22 [19]: No upstream proxy for 18.214.132.12 CONNECT Apr 25 20:49:22 [19]: Established connection to host "18.214.132.12" using file descriptor 10. INFO Apr 25 20:49:22 [19]: Not sending client headers to remote machine INFO Apr 25 20:49:53 [19]: Closed connection between local client (fd:9) and remote client (fd:10) CONNECT Apr 25 20:49:53 [10]: Connect (file descriptor 9): apicast.internal [172.18.0.2] CONNECT Apr 25 20:49:53 [10]: Request (file descriptor 9): CONNECT 54.83.63.187:443 HTTP/1.1 INFO Apr 25 20:49:53 [10]: No upstream proxy for 54.83.63.187 CONNECT Apr 25 20:49:53 [10]: Established connection to host "54.83.63.187" using file descriptor 10. INFO Apr 25 20:49:54 [10]: Not sending client headers to remote machine INFO Apr 25 20:49:54 [10]: Closed connection between local client (fd:9) and remote client (fd:10) ERROR Apr 25 20:50:05 [16]: Error reading readble client_fd 9 WARNING Apr 25 20:50:05 [16]: Could not retrieve request entity