2023/09/21 14:10:59 [info] 25#25: *18 [lua] remote_v2.lua:195: call(): could not get configuration for service 2: invalid status: 404 (Not Found), client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:11:40 [info] 25#25: *28 [lua] remote_v2.lua:195: call(): could not get configuration for service 2: invalid status: 404 (Not Found), client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:11:59 [error] 25#25: *18 lua tcp socket connect timed out, when connecting to 93.184.216.34:81, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:11:59 [info] 25#25: *18 [lua] discovery.lua:113: openid_configuration(): failed to get OIDC Provider from http://example.com:81/.well-known/openid-configuration status: 0 body: nil, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:40 [error] 25#25: *28 lua tcp socket connect timed out, when connecting to 93.184.216.34:81, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:40 [info] 25#25: *28 [lua] discovery.lua:113: openid_configuration(): failed to get OIDC Provider from http://example.com:81/.well-known/openid-configuration status: 0 body: nil, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [error] 25#25: *18 lua tcp socket connect timed out, when connecting to 93.184.216.34:81, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [info] 25#25: *18 [lua] discovery.lua:113: openid_configuration(): failed to get OIDC Provider from http://example.com:81/.well-known/openid-configuration status: 0 body: nil, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [info] 25#25: *18 [lua] configuration_store.lua:132: store(): added service 8 configuration with hosts: mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com ttl: 0, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [info] 25#25: *18 [lua] configuration_store.lua:132: store(): added service 9 configuration with hosts: 2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com ttl: 0, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [info] 25#25: *18 [lua] proxy.lua:82: output_debug_headers(): usage: usage%5Bhits%5D=1 credentials: user_key=cb06e0750a28b098f0f4dd892bee7a11, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [info] 25#25: *18 [lua] proxy.lua:136: apicast cache miss key: 8:cb06e0750a28b098f0f4dd892bee7a11:usage%5Bhits%5D=1 value: nil, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [info] 25#25: *18 [lua] backend_client.lua:133: call_backend_transaction(): backend client uri: https://backend-3scale.mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com/transactions/authrep.xml?service_token=05bbe2a3c71ac6a28d2c4414d0cac929fd01917da143ded224a3a91bf50498fb&service_id=8&usage%5Bhits%5D=1&user_key=cb06e0750a28b098f0f4dd892bee7a11 ok: true status: 200 body: error: nil, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [info] 25#25: *18 [lua] cache_handler.lua:43: cache_handler(): apicast cache write key: 8:cb06e0750a28b098f0f4dd892bee7a11:usage%5Bhits%5D=1, ttl: nil, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [info] 25#25: *18 [lua] balancer.lua:108: set_current_peer(): balancer set peer 10.8.40.55:443 ok: true err: nil while connecting to upstream, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mockserver-tools.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [info] 25#25: *18 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too (104: Connection reset by peer) while connecting to upstream, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", upstream: "https://10.8.40.55:443/", host: "mockserver-tools.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:12:59 [info] 25#25: *18 [lua] proxy.lua:335: [async] skipping after action, no cached key while sending to client, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", upstream: "https://10.8.40.55:443/", host: "mockserver-tools.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" [21/Sep/2023:14:12:59 +0000] mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com:8080 192.168.134.1:46808 "GET / HTTP/1.1" 200 0 (120.463) 0 2023/09/21 14:13:07 [warn] 25#25: *45 [lua] synchronization.lua:70: run(): failed to acquire lock on key: 2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com error: timeout, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:13:07 [warn] 25#25: *45 [lua] configuration_loader.lua:257: rewrite(): failed to load config for host: 2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com error: nil, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:13:07 [warn] 25#25: *45 [lua] host_based_finder.lua:20: find_service(): service not found for host 2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:13:07 [warn] 25#25: *45 [lua] errors.lua:52: get_upstream(): could not find service for host: 2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" [21/Sep/2023:14:13:07 +0000] 2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com:8080 192.168.134.1:43164 "GET / HTTP/1.1" 404 5 (15.011) 0 2023/09/21 14:13:12 [info] 25#25: *50 [lua] remote_v2.lua:195: call(): could not get configuration for service 2: invalid status: 404 (Not Found), client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:13:23 [info] 25#25: *45 client 192.168.134.1 closed keepalive connection 2023/09/21 14:13:40 [error] 25#25: *28 lua tcp socket connect timed out, when connecting to 93.184.216.34:81, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:13:40 [info] 25#25: *28 [lua] discovery.lua:113: openid_configuration(): failed to get OIDC Provider from http://example.com:81/.well-known/openid-configuration status: 0 body: nil, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:13:40 [info] 25#25: *28 [lua] configuration_store.lua:132: store(): added service 8 configuration with hosts: mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com ttl: 0, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:13:40 [info] 25#25: *28 [lua] configuration_store.lua:132: store(): added service 9 configuration with hosts: 2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com ttl: 0, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:13:40 [warn] 25#25: *28 [lua] proxy.lua:203: handle_oauth(): failed to initialize OpenID Connect for service 9: missing OIDC configuration, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:13:40 [info] 25#25: *28 [lua] errors.lua:8: rewrite(): no credentials provided for service 9, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:13:40 [info] 25#25: *28 writev() failed (104: Connection reset by peer), client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" [21/Sep/2023:14:13:40 +0000] 2-mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com:8080 192.168.134.1:36556 "GET / HTTP/1.1" 403 0 (120.442) 0 2023/09/21 14:14:12 [error] 25#25: *50 lua tcp socket connect timed out, when connecting to 93.184.216.34:81, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:14:12 [info] 25#25: *50 [lua] discovery.lua:113: openid_configuration(): failed to get OIDC Provider from http://example.com:81/.well-known/openid-configuration status: 0 body: nil, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:14:17 [warn] 25#25: *62 [lua] synchronization.lua:70: run(): failed to acquire lock on key: mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com error: timeout, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:14:17 [warn] 25#25: *62 [lua] configuration_loader.lua:257: rewrite(): failed to load config for host: mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com error: nil, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:14:17 [warn] 25#25: *62 [lua] host_based_finder.lua:20: find_service(): service not found for host mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" 2023/09/21 14:14:17 [warn] 25#25: *62 [lua] errors.lua:52: get_upstream(): could not find service for host: mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com, client: 192.168.134.1, server: _, request: "GET / HTTP/1.1", host: "mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com" [21/Sep/2023:14:14:17 +0000] mdujava-apicast-mdujava.apps.ocp-cluster.osp.api-qe.eng.rdu2.redhat.com:8080 192.168.134.1:42302 "GET / HTTP/1.1" 404 5 (15.005) 0