Critical When apicast and API (backend that you are exposing) are supporting HTTP/2 protocol. You can enable the grpc policy to allow full traffic HTTP/2. So the requests between each entity user '<---> apicast <---> API should be HTTP/2. This work as expected when 3scale product has only 1 backend.
The issue is when the product has more than 1 backend. Then the API is receiving HTTP/1.1 requests from apicast instead of HTTP/2. Thus grpc policy doesn't work on 3scale products with more than 1 backend.
In other words, the grpc policy is not working on products with multiple backends
Notes for devs
When a product has more than 1 backend. The request with HTTP/2 full traffic is logged as HTTP/2 on the apicast side. But the backend is returning that HTTP/1.1 was used.
It seems to be that the issue is the routing policy is higher than the grpc one. The APIAP is adding the routing policy on top of the policy chain.
logs of request with a single backend (see the log_request_with_1_backend.log file)
logs of request with 2 backends (see the log_request_with_2_backends.logs file)
apicast config for service with 1 backend (apicast_config_1_backend.json)
apicast config for service with 2 backends(apicast_config_2_backends.json)
The main difference what I found is:
1 backend
2020/03/09 12:50:47 [info] 27#27: *8 proxy.lua:328: [async] skipping after action, no cached key, requestID=2f3a99276b9724e1587d43c78063ae85 while sending to client, client: 10.130.0.1, server: _, request: "GET /info?user_key=229bc45f832d78d559bad48a16207f82 HTTP/2.0", upstream: "grpcs://172.30.108.202:8080", host: "go-httpbin.httpbin.svc.cluster.local:8080"
2 backends
2020/03/09 12:48:50 [info] 27#27: *31 proxy.lua:328: [async] skipping after action, no cached key, requestID=320fc9e2ef87f81d6505127a38b29455 while sending to client, client: 10.130.0.1, server: _, request: "GET /info?user_key=229bc45f832d78d559bad48a16207f82 HTTP/2.0", upstream: "https://172.30.108.202:8080/info?user_key=229bc45f832d78d559bad48a16207f82 ", host: "go-httpbin.httpbin.svc.cluster.local:8080"
Responses from curls:
1 backend
{"method":"GET","URL":"http://go-httpbin.httpbin.svc.cluster.local:8080/info?user_key=229bc45f832d78d559bad48a16207f82","proto":"HTTP/2.0","proto-major":2,"proto-minor":0,"headers":{"Accept":["*/*"],"Host":["go-httpbin.httpbin.svc.cluster.local:8080"],"User-Agent":["curl/7.68.0"],"X-3scale-Grpc-Secret-Token":["Shared_secret_sent_from_proxy_to_API_backend_e58d450232eda0b0"],"X-Real-Ip":["10.130.0.1"]},"transfer-encoding":null,"host":"go-httpbin.httpbin.svc.cluster.local:8080","args":{"user_key":["229bc45f832d78d559bad48a16207f82"]},"remote-address":"10.130.2.1:34456","request-uri":"/info?user_key=229bc45f832d78d559bad48a16207f82","tls":{"Version":771,"HandshakeComplete":true,"DidResume":false,"CipherSuite":49199,"NegotiatedProtocol":"h2","NegotiatedProtocolIsMutual":true,"ServerName":"","PeerCertificates":null,"VerifiedChains":null,"SignedCertificateTimestamps":null,"OCSPResponse":null,"TLSUnique":"Sb0qiWPc4WBOb5LZ"}}%
2 backends
{"method":"GET","URL":"http://go-httpbin.httpbin.svc.cluster.local:8080/info?user_key=229bc45f832d78d559bad48a16207f82","proto":"HTTP/1.1","proto-major":1,"proto-minor":1,"headers":{"Accept":["*/*"],"Host":["go-httpbin.httpbin.svc.cluster.local:8080"],"User-Agent":["curl/7.68.0"],"X-3scale-Proxy-Secret-Token":["Shared_secret_sent_from_proxy_to_API_backend_e58d450232eda0b0"],"X-Real-Ip":["10.130.0.1"]},"transfer-encoding":null,"host":"go-httpbin.httpbin.svc.cluster.local:8080","args":{"user_key":["229bc45f832d78d559bad48a16207f82"]},"remote-address":"10.129.0.1:56006","request-uri":"/info?user_key=229bc45f832d78d559bad48a16207f82","tls":{"Version":771,"HandshakeComplete":true,"DidResume":false,"CipherSuite":49199,"NegotiatedProtocol":"","NegotiatedProtocolIsMutual":true,"ServerName":"go-httpbin.httpbin.svc.cluster.local","PeerCertificates":null,"VerifiedChains":null,"SignedCertificateTimestamps":null,"OCSPResponse":null,"TLSUnique":"XNoowC5PX9beHwhQ"}}%
