Description of problem:

The message "http2: server: error reading preface from client" is flooding the lokistack-gateway logs.

Version-Release number of selected component (if applicable):

RHOL 5.7.3

Loki

How reproducible:

Always

Steps to Reproduce:

1.  Configure http2 cluster-wide in the following the OCP documentation [1] (or use a cluster with it enabled like OSD/ROSA).
2.  Check the lokistack-gateway logs

   $ oc get logs -n openshift-logging -c gateway [lokistack-gateway-pod_name]
   

Actual results:

The following message is flooding the log of the pod, as it's shown one time per router each 5 seconds.

http2: server: error reading preface from client 10.x.y.z:44444: read tcp 10.v.z.n:8443->10.x.y.z:44444: read: connection reset by peer

Expected results:

The above message shouldn't be shown in the log.

Additional info:

The HAProxy health check is running each 5 seconds per router pod, and it's causing that connection. Looks like a debug configuration in a Go application could cause those messages as explained in [2] (could be other parameter, and not only the "GODEBUG=http2debug=1" mentioned there.

[1] https://docs.openshift.com/container-platform/4.12/networking/ingress-operator.html#nw-http2-haproxy_configuring-ingress
[2] https://access.redhat.com/solutions/6999058