Packet loss on Windows 2019 nodes

Workloads on Windows Server 2019, are having packet loss. When a pod health check is trying to reach a Windows webserver pod IP, the destination IP of reply packets are getting changed to some other IP(pod IP) causing the packet to never reach the host layer.

SourceVIP on the HNS Load Balancer is being set to the host IP. This should not be the case.
    

OCP 4.12
    

Occurs only on Windows 2019 after several hours of runtime.
    

    1. Add a Windows 2019 node to OCP cluster
    2. Deploy 5-10 Windows pods behind a cluster IP

kind: Deployment
apiVersion: apps/v1
metadata:
  name: win-webserverlog-2019
  labels:
    app: win-webserver-log-2019
spec:
  replicas: 5
  selector:
    matchLabels:
      app: win-webserver-log-2019
  template:
    metadata:
      name: win-webserverlog-2019
      labels:
        app: win-webserver-log-2019
    spec:
      nodeSelector:
        kubernetes.io/os: windows
      restartPolicy: Always
      runtimeClassName: windows-2019
      containers:
        - resources: {}
          readinessProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            initialDelaySeconds: 20
            timeoutSeconds: 2
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 10
          terminationMessagePath: /dev/termination-log
          name: windowswebserverlog
          livenessProbe:
            httpGet:
              path: /
              port: 80
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 2
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 10
          imagePullPolicy: IfNotPresent
          volumeMounts:
            - name: foo
              mountPath: 'C:\Temp\pod'
          terminationMessagePolicy: File
          image: 'mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019'
      volumes:
        - name: foo
          hostPath:
            path: 'C:\Temp\pod'
            type: ''
      dnsPolicy: ClusterFirst
      tolerations:
        - key: os
          value: Windows
---
kind: Service
apiVersion: v1
metadata:
  name: windows-service-2019
spec:
  ipFamilies:
    - IPv4
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80
  internalTrafficPolicy: Cluster
  type: ClusterIP
  ipFamilyPolicy: SingleStack
  sessionAffinity: None
  selector:
    app: win-webserver-log-2019
---
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: windows-2022
handler: 'runhcs-wcow-process'
scheduling:
  nodeSelector:
    kubernetes.io/os: 'windows'
    kubernetes.io/arch: 'amd64'
    node.kubernetes.io/windows-build: '10.0.20348'
  tolerations:
  - effect: NoSchedule
    key: os
    operator: Equal
    value: "windows"

    3. With no changes, pods will restart on their own due within 1-2 hours due to health probe timeouts
    4. More the number of pods, more the changes of pods restarting quickly
    

All the Pod restarts or a good number of pods restarts on a node due to Kube health check timeout

Healthcheck packets from Kubelet should not be dropped and result in UN-necessary pod restarts