Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-74347

Show 3x inflation network traffic in the web-console when sustain 1000 MB/sec NFS I/O in the pod

    • Icon: Bug Bug
    • Resolution: Unresolved
    • Icon: Normal Normal
    • 4.22.0
    • 4.18.z
    • Monitoring
    • None
    • None
    • False
    • Hide

      None

      Show
      None
    • None
    • Important
    • None
    • None
    • None
    • MON Sprint 283
    • 1
    • None
    • None
    • None
    • None
    • None
    • None
    • None

      Description of problem:

      Show 3x inflation network traffic on the web-console when sustain 1000 MB/sec NFS I/O in the pod.

      Version-Release number of selected component (if applicable):

          

      How reproducible:

      1 OCP node using 2 different IP range, 192.168.0.192/26 is provided for OCP manager, 172.168.0.192/26 is provider for NFS storage and the network interface have configured bond0. 
      
      2 Create PVC and use NFS as backend storage, Then reated pod and mount NFS to /data directory.
      # cat debug.yaml
      
      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: debug-pvc
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 200Gi  
        storageClassName: "nfs-client" 
      ---
      apiVersion: v1
      kind: Pod
      metadata:
        name: troubleshooting-pvc-pod
      spec:
        nodeSelector:
          kubernetes.io/hostname: ocp-tnkfb-worker-0
        securityContext:
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        containers:
        - name: tool-container
          image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:651732e89586140ee9fd7ca8772e0adbf9606fff461759235dbbd42af5636eae
          imagePullPolicy: IfNotPresent
          command: [ "sleep", "3600" ]
          securityContext:
            allowPrivilegeEscalation: false
          volumeMounts:
          - name: storage-volume
            mountPath: /data  
        volumes:
        - name: storage-volume
          persistentVolumeClaim:
            claimName: debug-pvc
      
      3 Run command in the /data directory on the pod
      dd if=/dev/zero of=test_direct.txt oflag=direct bs=1M count=100000 status=progress 2>&1 | tr '\r' '\n'
      
      4 Check the network traffic on the web console.
      
      https://files.slack.com/files-tmb/T030G10V24F-F0ABZNL0ATA-f814467d76/image_720.png
      
      

      Steps to Reproduce:

          1. Run the command in the NFS dir on the pods.
      dd if=/dev/zero bs=100M | pv -ftrabW -L1000M 2> progress.txt | dd of=sandbag.txt bs=100M

       

      Actual results:

      Show 3x inflation network traffic on the web-console when sustain 1000 MB/sec  NFS I/O

       

      Expected results:

      Show normal network traffic on the web-console when sustain 1000 MB/sec  NFS I/O

      Additional info:

      Please check the docs. https://docs.google.com/document/d/1sJVozHhjXgNIf4a4FToDJu-3sA0rUDBIKp2GxwJluzo/edit?tab=t.0

      The remote meeting record video:  https://drive.google.com/file/d/1C4J3KLZC86RTyootKpo7iS4-pgoQ_rNh/view

      Prometheus data from Jan 23: https://attachments.access.redhat.com/hydra/rest/cases/04328229/attachments/aee910da-00c8-4818-b40c-f8e184f4895b?usePresignedUrl=true

       

              jfajersk@redhat.com Jan Fajerski
              rhn-support-danliu Qihuan Liu
              None
              None
              YaDan Pei YaDan Pei
              None
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated: