Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-12358

PVC mount failed with "system call failed: Structure needs cleaning" when starting linuxptp-daemon pod

    XMLWordPrintable

Details

    • Important
    • No
    • CNF RAN Sprint 235, CNF RAN Sprint 236, CNF RAN Sprint 237, CNF RAN Sprint 238, CNF RAN Sprint 239
    • 5
    • False
    • Hide

      None

      Show
      None
    • Hide
      When deployed through ptp-operator and ZTP pipeline using HTTP transport, the linuxptp-daemon pod intermittently fails to start due to PVC mount error "system call failed: Structure needs cleaning". The workaround is to delete the cloud-event-proxy-store-storage-class-http-events PVC and re-deploy.
      Show
      When deployed through ptp-operator and ZTP pipeline using HTTP transport, the linuxptp-daemon pod intermittently fails to start due to PVC mount error "system call failed: Structure needs cleaning". The workaround is to delete the cloud-event-proxy-store-storage-class-http-events PVC and re-deploy.
    • Hide
      7/19: this will be closed once config map is added to 4.13
      4/26: telco review pending (JD/KY)
      Rel Note for Telco: Yes (4.13) - Jack will write the text.
      Show
      7/19: this will be closed once config map is added to 4.13 4/26: telco review pending (JD/KY) Rel Note for Telco: Yes (4.13) - Jack will write the text.

    Description

      Description of problem:

      linuxptp-daemon pod does not start with the following volume mount error out of fresh install and config:
      
      
        Warning  FailedMount  17m                  kubelet  Unable to attach or mount volumes: unmounted volumes=[pubsubstore], unattached volumes=[config-volume pubsubstore event-bus-socket socket-dir kube-api-access-kltbp linuxptp-certs]: timed out waiting for the condition
        Warning  FailedMount  15m (x3 over 24m)    kubelet  Unable to attach or mount volumes: unmounted volumes=[pubsubstore], unattached volumes=[socket-dir kube-api-access-kltbp linuxptp-certs config-volume pubsubstore event-bus-socket]: timed out waiting for the condition
        Warning  FailedMount  4m8s                 kubelet  Unable to attach or mount volumes: unmounted volumes=[pubsubstore], unattached volumes=[linuxptp-certs config-volume pubsubstore event-bus-socket socket-dir kube-api-access-kltbp]: timed out waiting for the condition
        Warning  FailedMount  4m2s (x16 over 24m)  kubelet  (combined from similar events): MountVolume.MountDevice failed for volume "local-pv-bc42d358" : local: failed to mount device /mnt/local-storage/storage-class-http-events/scsi-36f4ee08039aa91002a97d2f8da695437-part5 at /var/lib/kubelet/plugins/kubernetes.io/local-volume/mounts/local-pv-bc42d358 (fstype: xfs), error mount failed: exit status 32
      Mounting command: systemd-run
      Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/plugins/kubernetes.io/local-volume/mounts/local-pv-bc42d358 --scope -- mount -t xfs -o defaults /mnt/local-storage/storage-class-http-events/scsi-36f4ee08039aa91002a97d2f8da695437-part5 /var/lib/kubelet/plugins/kubernetes.io/local-volume/mounts/local-pv-bc42d358
      Output: Running scope as unit: run-r63873a5c66584274bed05ed7edfc8001.scope
      mount: /var/lib/kubelet/plugins/kubernetes.io/local-volume/mounts/local-pv-bc42d358: mount(2) system call failed: Structure needs cleaning.
      

      Version-Release number of selected component (if applicable):

      4.13

      How reproducible:

      Seems to be producible - happened 2 out of 2 times in recent 4.13 ptp CIs. 

      Steps to Reproduce:

      1. Install SNO DU via ZTP with http events configured for both ptp and bmer
      2. 
      3.
      

      Actual results:

      Expected results:

      bmer pod started properly with PVC mounted
      ptp daemon pod failed to start with PVC mount error

      Additional info:

      Restarting ptplinux-daemon pod did not resolve this issue

       

      Attachments

        Activity

          People

            jacding@redhat.com Jack Ding
            rhn-support-yliu1 Yang Liu
            Tao Liu Tao Liu
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: