Uploaded image for project: 'RHEL'
  1. RHEL
  2. RHEL-15148

kernel crash when one process reads from /proc/fs/nfsd/clients at the same time as nfsd processes an unmount

    • kernel-4.18.0-553.el8_10
    • Normal
    • sst_filesystems
    • ssg_filesystems_storage_and_HA
    • 9
    • 12
    • 2
    • QE ack
    • False
    • Hide

      None

      Show
      None
    • None
    • Red Hat Enterprise Linux
    • None
    • None

      What were you trying to do that didn't work?

      kernel panic when one process reads from procfiles under /proc/fs/nfsd/clients at the same time as an nfsd completes an unmount for a client.

      [  147.468851] WARNING: CPU: 5 PID: 1921 at fs/nfsd/nfsctl.c:1261 nfsdfs_remove_files+0xcd/0xf0 [nfsd]
      [  147.468942] Kernel panic - not syncing: panic_on_warn set ...[  147.469363] CPU: 5 PID: 1921 Comm: nfsd Kdump: loaded Not tainted 4.18.0-477.15.1.el8_8.x86_64 #1
      [  147.469581] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc37 04/01/2014
      [  147.469738] Call Trace:
      [  147.469862]  dump_stack+0x41/0x60
      [  147.470043]  panic+0xe7/0x2ac
      [  147.470250]  ? nfsdfs_remove_files+0xcd/0xf0 [nfsd]
      [  147.470525]  __warn.cold.14+0x31/0x38
      [  147.470702]  ? nfsdfs_remove_files+0xcd/0xf0 [nfsd]
      [  147.470985]  ? nfsdfs_remove_files+0xcd/0xf0 [nfsd]
      [  147.471333]  report_bug+0xb1/0xe0
      [  147.471508]  ? internal_add_timer+0x42/0x70
      [  147.471713]  do_error_trap+0x9e/0xd0
      [  147.471907]  do_invalid_op+0x36/0x40
      [  147.472092]  ? nfsdfs_remove_files+0xcd/0xf0 [nfsd]
      [  147.472432]  invalid_op+0x14/0x20
      [  147.472632] RIP: 0010:nfsdfs_remove_files+0xcd/0xf0 [nfsd]
      [  147.472967] Code: 48 8b 95 90 00 00 00 48 8d 85 90 00 00 00 48 89 eb 48 8d aa 70 ff ff ff 4c 39 e0 74 11 48 8b 43 30 48 85 c0 0f 85 70 ff ff ff <0f> 0b eb d2 5b 5d 41 5c 41 5d 41 5e e9 42 24 dd d6 0f 0b eb c1 66
      [  147.473525] RSP: 0018:ffffba28c192fcd0 EFLAGS: 00010246
      [  147.473729] RAX: 0000000000000000 RBX: ffff8f9bd0007c00 RCX: ffff8f9b406be380
      [  147.473982] RDX: ffff8f9b5383b160 RSI: ffff8f9b406be380 RDI: ffff8f9b5387e058
      [  147.474258] RBP: ffff8f9b5383b0d0 R08: ffff8f9b02414c88 R09: ffffffff98687708
      [  147.474497] R10: ffff8f9b40262758 R11: 0000000000000000 R12: ffff8f9b5383b160
      [  147.474728] R13: ffff8f9b5383b0c0 R14: 0000000000000000 R15: ffff8f9b53fbe5a0
      [  147.475019]  nfsd_client_rmdir+0x31/0xc0 [nfsd]
      [  147.475350]  free_client+0x9d/0x100 [nfsd]
      [  147.475630]  __destroy_client+0x1df/0x230 [nfsd]
      [  147.475829]  nfsd4_setclientid_confirm+0x1f4/0x410 [nfsd]
      [  147.476083]  nfsd4_proc_compound+0x30e/0x670 [nfsd]
      [  147.476420]  nfsd_dispatch+0x15e/0x290 [nfsd]
      [  147.476647]  svc_process_common+0x360/0x5c0 [sunrpc]
      [  147.477223]  ? svc_sock_secure_port+0x12/0x40 [sunrpc]
      [  147.477498]  ? nfsd_svc+0x2f0/0x2f0 [nfsd]
      [  147.477697]  ? nfsd_shutdown_threads+0x80/0x80 [nfsd]
      [  147.478003]  svc_process+0xb7/0xf0 [sunrpc]
      [  147.478261]  nfsd+0xe3/0x140 [nfsd]
      [  147.478456]  kthread+0x134/0x150
      [  147.478637]  ? set_kthread_struct+0x50/0x50
      [  147.478831]  ret_from_fork+0x35/0x40

       

      Please provide the package NVR for which bug is seen:

      kernel-4.18.0-477.15.1.el8_8.x86_64

      How reproducible:

      easy

      Steps to reproduce

      on nfs server:

          while [[ 42 ]] ; do find /proc/fs/nfsd/clients/ -name info -exec cat {} \; >/dev/null ; echo -n . ; done

      on nfs client:

        while [[ 42 ]] ; do umount /mnt/tmp ; mount server:/exports /mnt/tmp -overs=4.0,sec=sys ; echo -n . ; done

      Expected results

      No WARNING/crash

      Actual results

      kernel WARNING

       

      Also, observed the following panic involving rpc.mountd reading from /proc/fs/nfsd/clients/67601/info

      PID: 93776    TASK: ffff9b3712074000  CPU: 14   COMMAND: "rpc.mountd"
          [exception RIP: __slab_free+0x19e]
       #7 [ffffb6a2a2597de8] kfree at ffffffffb0f2623e
       #8 [ffffb6a2a2597e28] __free_client at ffffffffc0893f99 [nfsd]
       #9 [ffffb6a2a2597e40] client_info_show at ffffffffc0894175 [nfsd]
      #10 [ffffb6a2a2597e68] seq_read at ffffffffb0f8f453
      #11 [ffffb6a2a2597ec8] vfs_read at ffffffffb0f64921
      #12 [ffffb6a2a2597f00] ksys_read at ffffffffb0f64d8f
      #13 [ffffb6a2a2597f38] do_syscall_64 at ffffffffb0c052fb

      which appears to be the same issue, only reversed

            jlayton@redhat.com Jeffrey Layton
            rhn-support-fsorenso Frank Sorenson
            NFS Team NFS Team
            Yongcheng Yang Yongcheng Yang
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: