Loading...

Details

Type: Bug
Resolution: Done
Priority: Critical
Fix Version/s: None
Affects Version/s: 4.13.0
Component/s: Storage
Labels:
- trt

Severity:
Critical
Regression:
Yes
Sprint:
Storage Sprint 232
sprint_count:
1
Release Blocker:
Approved
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Target Version:

4.13.0

SFDC Cases Counter:
SFDC Cases Links:

Description

With the RHCOS 9.2 testing being done before merging in PR https://github.com/openshift/machine-config-operator/pull/3485 we seem to be seeing between 1-3 Azure workers dying during the post-upgrade conformance test. This problem is prevalent in seemingly all sub-jobs of https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/aggregator-periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-azure-sdn-upgrade/1625681132605411328 but we'll focus on just one example for this bug:

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/openshift-machine-config-operator-3485-ci-4.13-upgrade-from-stable-4.12-e2e-azure-sdn-upgrade/1625681121771524096

In this run if you view gather-extra nodes.json, we see two nodes had kubelet stop reporting status late in the run, worker w2776 stopped reporting at 5:38:29, and worker 7rftw stopped reporting at 5:18:48. These are last transition times so possible the actual time of death was a timeout period before. (possibly 30s, more details on this in a minute)

The death of the nodes is preventing a number of artifacts from being gathered, TRT will fix this ASAP in ~~TRT-855~~, but this is why we don't see spyglass charts and other artifacts for the conformance suite run, only the upgrade suite which runs prior.

However I was able to identify system journal logs for the affected node in loki:

https://grafana-loki.ci.openshift.org/explore?orgId=1&left=%7B%22datasource%22:%22Grafana%20Cloud%22,%22queries%22:%5B%7B%22expr%22:%22%7Binvoker%3D%5C%22openshift-internal-ci%2Fopenshift-machine-config-operator-3485-ci-4.13-upgrade-from-stable-4.12-e2e-azure-sdn-upgrade%2F1625681121771524096%5C%22%7D%20%7C%20unpack%20%7C%20host%3D%5C%22ci-op-v9k3j1h2-696e9-c68h4-worker-centralus3-7rftw%5C%22%5Cn%22,%22refId%22:%22A%22%7D%5D,%22range%22:%7B%22from%22:%22now-24h%22,%22to%22:%22now%22%7D%7D

This shows the last reported logs we got for the 7rftw node:

I0215 05:18:14.339779       1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/GetPluginInfo","Request":{},"Response":{"name":"csi-mock-e2e-csi-mock-volumes-6973","vendor_version":"v1.7.3"},"Error":"","FullError":null}
I0215 05:18:13.925254       1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0215 05:18:14.339737       1 server.go:114] GRPC response: {"name":"csi-mock-e2e-csi-mock-volumes-6973","vendor_version":"v1.7.3"}
I0215 05:18:13.925036       1 controller.go:1279] provision "e2e-csi-mock-volumes-6973/pvc-26t2d" class "e2e-csi-mock-volumes-6973-sccszgp": started
I0215 05:18:11.809725       1 controller.go:859] Started provisioner controller csi-mock-e2e-csi-mock-volumes-6973_csi-mockplugin-0_1e0cbb41-bbbc-40c7-8188-23935d0bb52b!
I0215 05:18:14.339730       1 identityserver.go:28] Using default GetPluginInfo
I0215 05:18:11.809649       1 shared_informer.go:270] caches populated
I0215 05:18:14.339684       1 server.go:105] GRPC request: {}
I0215 05:18:11.709270       1 reflector.go:255] Listing and watching *v1.PersistentVolume from sigs.k8s.io/sig-storage-lib-external-provisioner/v7/controller/controller.go:844
I0215 05:18:14.339663       1 server.go:101] GRPC call: /csi.v1.Identity/GetPluginInfo
I0215 05:18:11.709125       1 reflector.go:219] Starting reflector *v1.PersistentVolume (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v7/controller/controller.go:844
I0215 05:18:14.338746       1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{},"Error":"","FullError":null}
I0215 05:18:11.709166       1 reflector.go:255] Listing and watching *v1.StorageClass from sigs.k8s.io/sig-storage-lib-external-provisioner/v7/controller/controller.go:847
I0215 05:18:14.338726       1 server.go:114] GRPC response: {}
I0215 05:18:11.709125       1 reflector.go:219] Starting reflector *v1.StorageClass (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v7/controller/controller.go:847
I0215 05:18:14.338668       1 server.go:105] GRPC request: {}
I0215 05:18:11.708919       1 volume_store.go:97] Starting save volume queue
I0215 05:18:11.708909       1 clone_controller.go:82] Started CloningProtection controller
I0215 05:18:14.338617       1 server.go:101] GRPC call: /csi.v1.Identity/Probe
I0215 05:18:11.708813       1 clone_controller.go:66] Starting CloningProtection controller

30 seconds later roughly, we've transitioned the node kubelet Ready condition to Unknown. Feels like a 30s timeout before we give up on the node response.

Worker ci-op-v9k3j1h2-696e9-c68h4-worker-centralus2-w2776 which died at 5:38:29 has different logs, but still in the storage area:

2023-02-15 01:37:43	
I0215 05:37:42.812659       1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{},"Error":"","FullError":null}
I0215 05:37:42.812631       1 server.go:114] GRPC response: {}
I0215 05:37:42.812548       1 server.go:105] GRPC request: {}
I0215 05:37:42.812510       1 server.go:101] GRPC call: /csi.v1.Identity/Probe
I0215 05:37:42.812659       1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{},"Error":"","FullError":null}
I0215 05:37:42.812631       1 server.go:114] GRPC response: {}
I0215 05:37:42.812548       1 server.go:105] GRPC request: {}
I0215 05:37:42.812510       1 server.go:101] GRPC call: /csi.v1.Identity/Probe
I0215 05:37:42.812659       1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{},"Error":"","FullError":null}
I0215 05:37:42.812631       1 server.go:114] GRPC response: {}
I0215 05:37:42.812548       1 server.go:105] GRPC request: {}
I0215 05:37:42.812510       1 server.go:101] GRPC call: /csi.v1.Identity/Probe
I0215 05:37:42.812659       1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{},"Error":"","FullError":null}
I0215 05:37:42.812631       1 server.go:114] GRPC response: {}
I0215 05:37:42.812548       1 server.go:105] GRPC request: {}
I0215 05:37:42.812510       1 server.go:101] GRPC call: /csi.v1.Identity/Probe
I0215 05:37:42.117439       1 controller.go:151] Successfully synced 'e2e-volume-8456/hostpath-injector'
I0215 05:37:42.117379       1 controller.go:192] Received pod 'hostpath-injector'

Kernel panic logs found for both dead nodes:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/openshift-machine-config-operator-3485-ci-4.13-upgrade-from-stable-4.12-e2e-azure-sdn-upgrade/1625681121771524096/artifacts/e2e-azure-sdn-upgrade/gather-azure-cli/artifacts/ci-op-v9k3j1h2-696e9-c68h4-worker-centralus2-w2776-boot.log

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/openshift-machine-config-operator-3485-ci-4.13-upgrade-from-stable-4.12-e2e-azure-sdn-upgrade/1625681121771524096/artifacts/e2e-azure-sdn-upgrade/gather-azure-cli/artifacts/ci-op-v9k3j1h2-696e9-c68h4-worker-centralus3-7rftw-boot.log

Extracted Kernel Panic

[ 1464.958061] general protection fault, probably for non-canonical address 0x13c0080013a0108: 0000 [#1] PREEMPT SMP PTI
[ 1464.962560] CPU: 2 PID: 134527 Comm: runc Not tainted 5.14.0-252.el9.x86_64 #1
[ 1464.965635] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 05/09/2022
[ 1464.970125] RIP: 0010:__mod_lruvec_page_state+0x80/0x160
[ 1464.972441] Code: 48 83 e0 fc 83 e2 02 74 04 48 8b 40 10 48 85 c0 0f 84 aa 00 00 00 0f 1f 44 00 00 49 63 96 00 a2 02 00 48 8b bc d0 88 08 00 00 <4c> 3b b7 88 00 00 00 0f 85 b8 00 00 00 44 89 e2 89 ee e8 19 ff ff
[ 1464.980206] RSP: 0018:ffffafa4c5b37ad8 EFLAGS: 00010086
[ 1464.982341] RAX: ffff9ec2eed7b000 RBX: ffffeb2849b038c0 RCX: 0000000000000000
[ 1464.985183] RDX: 0000000000000000 RSI: 0000000000000013 RDI: 013c0080013a0080
[ 1464.988173] RBP: 0000000000000013 R08: ffffffffffffffc0 R09: fffffffffffffffe
[ 1464.991184] R10: 0000000000000040 R11: 0000000000000003 R12: 00000000ffffffff
[ 1464.994003] R13: ffffeb2849b038c0 R14: ffff9ec53ffd5000 R15: ffff9ec4126920f8
[ 1464.996953] FS: 00007f8507f8d640(0000) GS:ffff9ec52fd00000(0000) knlGS:0000000000000000
[ 1464.999831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1465.002083] CR2: 00007f8507751000 CR3: 00000003421ce004 CR4: 00000000003706e0
[ 1465.004977] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1465.007534] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1465.010194] Call Trace:
[ 1465.011257] <TASK>
[ 1465.012046] filemap_unaccount_folio+0x61/0x1c0
[ 1465.013802] __filemap_remove_folio+0x39/0x240
[ 1465.015481] ? xas_load+0x5/0xa0
[ 1465.016728] ? xas_find+0x171/0x1c0
[ 1465.017948] ? find_lock_entries+0x18e/0x250
[ 1465.019549] filemap_remove_folio+0x3f/0xa0
[ 1465.021113] truncate_inode_folio+0x1f/0x40
[ 1465.022904] shmem_undo_range+0x1ab/0x6e0
[ 1465.024392] ? try_to_unlazy+0x4c/0x90
[ 1465.025779] ? terminate_walk+0x61/0xf0
[ 1465.027332] ? avc_has_perm_noaudit+0x94/0x110
[ 1465.029022] ? rcu_nocb_try_bypass+0x4d/0x440
[ 1465.030600] shmem_evict_inode+0x103/0x270
[ 1465.032123] ? __ia32_sys_membarrier+0x20/0x20
[ 1465.033845] evict+0xcf/0x1d0
[ 1465.035042] do_unlinkat+0x1dc/0x2e0
[ 1465.036415] __x64_sys_unlinkat+0x33/0x60
[ 1465.037864] do_syscall_64+0x5c/0x90
[ 1465.039194] ? syscall_exit_work+0x11a/0x150
[ 1465.040875] ? syscall_exit_to_user_mode+0x12/0x30
[ 1465.043436] ? do_syscall_64+0x69/0x90
[ 1465.045527] ? syscall_exit_to_user_mode+0x12/0x30
[ 1465.048047] ? do_syscall_64+0x69/0x90
[ 1465.050176] ? syscall_exit_work+0x11a/0x150
[ 1465.052548] ? syscall_exit_to_user_mode+0x12/0x30
[ 1465.054987] ? do_syscall_64+0x69/0x90
[ 1465.057104] ? do_syscall_64+0x69/0x90
[ 1465.059253] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 1465.062259] RIP: 0033:0x5587bac69d7b
[ 1465.064433] Code: e8 4a 2e fb ff eb 88 cc cc cc cc cc cc cc cc e8 bb 75 fb ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
[ 1465.072361] RSP: 002b:000000c0001badb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000107
[ 1465.075961] RAX: ffffffffffffffda RBX: 000000c000032500 RCX: 00005587bac69d7b
[ 1465.079123] RDX: 0000000000000000 RSI: 000000c000092420 RDI: 000000000000000a
[ 1465.082276] RBP: 000000c0001bae10 R08: 00007f852f3c0501 R09: 0000000000000001
[ 1465.085456] R10: 00007f850774d910 R11: 0000000000000202 R12: 0000000000000000
[ 1465.088630] R13: 0000000000000000 R14: 000000c0000041a0 R15: 0000000000000042
[ 1465.091992] </TASK>
[ 1465.093442] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 nfs fscache netfs nfsd auth_rpcgss nfs_acl lockd grace loop nls_utf8 cifs cifs_arc4 cifs_md4 dns_resolver dummy xt_sctp veth nf_conntrack_netlink xt_recent xt_statistic xt_nat xt_addrtype xt_LOG nf_log_syslog ipt_REJECT nf_reject_ipv4 xt_CT xt_MASQUERADE nft_chain_nat xt_mark xt_conntrack xt_comment nft_compat nft_counter nf_tables rfkill vxlan ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay ext4 mbcache jbd2 intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit libnvdimm kvm_intel kvm hyperv_drm irqbypass rapl drm_shmem_helper drm_kms_helper syscopyarea pcspkr sysfillrect sysimgblt hv_balloon fb_sys_fops hv_utils joydev ip_tables drm rpcrdma sunrpc rdma_ucm ib_srpt xfs ib_isert iscsi_target_mod target_core_mod ib_iser libcrc32c libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm dm_multipath ib_cm mlx5_ib
[ 1465.093517] ib_uverbs ib_core mlx5_core mlxfw psample tls pci_hyperv pci_hyperv_intf sd_mod sg nvme_tcp nvme_fabrics nvme serio_raw hid_hyperv nvme_core hv_netvsc hv_storvsc hyperv_keyboard scsi_transport_fc nvme_common t10_pi crct10dif_pclmul crc32_pclmul crc32c_intel hv_vmbus ghash_clmulni_intel dm_mirror dm_region_hash dm_log dm_mod fuse
[ 1465.143626] ---[ end trace 155f508cadcc322c ]---
[ 1465.146224] RIP: 0010:__mod_lruvec_page_state+0x80/0x160
[ 1465.148861] Code: 48 83 e0 fc 83 e2 02 74 04 48 8b 40 10 48 85 c0 0f 84 aa 00 00 00 0f 1f 44 00 00 49 63 96 00 a2 02 00 48 8b bc d0 88 08 00 00 <4c> 3b b7 88 00 00 00 0f 85 b8 00 00 00 44 89 e2 89 ee e8 19 ff ff
[ 1465.157470] RSP: 0018:ffffafa4c5b37ad8 EFLAGS: 00010086
[ 1465.160294] RAX: ffff9ec2eed7b000 RBX: ffffeb2849b038c0 RCX: 0000000000000000
[ 1465.163748] RDX: 0000000000000000 RSI: 0000000000000013 RDI: 013c0080013a0080
[ 1465.167162] RBP: 0000000000000013 R08: ffffffffffffffc0 R09: fffffffffffffffe
[ 1465.170654] R10: 0000000000000040 R11: 0000000000000003 R12: 00000000ffffffff
[ 1465.174005] R13: ffffeb2849b038c0 R14: ffff9ec53ffd5000 R15: ffff9ec4126920f8
[ 1465.177511] FS: 00007f8507f8d640(0000) GS:ffff9ec52fd00000(0000) knlGS:0000000000000000
[ 1465.181148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1465.184004] CR2: 00007f8507751000 CR3: 00000003421ce004 CR4: 00000000003706e0
[ 1465.187763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1465.191128] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1465.194690] Kernel panic - not syncing: Fatal exception
[ 1465.198795] Kernel Offset: 0x20400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1465.223696] Rebooting in 10 seconds..
[ 1475.227502] list_add double add: new=ffffffffa2e331c0, prev=ffffffffa2e2cf28, next=ffffffffa2e331c0.
[ 1475.232317] ------------[ cut here ]------------
[ 1475.234979] kernel BUG at lib/list_debug.c:29!
[ 1475.237600] invalid opcode: 0000 [#2] PREEMPT SMP PTI
[ 1475.240503] CPU: 2 PID: 134527 Comm: runc Tainted: G D --------- --- 5.14.0-252.el9.x86_64 #1
[ 1475.245316] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 05/09/2022
[ 1475.250685] RIP: 0010:__list_add_valid.cold+0x26/0x3f
[ 1475.253826] Code: f7 62 a8 ff 4c 89 c1 48 c7 c7 30 f9 84 a2 e8 0b f1 fe ff 0f 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 e0 f9 84 a2 e8 f4 f0 fe ff <0f> 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 88 f9 84 a2 e8 dd f0 fe
[ 1475.263241] RSP: 0018:ffffafa4c5b37860 EFLAGS: 00010046
[ 1475.266275] RAX: 0000000000000058 RBX: ffffffffa2e331c0 RCX: 0000000000000027
[ 1475.270345] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9ec52fd198a0
[ 1475.274039] RBP: ffffffffa2e2cf28 R08: 0000000000000000 R09: 00000000ffff7fff
[ 1475.278029] R10: ffffafa4c5b37708 R11: ffffffffa31e9608 R12: ffffffffa2e331c0
[ 1475.281722] R13: 0000000000000000 R14: 0000000000000092 R15: ffffffffa2e2cf20
[ 1475.285487] FS: 00007f8507f8d640(0000) GS:ffff9ec52fd00000(0000) knlGS:0000000000000000
[ 1475.289419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1475.292873] CR2: 00007f8507751000 CR3: 00000003421ce004 CR4: 00000000003706e0
[ 1475.296504] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1475.300061] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1475.303628] Call Trace:
[ 1475.305472] <TASK>
[ 1475.307126] __register_nmi_handler+0xd3/0x130
[ 1475.309771] nmi_shootdown_cpus+0x38/0x90
[ 1475.312185] native_machine_emergency_restart+0x1b0/0x1d0
[ 1475.314903] panic+0x262/0x2c6
[ 1475.317129] oops_end.cold+0x18/0x18
[ 1475.319163] exc_general_protection+0x1b3/0x400
[ 1475.321660] asm_exc_general_protection+0x22/0x30
[ 1475.324118] RIP: 0010:__mod_lruvec_page_state+0x80/0x160
[ 1475.326807] Code: 48 83 e0 fc 83 e2 02 74 04 48 8b 40 10 48 85 c0 0f 84 aa 00 00 00 0f 1f 44 00 00 49 63 96 00 a2 02 00 48 8b bc d0 88 08 00 00 <4c> 3b b7 88 00 00 00 0f 85 b8 00 00 00 44 89 e2 89 ee e8 19 ff ff
[ 1475.335288] RSP: 0018:ffffafa4c5b37ad8 EFLAGS: 00010086
[ 1475.337989] RAX: ffff9ec2eed7b000 RBX: ffffeb2849b038c0 RCX: 0000000000000000
[ 1475.341320] RDX: 0000000000000000 RSI: 0000000000000013 RDI: 013c0080013a0080
[ 1475.344575] RBP: 0000000000000013 R08: ffffffffffffffc0 R09: fffffffffffffffe
[ 1475.347866] R10: 0000000000000040 R11: 0000000000000003 R12: 00000000ffffffff
[ 1475.351195] R13: ffffeb2849b038c0 R14: ffff9ec53ffd5000 R15: ffff9ec4126920f8
[ 1475.354434] ? __mod_lruvec_page_state+0x3e/0x160
[ 1475.356810] filemap_unaccount_folio+0x61/0x1c0
[ 1475.359215] __filemap_remove_folio+0x39/0x240
[ 1475.361610] ? xas_load+0x5/0xa0
[ 1475.363492] ? xas_find+0x171/0x1c0
[ 1475.365531] ? find_lock_entries+0x18e/0x250
[ 1475.367763] filemap_remove_folio+0x3f/0xa0
[ 1475.369999] truncate_inode_folio+0x1f/0x40
[ 1475.372361] shmem_undo_range+0x1ab/0x6e0
[ 1475.374430] ? try_to_unlazy+0x4c/0x90
[ 1475.376528] ? terminate_walk+0x61/0xf0
[ 1475.378603] ? avc_has_perm_noaudit+0x94/0x110
[ 1475.380932] ? rcu_nocb_try_bypass+0x4d/0x440
[ 1475.383093] shmem_evict_inode+0x103/0x270
[ 1475.385135] ? __ia32_sys_membarrier+0x20/0x20
[ 1475.387296] evict+0xcf/0x1d0
[ 1475.388947] do_unlinkat+0x1dc/0x2e0
[ 1475.390893] __x64_sys_unlinkat+0x33/0x60
[ 1475.392955] do_syscall_64+0x5c/0x90
[ 1475.394751] ? syscall_exit_work+0x11a/0x150
[ 1475.396993] ? syscall_exit_to_user_mode+0x12/0x30
[ 1475.399207] ? do_syscall_64+0x69/0x90
[ 1475.401059] ? syscall_exit_to_user_mode+0x12/0x30
[ 1475.403470] ? do_syscall_64+0x69/0x90
[ 1475.405458] ? syscall_exit_work+0x11a/0x150
[ 1475.407707] ? syscall_exit_to_user_mode+0x12/0x30
[ 1475.409869] ? do_syscall_64+0x69/0x90
[ 1475.411808] ? do_syscall_64+0x69/0x90
[ 1475.413633] entry_SYSCALL_64_after_hwframe+0x63/0xcd
[ 1475.415902] RIP: 0033:0x5587bac69d7b
[ 1475.417695] Code: e8 4a 2e fb ff eb 88 cc cc cc cc cc cc cc cc e8 bb 75 fb ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
[ 1475.425184] RSP: 002b:000000c0001badb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000107
[ 1475.428625] RAX: ffffffffffffffda RBX: 000000c000032500 RCX: 00005587bac69d7b
[ 1475.431905] RDX: 0000000000000000 RSI: 000000c000092420 RDI: 000000000000000a
[ 1475.435140] RBP: 000000c0001bae10 R08: 00007f852f3c0501 R09: 0000000000000001
[ 1475.438132] R10: 00007f850774d910 R11: 0000000000000202 R12: 0000000000000000
[ 1475.441230] R13: 0000000000000000 R14: 000000c0000041a0 R15: 0000000000000042
[ 1475.444543] </TASK>
[ 1475.445891] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 nfs fscache netfs nfsd auth_rpcgss nfs_acl lockd grace loop nls_utf8 cifs cifs_arc4 cifs_md4 dns_resolver dummy xt_sctp veth nf_conntrack_netlink xt_recent xt_statistic xt_nat xt_addrtype xt_LOG nf_log_syslog ipt_REJECT nf_reject_ipv4 xt_CT xt_MASQUERADE nft_chain_nat xt_mark xt_conntrack xt_comment nft_compat nft_counter nf_tables rfkill vxlan ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay ext4 mbcache jbd2 intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit libnvdimm kvm_intel kvm hyperv_drm irqbypass rapl drm_shmem_helper drm_kms_helper syscopyarea pcspkr sysfillrect sysimgblt hv_balloon fb_sys_fops hv_utils joydev ip_tables drm rpcrdma sunrpc rdma_ucm ib_srpt xfs ib_isert iscsi_target_mod target_core_mod ib_iser libcrc32c libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm dm_multipath ib_cm mlx5_ib
[ 1475.445971] ib_uverbs ib_core mlx5_core mlxfw psample tls pci_hyperv pci_hyperv_intf sd_mod sg nvme_tcp nvme_fabrics nvme serio_raw hid_hyperv nvme_core hv_netvsc hv_storvsc hyperv_keyboard scsi_transport_fc nvme_common t10_pi crct10dif_pclmul crc32_pclmul crc32c_intel hv_vmbus ghash_clmulni_intel dm_mirror dm_region_hash dm_log dm_mod fuse
[ 1475.493461] ---[ end trace 155f508cadcc322d ]---
[ 1475.495786] RIP: 0010:__mod_lruvec_page_state+0x80/0x160
[ 1475.498430] Code: 48 83 e0 fc 83 e2 02 74 04 48 8b 40 10 48 85 c0 0f 84 aa 00 00 00 0f 1f 44 00 00 49 63 96 00 a2 02 00 48 8b bc d0 88 08 00 00 <4c> 3b b7 88 00 00 00 0f 85 b8 00 00 00 44 89 e2 89 ee e8 19 ff ff
[ 1475.506822] RSP: 0018:ffffafa4c5b37ad8 EFLAGS: 00010086
[ 1475.509841] RAX: ffff9ec2eed7b000 RBX: ffffeb2849b038c0 RCX: 0000000000000000
[ 1475.513155] RDX: 0000000000000000 RSI: 0000000000000013 RDI: 013c0080013a0080
[ 1475.516479] RBP: 0000000000000013 R08: ffffffffffffffc0 R09: fffffffffffffffe
[ 1475.519839] R10: 0000000000000040 R11: 0000000000000003 R12: 00000000ffffffff
[ 1475.523178] R13: ffffeb2849b038c0 R14: ffff9ec53ffd5000 R15: ffff9ec4126920f8
[ 1475.526502] FS: 00007f8507f8d640(0000) GS:ffff9ec52fd00000(0000) knlGS:0000000000000000
[ 1475.530228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1475.533110] CR2: 00007f8507751000 CR3: 00000003421ce004 CR4: 00000000003706e0
[ 1475.536549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1475.539901] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1475.543512] Kernel panic - not syncing: Fatal exception
[ 1475.546193] Kernel Offset: 0x20400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)