With the RHCOS 9.2 testing being done before merging in PR https://github.com/openshift/machine-config-operator/pull/3485 we seem to be seeing between 1-3 Azure workers dying during the post-upgrade conformance test. This problem is prevalent in seemingly all sub-jobs of https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/aggregator-periodic-ci-openshift-release-master-ci-4.13-upgrade-from-stable-4.12-e2e-azure-sdn-upgrade/1625681132605411328 but we'll focus on just one example for this bug:
In this run if you view gather-extra nodes.json, we see two nodes had kubelet stop reporting status late in the run, worker w2776 stopped reporting at 5:38:29, and worker 7rftw stopped reporting at 5:18:48. These are last transition times so possible the actual time of death was a timeout period before. (possibly 30s, more details on this in a minute)
The death of the nodes is preventing a number of artifacts from being gathered, TRT will fix this ASAP in TRT-855, but this is why we don't see spyglass charts and other artifacts for the conformance suite run, only the upgrade suite which runs prior.
However I was able to identify system journal logs for the affected node in loki:
This shows the last reported logs we got for the 7rftw node:
I0215 05:18:14.339779 1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/GetPluginInfo","Request":{},"Response":{"name":"csi-mock-e2e-csi-mock-volumes-6973","vendor_version":"v1.7.3"},"Error":"","FullError":null}
I0215 05:18:13.925254 1 connection.go:183] GRPC call: /csi.v1.Controller/CreateVolume
I0215 05:18:14.339737 1 server.go:114] GRPC response: {"name":"csi-mock-e2e-csi-mock-volumes-6973","vendor_version":"v1.7.3"}
I0215 05:18:13.925036 1 controller.go:1279] provision "e2e-csi-mock-volumes-6973/pvc-26t2d" class "e2e-csi-mock-volumes-6973-sccszgp": started
I0215 05:18:11.809725 1 controller.go:859] Started provisioner controller csi-mock-e2e-csi-mock-volumes-6973_csi-mockplugin-0_1e0cbb41-bbbc-40c7-8188-23935d0bb52b!
I0215 05:18:14.339730 1 identityserver.go:28] Using default GetPluginInfo
I0215 05:18:11.809649 1 shared_informer.go:270] caches populated
I0215 05:18:14.339684 1 server.go:105] GRPC request: {}
I0215 05:18:11.709270 1 reflector.go:255] Listing and watching *v1.PersistentVolume from sigs.k8s.io/sig-storage-lib-external-provisioner/v7/controller/controller.go:844
I0215 05:18:14.339663 1 server.go:101] GRPC call: /csi.v1.Identity/GetPluginInfo
I0215 05:18:11.709125 1 reflector.go:219] Starting reflector *v1.PersistentVolume (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v7/controller/controller.go:844
I0215 05:18:14.338746 1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{},"Error":"","FullError":null}
I0215 05:18:11.709166 1 reflector.go:255] Listing and watching *v1.StorageClass from sigs.k8s.io/sig-storage-lib-external-provisioner/v7/controller/controller.go:847
I0215 05:18:14.338726 1 server.go:114] GRPC response: {}
I0215 05:18:11.709125 1 reflector.go:219] Starting reflector *v1.StorageClass (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v7/controller/controller.go:847
I0215 05:18:14.338668 1 server.go:105] GRPC request: {}
I0215 05:18:11.708919 1 volume_store.go:97] Starting save volume queue
I0215 05:18:11.708909 1 clone_controller.go:82] Started CloningProtection controller
I0215 05:18:14.338617 1 server.go:101] GRPC call: /csi.v1.Identity/Probe
I0215 05:18:11.708813 1 clone_controller.go:66] Starting CloningProtection controller
30 seconds later roughly, we've transitioned the node kubelet Ready condition to Unknown. Feels like a 30s timeout before we give up on the node response.
Worker ci-op-v9k3j1h2-696e9-c68h4-worker-centralus2-w2776 which died at 5:38:29 has different logs, but still in the storage area:
2023-02-15 01:37:43
I0215 05:37:42.812659 1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{},"Error":"","FullError":null}
I0215 05:37:42.812631 1 server.go:114] GRPC response: {}
I0215 05:37:42.812548 1 server.go:105] GRPC request: {}
I0215 05:37:42.812510 1 server.go:101] GRPC call: /csi.v1.Identity/Probe
I0215 05:37:42.812659 1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{},"Error":"","FullError":null}
I0215 05:37:42.812631 1 server.go:114] GRPC response: {}
I0215 05:37:42.812548 1 server.go:105] GRPC request: {}
I0215 05:37:42.812510 1 server.go:101] GRPC call: /csi.v1.Identity/Probe
I0215 05:37:42.812659 1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{},"Error":"","FullError":null}
I0215 05:37:42.812631 1 server.go:114] GRPC response: {}
I0215 05:37:42.812548 1 server.go:105] GRPC request: {}
I0215 05:37:42.812510 1 server.go:101] GRPC call: /csi.v1.Identity/Probe
I0215 05:37:42.812659 1 server.go:159] gRPCCall: {"Method":"/csi.v1.Identity/Probe","Request":{},"Response":{},"Error":"","FullError":null}
I0215 05:37:42.812631 1 server.go:114] GRPC response: {}
I0215 05:37:42.812548 1 server.go:105] GRPC request: {}
I0215 05:37:42.812510 1 server.go:101] GRPC call: /csi.v1.Identity/Probe
I0215 05:37:42.117439 1 controller.go:151] Successfully synced 'e2e-volume-8456/hostpath-injector'
I0215 05:37:42.117379 1 controller.go:192] Received pod 'hostpath-injector'
Kernel panic logs found for both dead nodes:
Extracted Kernel Panic
[ 1464.958061] general protection fault, probably for non-canonical address 0x13c0080013a0108: 0000 [#1] PREEMPT SMP PTI [ 1464.962560] CPU: 2 PID: 134527 Comm: runc Not tainted 5.14.0-252.el9.x86_64 #1 [ 1464.965635] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 05/09/2022 [ 1464.970125] RIP: 0010:__mod_lruvec_page_state+0x80/0x160 [ 1464.972441] Code: 48 83 e0 fc 83 e2 02 74 04 48 8b 40 10 48 85 c0 0f 84 aa 00 00 00 0f 1f 44 00 00 49 63 96 00 a2 02 00 48 8b bc d0 88 08 00 00 <4c> 3b b7 88 00 00 00 0f 85 b8 00 00 00 44 89 e2 89 ee e8 19 ff ff [ 1464.980206] RSP: 0018:ffffafa4c5b37ad8 EFLAGS: 00010086 [ 1464.982341] RAX: ffff9ec2eed7b000 RBX: ffffeb2849b038c0 RCX: 0000000000000000 [ 1464.985183] RDX: 0000000000000000 RSI: 0000000000000013 RDI: 013c0080013a0080 [ 1464.988173] RBP: 0000000000000013 R08: ffffffffffffffc0 R09: fffffffffffffffe [ 1464.991184] R10: 0000000000000040 R11: 0000000000000003 R12: 00000000ffffffff [ 1464.994003] R13: ffffeb2849b038c0 R14: ffff9ec53ffd5000 R15: ffff9ec4126920f8 [ 1464.996953] FS: 00007f8507f8d640(0000) GS:ffff9ec52fd00000(0000) knlGS:0000000000000000 [ 1464.999831] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1465.002083] CR2: 00007f8507751000 CR3: 00000003421ce004 CR4: 00000000003706e0 [ 1465.004977] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1465.007534] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1465.010194] Call Trace: [ 1465.011257] <TASK> [ 1465.012046] filemap_unaccount_folio+0x61/0x1c0 [ 1465.013802] __filemap_remove_folio+0x39/0x240 [ 1465.015481] ? xas_load+0x5/0xa0 [ 1465.016728] ? xas_find+0x171/0x1c0 [ 1465.017948] ? find_lock_entries+0x18e/0x250 [ 1465.019549] filemap_remove_folio+0x3f/0xa0 [ 1465.021113] truncate_inode_folio+0x1f/0x40 [ 1465.022904] shmem_undo_range+0x1ab/0x6e0 [ 1465.024392] ? try_to_unlazy+0x4c/0x90 [ 1465.025779] ? terminate_walk+0x61/0xf0 [ 1465.027332] ? avc_has_perm_noaudit+0x94/0x110 [ 1465.029022] ? rcu_nocb_try_bypass+0x4d/0x440 [ 1465.030600] shmem_evict_inode+0x103/0x270 [ 1465.032123] ? __ia32_sys_membarrier+0x20/0x20 [ 1465.033845] evict+0xcf/0x1d0 [ 1465.035042] do_unlinkat+0x1dc/0x2e0 [ 1465.036415] __x64_sys_unlinkat+0x33/0x60 [ 1465.037864] do_syscall_64+0x5c/0x90 [ 1465.039194] ? syscall_exit_work+0x11a/0x150 [ 1465.040875] ? syscall_exit_to_user_mode+0x12/0x30 [ 1465.043436] ? do_syscall_64+0x69/0x90 [ 1465.045527] ? syscall_exit_to_user_mode+0x12/0x30 [ 1465.048047] ? do_syscall_64+0x69/0x90 [ 1465.050176] ? syscall_exit_work+0x11a/0x150 [ 1465.052548] ? syscall_exit_to_user_mode+0x12/0x30 [ 1465.054987] ? do_syscall_64+0x69/0x90 [ 1465.057104] ? do_syscall_64+0x69/0x90 [ 1465.059253] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 1465.062259] RIP: 0033:0x5587bac69d7b [ 1465.064433] Code: e8 4a 2e fb ff eb 88 cc cc cc cc cc cc cc cc e8 bb 75 fb ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30 [ 1465.072361] RSP: 002b:000000c0001badb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000107 [ 1465.075961] RAX: ffffffffffffffda RBX: 000000c000032500 RCX: 00005587bac69d7b [ 1465.079123] RDX: 0000000000000000 RSI: 000000c000092420 RDI: 000000000000000a [ 1465.082276] RBP: 000000c0001bae10 R08: 00007f852f3c0501 R09: 0000000000000001 [ 1465.085456] R10: 00007f850774d910 R11: 0000000000000202 R12: 0000000000000000 [ 1465.088630] R13: 0000000000000000 R14: 000000c0000041a0 R15: 0000000000000042 [ 1465.091992] </TASK> [ 1465.093442] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 nfs fscache netfs nfsd auth_rpcgss nfs_acl lockd grace loop nls_utf8 cifs cifs_arc4 cifs_md4 dns_resolver dummy xt_sctp veth nf_conntrack_netlink xt_recent xt_statistic xt_nat xt_addrtype xt_LOG nf_log_syslog ipt_REJECT nf_reject_ipv4 xt_CT xt_MASQUERADE nft_chain_nat xt_mark xt_conntrack xt_comment nft_compat nft_counter nf_tables rfkill vxlan ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay ext4 mbcache jbd2 intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit libnvdimm kvm_intel kvm hyperv_drm irqbypass rapl drm_shmem_helper drm_kms_helper syscopyarea pcspkr sysfillrect sysimgblt hv_balloon fb_sys_fops hv_utils joydev ip_tables drm rpcrdma sunrpc rdma_ucm ib_srpt xfs ib_isert iscsi_target_mod target_core_mod ib_iser libcrc32c libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm dm_multipath ib_cm mlx5_ib [ 1465.093517] ib_uverbs ib_core mlx5_core mlxfw psample tls pci_hyperv pci_hyperv_intf sd_mod sg nvme_tcp nvme_fabrics nvme serio_raw hid_hyperv nvme_core hv_netvsc hv_storvsc hyperv_keyboard scsi_transport_fc nvme_common t10_pi crct10dif_pclmul crc32_pclmul crc32c_intel hv_vmbus ghash_clmulni_intel dm_mirror dm_region_hash dm_log dm_mod fuse [ 1465.143626] ---[ end trace 155f508cadcc322c ]--- [ 1465.146224] RIP: 0010:__mod_lruvec_page_state+0x80/0x160 [ 1465.148861] Code: 48 83 e0 fc 83 e2 02 74 04 48 8b 40 10 48 85 c0 0f 84 aa 00 00 00 0f 1f 44 00 00 49 63 96 00 a2 02 00 48 8b bc d0 88 08 00 00 <4c> 3b b7 88 00 00 00 0f 85 b8 00 00 00 44 89 e2 89 ee e8 19 ff ff [ 1465.157470] RSP: 0018:ffffafa4c5b37ad8 EFLAGS: 00010086 [ 1465.160294] RAX: ffff9ec2eed7b000 RBX: ffffeb2849b038c0 RCX: 0000000000000000 [ 1465.163748] RDX: 0000000000000000 RSI: 0000000000000013 RDI: 013c0080013a0080 [ 1465.167162] RBP: 0000000000000013 R08: ffffffffffffffc0 R09: fffffffffffffffe [ 1465.170654] R10: 0000000000000040 R11: 0000000000000003 R12: 00000000ffffffff [ 1465.174005] R13: ffffeb2849b038c0 R14: ffff9ec53ffd5000 R15: ffff9ec4126920f8 [ 1465.177511] FS: 00007f8507f8d640(0000) GS:ffff9ec52fd00000(0000) knlGS:0000000000000000 [ 1465.181148] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1465.184004] CR2: 00007f8507751000 CR3: 00000003421ce004 CR4: 00000000003706e0 [ 1465.187763] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1465.191128] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1465.194690] Kernel panic - not syncing: Fatal exception [ 1465.198795] Kernel Offset: 0x20400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff) [ 1465.223696] Rebooting in 10 seconds.. [ 1475.227502] list_add double add: new=ffffffffa2e331c0, prev=ffffffffa2e2cf28, next=ffffffffa2e331c0. [ 1475.232317] ------------[ cut here ]------------ [ 1475.234979] kernel BUG at lib/list_debug.c:29! [ 1475.237600] invalid opcode: 0000 [#2] PREEMPT SMP PTI [ 1475.240503] CPU: 2 PID: 134527 Comm: runc Tainted: G D --------- --- 5.14.0-252.el9.x86_64 #1 [ 1475.245316] Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.1 05/09/2022 [ 1475.250685] RIP: 0010:__list_add_valid.cold+0x26/0x3f [ 1475.253826] Code: f7 62 a8 ff 4c 89 c1 48 c7 c7 30 f9 84 a2 e8 0b f1 fe ff 0f 0b 48 89 f2 4c 89 c1 48 89 fe 48 c7 c7 e0 f9 84 a2 e8 f4 f0 fe ff <0f> 0b 48 89 d1 4c 89 c6 4c 89 ca 48 c7 c7 88 f9 84 a2 e8 dd f0 fe [ 1475.263241] RSP: 0018:ffffafa4c5b37860 EFLAGS: 00010046 [ 1475.266275] RAX: 0000000000000058 RBX: ffffffffa2e331c0 RCX: 0000000000000027 [ 1475.270345] RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff9ec52fd198a0 [ 1475.274039] RBP: ffffffffa2e2cf28 R08: 0000000000000000 R09: 00000000ffff7fff [ 1475.278029] R10: ffffafa4c5b37708 R11: ffffffffa31e9608 R12: ffffffffa2e331c0 [ 1475.281722] R13: 0000000000000000 R14: 0000000000000092 R15: ffffffffa2e2cf20 [ 1475.285487] FS: 00007f8507f8d640(0000) GS:ffff9ec52fd00000(0000) knlGS:0000000000000000 [ 1475.289419] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1475.292873] CR2: 00007f8507751000 CR3: 00000003421ce004 CR4: 00000000003706e0 [ 1475.296504] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1475.300061] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1475.303628] Call Trace: [ 1475.305472] <TASK> [ 1475.307126] __register_nmi_handler+0xd3/0x130 [ 1475.309771] nmi_shootdown_cpus+0x38/0x90 [ 1475.312185] native_machine_emergency_restart+0x1b0/0x1d0 [ 1475.314903] panic+0x262/0x2c6 [ 1475.317129] oops_end.cold+0x18/0x18 [ 1475.319163] exc_general_protection+0x1b3/0x400 [ 1475.321660] asm_exc_general_protection+0x22/0x30 [ 1475.324118] RIP: 0010:__mod_lruvec_page_state+0x80/0x160 [ 1475.326807] Code: 48 83 e0 fc 83 e2 02 74 04 48 8b 40 10 48 85 c0 0f 84 aa 00 00 00 0f 1f 44 00 00 49 63 96 00 a2 02 00 48 8b bc d0 88 08 00 00 <4c> 3b b7 88 00 00 00 0f 85 b8 00 00 00 44 89 e2 89 ee e8 19 ff ff [ 1475.335288] RSP: 0018:ffffafa4c5b37ad8 EFLAGS: 00010086 [ 1475.337989] RAX: ffff9ec2eed7b000 RBX: ffffeb2849b038c0 RCX: 0000000000000000 [ 1475.341320] RDX: 0000000000000000 RSI: 0000000000000013 RDI: 013c0080013a0080 [ 1475.344575] RBP: 0000000000000013 R08: ffffffffffffffc0 R09: fffffffffffffffe [ 1475.347866] R10: 0000000000000040 R11: 0000000000000003 R12: 00000000ffffffff [ 1475.351195] R13: ffffeb2849b038c0 R14: ffff9ec53ffd5000 R15: ffff9ec4126920f8 [ 1475.354434] ? __mod_lruvec_page_state+0x3e/0x160 [ 1475.356810] filemap_unaccount_folio+0x61/0x1c0 [ 1475.359215] __filemap_remove_folio+0x39/0x240 [ 1475.361610] ? xas_load+0x5/0xa0 [ 1475.363492] ? xas_find+0x171/0x1c0 [ 1475.365531] ? find_lock_entries+0x18e/0x250 [ 1475.367763] filemap_remove_folio+0x3f/0xa0 [ 1475.369999] truncate_inode_folio+0x1f/0x40 [ 1475.372361] shmem_undo_range+0x1ab/0x6e0 [ 1475.374430] ? try_to_unlazy+0x4c/0x90 [ 1475.376528] ? terminate_walk+0x61/0xf0 [ 1475.378603] ? avc_has_perm_noaudit+0x94/0x110 [ 1475.380932] ? rcu_nocb_try_bypass+0x4d/0x440 [ 1475.383093] shmem_evict_inode+0x103/0x270 [ 1475.385135] ? __ia32_sys_membarrier+0x20/0x20 [ 1475.387296] evict+0xcf/0x1d0 [ 1475.388947] do_unlinkat+0x1dc/0x2e0 [ 1475.390893] __x64_sys_unlinkat+0x33/0x60 [ 1475.392955] do_syscall_64+0x5c/0x90 [ 1475.394751] ? syscall_exit_work+0x11a/0x150 [ 1475.396993] ? syscall_exit_to_user_mode+0x12/0x30 [ 1475.399207] ? do_syscall_64+0x69/0x90 [ 1475.401059] ? syscall_exit_to_user_mode+0x12/0x30 [ 1475.403470] ? do_syscall_64+0x69/0x90 [ 1475.405458] ? syscall_exit_work+0x11a/0x150 [ 1475.407707] ? syscall_exit_to_user_mode+0x12/0x30 [ 1475.409869] ? do_syscall_64+0x69/0x90 [ 1475.411808] ? do_syscall_64+0x69/0x90 [ 1475.413633] entry_SYSCALL_64_after_hwframe+0x63/0xcd [ 1475.415902] RIP: 0033:0x5587bac69d7b [ 1475.417695] Code: e8 4a 2e fb ff eb 88 cc cc cc cc cc cc cc cc e8 bb 75 fb ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30 [ 1475.425184] RSP: 002b:000000c0001badb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000107 [ 1475.428625] RAX: ffffffffffffffda RBX: 000000c000032500 RCX: 00005587bac69d7b [ 1475.431905] RDX: 0000000000000000 RSI: 000000c000092420 RDI: 000000000000000a [ 1475.435140] RBP: 000000c0001bae10 R08: 00007f852f3c0501 R09: 0000000000000001 [ 1475.438132] R10: 00007f850774d910 R11: 0000000000000202 R12: 0000000000000000 [ 1475.441230] R13: 0000000000000000 R14: 000000c0000041a0 R15: 0000000000000042 [ 1475.444543] </TASK> [ 1475.445891] Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 nfs fscache netfs nfsd auth_rpcgss nfs_acl lockd grace loop nls_utf8 cifs cifs_arc4 cifs_md4 dns_resolver dummy xt_sctp veth nf_conntrack_netlink xt_recent xt_statistic xt_nat xt_addrtype xt_LOG nf_log_syslog ipt_REJECT nf_reject_ipv4 xt_CT xt_MASQUERADE nft_chain_nat xt_mark xt_conntrack xt_comment nft_compat nft_counter nf_tables rfkill vxlan ip6_udp_tunnel udp_tunnel nfnetlink_cttimeout nfnetlink openvswitch nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay ext4 mbcache jbd2 intel_rapl_msr intel_rapl_common isst_if_mbox_msr isst_if_common nfit libnvdimm kvm_intel kvm hyperv_drm irqbypass rapl drm_shmem_helper drm_kms_helper syscopyarea pcspkr sysfillrect sysimgblt hv_balloon fb_sys_fops hv_utils joydev ip_tables drm rpcrdma sunrpc rdma_ucm ib_srpt xfs ib_isert iscsi_target_mod target_core_mod ib_iser libcrc32c libiscsi scsi_transport_iscsi ib_umad rdma_cm ib_ipoib iw_cm dm_multipath ib_cm mlx5_ib [ 1475.445971] ib_uverbs ib_core mlx5_core mlxfw psample tls pci_hyperv pci_hyperv_intf sd_mod sg nvme_tcp nvme_fabrics nvme serio_raw hid_hyperv nvme_core hv_netvsc hv_storvsc hyperv_keyboard scsi_transport_fc nvme_common t10_pi crct10dif_pclmul crc32_pclmul crc32c_intel hv_vmbus ghash_clmulni_intel dm_mirror dm_region_hash dm_log dm_mod fuse [ 1475.493461] ---[ end trace 155f508cadcc322d ]--- [ 1475.495786] RIP: 0010:__mod_lruvec_page_state+0x80/0x160 [ 1475.498430] Code: 48 83 e0 fc 83 e2 02 74 04 48 8b 40 10 48 85 c0 0f 84 aa 00 00 00 0f 1f 44 00 00 49 63 96 00 a2 02 00 48 8b bc d0 88 08 00 00 <4c> 3b b7 88 00 00 00 0f 85 b8 00 00 00 44 89 e2 89 ee e8 19 ff ff [ 1475.506822] RSP: 0018:ffffafa4c5b37ad8 EFLAGS: 00010086 [ 1475.509841] RAX: ffff9ec2eed7b000 RBX: ffffeb2849b038c0 RCX: 0000000000000000 [ 1475.513155] RDX: 0000000000000000 RSI: 0000000000000013 RDI: 013c0080013a0080 [ 1475.516479] RBP: 0000000000000013 R08: ffffffffffffffc0 R09: fffffffffffffffe [ 1475.519839] R10: 0000000000000040 R11: 0000000000000003 R12: 00000000ffffffff [ 1475.523178] R13: ffffeb2849b038c0 R14: ffff9ec53ffd5000 R15: ffff9ec4126920f8 [ 1475.526502] FS: 00007f8507f8d640(0000) GS:ffff9ec52fd00000(0000) knlGS:0000000000000000 [ 1475.530228] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1475.533110] CR2: 00007f8507751000 CR3: 00000003421ce004 CR4: 00000000003706e0 [ 1475.536549] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 1475.539901] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 1475.543512] Kernel panic - not syncing: Fatal exception [ 1475.546193] Kernel Offset: 0x20400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
- blocks
-
COS-1926 Move RHCOS to RHEL 9.2 in OCP 4.13
-
- Closed
-
- is duplicated by
-
OCPBUGS-7821 RHEL CoreOS 9.2: Metal IPI Worker Nodes Rebooting During E2E Testing
-
- Closed
-
- links to
