Loading...

Type: Bug
Resolution: Unresolved
Priority: Critical
Fix Version/s: rhel-10.0.beta
Affects Version/s: rhel-10.0.beta
Component/s: kexec-tools
Labels:
- cki-datawarehouse

Fixed in Build:
kexec-tools-2.0.28-12.el10
Regression:
None
Severity:
None

Pool Team:

sst_kernel_debug
Sub-System Group:

ssg_core_kernel

Dev Target Milestone:
18
Internal Target Milestone:
20
Story Points:
2
ACKs Check:

Dev ack
Blocked:
False
Blocked Reason:

Hide

None

Show
None
Product Documentation Required:
None
Sprint:
None

Preliminary Testing:
Pass
Errata Link:
https://errata.engineering.redhat.com/advisory/131448
Test Coverage:
None

Experience:

SFDC Cases Counter:
SFDC Cases Open:
SFDC Cases Links:

Planning:
None

Tested on RHEL-10.0-20240605.93+ kernel-6.10.0-0.rc2.10.el10 on ppc64le and aarch64,trigger panic with the default kdump config and used crash to against this vmcore file:

ppc64le:
(1) crash against the vmcore file,crash command 'swap' and 'kmem -s ' report 'crash: page excluded: kernel virtual address: c0000000040d56b0 type: "xa_node shift"'
(2) crash on live system,it report "invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size"
aarch64:
(1) crash against the vmcore file,it report 'crash: page excluded: kernel virtual address: ffff0000c1c50700 type: "xa_node.slots[off]"' or 'kmem -s' report many error messages
(2) crash on live system, crash command 'swap' and 'kmem -s ' report "invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size".

ppc64le - crash against the vmcore file:

# crash /usr/lib/debug/lib/modules/6.10.0-0.rc2.10.el10.ppc64le/vmlinux /var/crash/127.0.0.1-2024-06-11-16:47:04/vmcore
crash 8.0.5-1.el10
Copyright (C) 2002-2024  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
NOTE: stdin: not a tty

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...


please wait... (gathering kmem slab cache data)
                                                

please wait... (gathering module symbol data)
                                              

please wait... (gathering task table data)
crash: page excluded: kernel virtual address: c0000000040d56b0  type: "xa_node shift"

ppc64le - crash on live system:

# crash
crash 8.0.5-1.el10
Copyright (C) 2002-2024  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
NOTE: stdin: not a tty

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "powerpc64le-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

      KERNEL: /usr/lib/debug/lib/modules/6.10.0-0.rc2.10.el10.ppc64le/vmlinux
    DUMPFILE: /proc/kcore
        CPUS: 8
        DATE: Tue Jun 11 16:50:52 EDT 2024
      UPTIME: 00:03:10
LOAD AVERAGE: 1.57, 1.32, 0.55
       TASKS: 704
    NODENAME: ibm-p9z-25-lp9.virt.pnr.lab.eng.rdu2.redhat.com
     RELEASE: 6.10.0-0.rc2.10.el10.ppc64le
     VERSION: #1 SMP Mon Jun 10 16:52:40 EDT 2024
     MACHINE: ppc64le  (2900 Mhz)
      MEMORY: 8 GB
         PID: 7190
     COMMAND: "crash"
        TASK: c000000078b73200  [THREAD_INFO: c000000078b73200]
         CPU: 1
       STATE: TASK_RUNNING (ACTIVE)

crash> swap

swap: invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size
      FILE: memory.c  LINE: 16031  FUNCTION: dump_swap_info()

SWAP_INFO_STRUCT    TYPE       SIZE       USED     PCT  PRI  FILENAME
[/usr/bin/crash] error trace: 1144c9694 => 1143492f8 => 1143a5c00 => 1144bdd58

swap: invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size
      FILE: memory.c  LINE: 16031  FUNCTION: dump_swap_info()
crash> kmem -i

kmem: invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size
      FILE: memory.c  LINE: 16031  FUNCTION: dump_swap_info()

                 PAGES        TOTAL      PERCENTAGE
    TOTAL MEM   121140       7.4 GB         ----
         FREE    40514       2.5 GB   33% of TOTAL MEM
         USED    80626       4.9 GB   66% of TOTAL MEM
       SHARED    62329       3.8 GB   51% of TOTAL MEM
      BUFFERS       71       4.4 MB    0% of TOTAL MEM
       CACHED    63445       3.9 GB   52% of TOTAL MEM
         SLAB     5538     346.1 MB    4% of TOTAL MEM

   TOTAL HUGE        0            0         ----
    HUGE FREE        0            0    0% of TOTAL HUGE

[/usr/bin/crash] error trace: 11437ddf0 => 1143891f0 => 1143a5c00 => 1144bdd58

kmem: invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size
      FILE: memory.c  LINE: 16031  FUNCTION: dump_swap_info()

beaker job: https://beaker.engineering.redhat.com/recipes/16323458#tasks

aarch64 - crash against the vmcore file:
a. it report 'crash: page excluded: kernel virtual address: ffff0000c1c50700 type: "xa_node.slots[off]':

# crash /usr/lib/debug/lib/modules/6.10.0-0.rc2.10.el10.aarch64/vmlinux /var/crash/127.0.0.1-2024-06-13-03:12:03/vmcore
crash 8.0.5-1.el10
Copyright (C) 2002-2024  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
NOTE: stdin: not a tty

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...


please wait... (gathering kmem slab cache data)
                                                

please wait... (gathering module symbol data)
                                              

please wait... (gathering task table data)
crash: page excluded: kernel virtual address: ffff6c0a006144b8  type: "xa_node.slots[off]"

b. crash command 'kmem -i' 'kmem -s' report many error messages:

# crash /usr/lib/debug/lib/modules/6.10.0-0.rc2.10.el10.aarch64/vmlinux /var/crash/127.0.0.1-2024-06-11-07:27:23/vmcore
crash 8.0.5-1.el10
Copyright (C) 2002-2024  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
NOTE: stdin: not a tty

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...


please wait... (gathering kmem slab cache data)
                                                

please wait... (gathering module symbol data)
                                              

please wait... (gathering task table data)
crash: cannot read pid struct from xarray

crash: cannot read pid struct from xarray

crash: cannot read pid struct from xarray

crash: cannot read pid struct from xarray

.....
crash: page excluded: kernel virtual address: ffff07ff902e0000  type: "fill_task_struct"

crash: page excluded: kernel virtual address: ffff07ff8dd38000  type: "fill_task_struct"

please wait... (determining panic task)
                                        
      KERNEL: /usr/lib/debug/lib/modules/6.10.0-0.rc2.10.el10.aarch64/vmlinux
    DUMPFILE: /var/crash/127.0.0.1-2024-06-11-07:27:23/vmcore  [PARTIAL DUMP]
        CPUS: 128
        DATE: Tue Jun 11 07:26:56 EDT 2024
      UPTIME: 00:01:11
LOAD AVERAGE: 2.15, 0.90, 0.33
       TASKS: 1500
    NODENAME: ampere-mtsnow-altramax-36.khw.eng.rdu2.dc.redhat.com
     RELEASE: 6.10.0-0.rc2.10.el10.aarch64
     VERSION: #1 SMP PREEMPT_DYNAMIC Mon Jun 10 23:58:30 UTC 2024
     MACHINE: aarch64  (unknown Mhz)
      MEMORY: 63.8 GB
       PANIC: "Kernel panic - not syncing: sysrq triggered crash"
         PID: 3530
     COMMAND: "runtest.sh"
        TASK: ffff07ff96140000  [THREAD_INFO: ffff07ff96140000]
         CPU: 70
       STATE: TASK_RUNNING (PANIC)
crash> kmem -i

kmem: invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size
      FILE: memory.c  LINE: 16031  FUNCTION: dump_swap_info()

                 PAGES        TOTAL      PERCENTAGE
    TOTAL MEM  16229102      61.9 GB         ----
         FREE  15956126      60.9 GB   98% of TOTAL MEM
         USED   272976         1 GB    1% of TOTAL MEM
       SHARED    15860        62 MB    0% of TOTAL MEM
      BUFFERS     1065       4.2 MB    0% of TOTAL MEM
       CACHED    53809     210.2 MB    0% of TOTAL MEM
         SLAB    87879     343.3 MB    0% of TOTAL MEM

   TOTAL HUGE        0            0         ----
    HUGE FREE        0            0    0% of TOTAL HUGE

[/usr/bin/crash] error trace: aaaad73b6998 => aaaad73bf6c0 => aaaad73d55c0 => aaaad749c8b8

kmem: invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size
      FILE: memory.c  LINE: 16031  FUNCTION: dump_swap_info()
crash> kmem -s
CACHE             OBJSIZE  ALLOCATED     TOTAL  SLABS  SSIZE  NAME
ffff07ffcf7df000      664         13        46      1    32k  rpc_inode_cache
ffff07ffcf7dda00     2048          8        16      1    32k  rpc_buffers
ffff07ffcf7de800      224          8        64      1    16k  rpc_tasks
ffff07ff94a89600      776          3        41      1    32k  fat_inode_cache
......
kmem: proc_inode_cache: slab: fffffe1fbe260000 invalid freepointer: ffff07ff898002b8
ffff07ff8133b200      696       3794      9384    204    32k  proc_inode_cache
ffff07ff8133a600      120          5      8636    127     8k  seq_file
ffff07ff81339700       80          3      6630    130     4k  sigqueue
ffff07ff81338100     1584          8       120      6    32k  bdev_cache
ffff07ff8133a400      752       1389      5375    125    32k  shmem_inode_cache
ffff07ff8133b800       80        621      1377     27     4k  kernfs_iattrs_cache
ffff07ff81339800      136      71090     75060   1251     8k  kernfs_node_cache
ffff07ff8133bf00      336        432      2352     56    16k  mnt_cache
kmem: filp: slab: fffffe1fbe597800 invalid freepointer: ffff07ff965e02e8
kmem: filp: slab: fffffe1fbe599500 invalid freepointer: ffff07ff966542e8
ffff07ff8133ab00      232       1196     11200    175    16k  filp
ffff07ff8133af00      624      21491     27642    542    32k  inode_cache
kmem: dentry: slab: fffffe1fbe203380 invalid freepointer: ffff07ff880ceae0
kmem: dentry: slab: fffffe1fbe28af80 invalid freepointer: ffff07ff8a2beba0
kmem: dentry: slab: fffffe1fbe6b8880 invalid freepointer: ffff07ff9ae22ba0
kmem: dentry: slab: fffffe1fbe26ad80 invalid freepointer: ffff07ff89ab67e0
kmem: dentry: slab: fffffe1fbe248b80 invalid freepointer: ffff07ff8922ef60
kmem: dentry: slab: fffffe1fbe6bcb00 invalid freepointer: ffff07ff9af2c5a0
kmem: dentry: slab: fffffe1fbe940d00 invalid freepointer: ffff07ffa5034420
kmem: dentry: slab: fffffe1fbe94b600 invalid freepointer: ffff07ffa52d8ae0
kmem: dentry: slab: fffffe1fbe989300 invalid freepointer: ffff07ffa624cae0
kmem: dentry: slab: fffffe1fbe918100 invalid freepointer: ffff07ffa4604a20
kmem: dentry: slab: fffffe1fbe9c6d00 invalid freepointer: ffff07ffa71b4a20
kmem: dentry: slab: fffffe1fbe6bc000 invalid freepointer: ffff07ff9af00de0
kmem: dentry: slab: fffffe1fbe989380 invalid freepointer: ffff07ffa624e720
kmem: dentry: slab: fffffe1fbe9cc080 invalid freepointer: ffff07ffa7302660
kmem: dentry: slab: fffffe1fbea41000 invalid freepointer: ffff07ffa9040660
kmem: dentry: slab: fffffe1fbe9e8580 invalid freepointer: ffff07ffa7a16420
kmem: dentry: slab: fffffe1fbea02080 invalid freepointer: ffff07ffa80828a0
ffff07ff81339900      192      37372     48216   1148     8k  dentry
ffff07ff8133b500     4096          0      1024    128    32k  names_cache
ffff07ff81338300     4288          2        14      2    32k  net_namespace
ffff07ff8133ad00       96          0         0      0     4k  ima_iint_cache
ffff07ff81339f00       24      19684     19720    116     4k  hashtab_node
ffff07ff81338d00       64      48124     55872    873     4k  ebitmap_node
ffff07ff8133a700       36          0         0      0     4k  avtab_extended_perms
ffff07ff81338b00       24      93863     94010    553     4k  avtab_node
ffff07ff8133ba00       32          0         0      0     4k  avc_xperms_data
ffff07ff8133a000       48          0         0      0     4k  avc_xperms_decision_node
ffff07ff81338c00       56          0         0      0     4k  avc_xperms_node
kmem: avc_node: slab: fffffe1fbebb1ec0 invalid freepointer: ffff07ffaec7be30
ffff07ff81338400       72        515      8064    144     4k  avc_node
kmem: lsm_inode_cache: slab: fffffe1fbe7690c0 invalid freepointer: ffff07ff9da43818
kmem: lsm_inode_cache: slab: fffffe1fbe18a7c0 invalid freepointer: ffff07ff8629ff18
kmem: lsm_inode_cache: slab: fffffe1fbe36c180 invalid freepointer: ffff07ff8db06268
kmem: lsm_inode_cache: slab: fffffe1fbe087580 invalid freepointer: ffff07ff821d60a8
kmem: lsm_inode_cache: slab: fffffe1fbea61280 invalid freepointer: ffff07ffa984a508
kmem: lsm_inode_cache: slab: fffffe1fbe6c3240 invalid freepointer: ffff07ff9b0c9818
kmem: lsm_inode_cache: slab: fffffe1fbf395240 invalid freepointer: ffff07ffce5491f8
kmem: lsm_inode_cache: slab: fffffe1fbe3c0440 invalid freepointer: ffff07ff8f011188
kmem: lsm_inode_cache: slab: fffffe1fbe0e4640 invalid freepointer: ffff07ff83919b28
kmem: lsm_inode_cache: slab: fffffe1fbe25f380 invalid freepointer: ffff07ff897ce738
kmem: lsm_inode_cache: slab: fffffe1fbe3a7a40 invalid freepointer: ffff07ff8e9e9888
kmem: lsm_inode_cache: slab: fffffe1fbe6c9d40 invalid freepointer: ffff07ff9b275c78
ffff07ff81339c00      112      31366     37764   1049     4k  lsm_inode_cache
kmem: lsm_file_cache: slab: fffffe1fbe18a940 invalid freepointer: ffff07ff862a5528
ffff07ff8133b300       16       1224     33792    132     4k  lsm_file_cache
ffff07ff8133ae00      224         93      2496     39    16k  key_jar
ffff07ff81338000      432          5       222      6    16k  uts_namespace
ffff07ff8133a200       72         11      1344     24     4k  nsproxy
kmem: vma_lock: slab: fffffe1fbe117180 invalid freepointer: ffff07ff845c6a38
kmem: vma_lock: slab: fffffe1fbf399640 invalid freepointer: ffff07ffce6590b0
kmem: vma_lock: slab: fffffe1fbe135700 invalid freepointer: ffff07ff84d5c678
kmem: vma_lock: slab: fffffe1fbee11c80 invalid freepointer: ffff07ffb8472740
......
kmem: vma_lock: slab: fffffe1fbe8ba100 invalid freepointer: ffff07ffa2e84b50
kmem: vma_lock: slab: fffffe1fbe3a0a00 invalid freepointer: ffff07ff8e828f38
kmem: vma_lock: slab: fffffe1fbebcf200 invalid freepointer: ffff07ffaf3c86c8
kmem: vma_lock: slab: fffffe1fbf3d4580 invalid freepointer: ffff07ffcf516088
kmem: vma_lock: slab: fffffe1fbe3bccc0 invalid freepointer: ffff07ff8ef33880
kmem: vma_lock: slab: fffffe1fbe0ace40 invalid freepointer: ffff07ff82b39b78
kmem: vma_lock: slab: fffffe1fbe354b80 invalid freepointer: ffff07ff8d52e970
ffff07ff81338700       40       8733     37332    366     4k  vma_lock
kmem: vm_area_struct: slab: fffffe1fbe357b80 invalid freepointer: ffff07ff8d5ee3c8
kmem: vm_area_struct: slab: fffffe1fbe567280 invalid freepointer: ffff07ff959caaa8
kmem: vm_area_struct: slab: fffffe1fbe258280 invalid freepointer: ffff07ff8960a5d8
kmem: vm_area_struct: slab: fffffe1fbe377900 invalid freepointer: ffff07ff8dde4e18
......
kmem: vm_area_struct: slab: fffffe1fbe394380 invalid freepointer: ffff07ff8e50e058
kmem: vm_area_struct: slab: fffffe1fbe381780 invalid freepointer: ffff07ff8e05e688
kmem: vm_area_struct: slab: fffffe1fbebf4c80 invalid freepointer: ffff07ffafd32ec8
kmem: vm_area_struct: slab: fffffe1fbe3cc500 invalid freepointer: ffff07ff8f314318
kmem: vm_area_struct: slab: fffffe1fbe8ead00 invalid freepointer: ffff07ffa3ab47e8
kmem: vm_area_struct: slab: fffffe1fbede8400 invalid freepointer: ffff07ffb7a10f78
ffff07ff81338200      176       6379     23322    507     8k  vm_area_struct
kmem: fs_cache: slab: fffffe1fbe640b40 invalid freepointer: ffff07ff9902dfd8
ffff07ff8133bd00       56        214      8576    134     4k  fs_cache
ffff07ff8133b700      704        196      6210    135    32k  files_cache
ffff07ff8133bc00     1112       1566      5180    185    32k  signal_cache
ffff07ff8133ac00     2080       1557      3510    234    32k  sighand_cache
kmem: task_struct: slab: fffffe1fbe60c600 invalid freepointer: ffff07ff98318a80
kmem: task_struct: slab: fffffe1fbe553e00 invalid freepointer: ffff07ff954f8a80
ffff07ff81338e00     5376       1584      2580    430    32k  task_struct
kmem: cred: slab: fffffe1fbe0fb800 invalid freepointer: ffff07ff83ee0718
kmem: cred: slab: fffffe1fbe1aae80 invalid freepointer: ffff07ff86abaf58
ffff07ff81339b00      184       1753      7560    180     8k  cred
kmem: anon_vma_chain: slab: fffffe1fbe8b1400 invalid freepointer: ffff07ffa2c50f60
kmem: anon_vma_chain: slab: fffffe1fbe2ebec0 invalid freepointer: ffff07ff8bafb3e0
kmem: anon_vma_chain: slab: fffffe1fbe138540 invalid freepointer: ffff07ff84e15860
kmem: anon_vma_chain: slab: fffffe1fbe13ad80 invalid freepointer: ffff07ff84eb6960
kmem: anon_vma_chain: slab: fffffe1fbe34fec0 invalid freepointer: ffff07ff8d3fb5e0
kmem: anon_vma_chain: slab: fffffe1fbeab3080 invalid freepointer: ffff07ffaacc2ba0
kmem: anon_vma_chain: slab: fffffe1fbe60b680 invalid freepointer: ffff07ff982dab20
kmem: anon_vma_chain: slab: fffffe1fbeb7b640 invalid freepointer: ffff07ffaded9620
kmem: anon_vma_chain: slab: fffffe1fbe8b7080 invalid freepointer: ffff07ffa2dc29e0
kmem: anon_vma_chain: slab: fffffe1fbe357700 invalid freepointer: ffff07ff8d5dce60
......
kmem: anon_vma_chain: slab: fffffe1fbe3b9bc0 invalid freepointer: ffff07ff8ee6fca0
kmem: anon_vma_chain: slab: fffffe1fbe511480 invalid freepointer: ffff07ff944520a0
kmem: anon_vma_chain: slab: fffffe1fbead6740 invalid freepointer: ffff07ffab59d9a0
kmem: anon_vma_chain: slab: fffffe1fbedebc00 invalid freepointer: ffff07ffb7af03a0
kmem: anon_vma_chain: slab: fffffe1fbe3bf880 invalid freepointer: ffff07ff8efe2020
kmem: anon_vma_chain: slab: fffffe1fbede7340 invalid freepointer: ffff07ffb79cd660
.....
kmem: anon_vma: slab: fffffe1fbee115c0 invalid freepointer: ffff07ffb8457950
kmem: anon_vma: slab: fffffe1fbedecf40 invalid freepointer: ffff07ffb7b3dbc0
kmem: anon_vma: slab: fffffe1fbee1a4c0 invalid freepointer: ffff07ffb8693a88
kmem: anon_vma: slab: fffffe1fbe0e4440 invalid freepointer: ffff07ff83911c90
kmem: anon_vma: slab: fffffe1fbea89380 invalid freepointer: ffff07ffaa24efd0
kmem: anon_vma: slab: fffffe1fbe1f3e80 invalid freepointer: ffff07ff87cfac90
kmem: anon_vma: slab: fffffe1fbeab3f80 invalid freepointer: ffff07ffaacfeaf0
kmem: anon_vma: slab: fffffe1fbe14b380 invalid freepointer: ffff07ff852ce818
ffff07ff81338a00       96       2098     23751    609     4k  anon_vma
kmem: pid: slab: fffffe1fbe1bd100 invalid freepointer: ffff07ff86f447c0
ffff07ff81339100      128       1508      9536    149     8k  pid
ffff07ff81339400       72       7275      7840    140     4k  Acpi-Operand
ffff07ff81338500       80          0       204      4     4k  Acpi-ParseExt
ffff07ff81338900       56          0       292      4     4k  Acpi-Parse
ffff07ff81339200       80          0       357      7     4k  Acpi-State
ffff07ff8133b900       48       1992      2295     27     4k  Acpi-Namespace
ffff07ff81338f00       48          0         0      0     4k  shared_policy_node
ffff07ff80007f00      144        128       168      3     8k  numa_policy
ffff07ff80007e00     1288          0         0      0    32k  perf_event
ffff07ff80007d00       96       2270      2352     56     4k  trace_event_file
ffff07ff80007c00       56       5068      5256     72     4k  ftrace_event_field
ffff07ff80007b00      512      17107     17728    277    32k  pool_workqueue
kmem: maple_node: slab: fffffe1fbea8cd00 invalid freepointer: ffff07ffaa334580
kmem: maple_node: slab: fffffe1fbf075b00 invalid freepointer: ffff07ffc1d6cc80
kmem: maple_node: slab: fffffe1fbe585b00 invalid freepointer: ffff07ff9616c480
kmem: maple_node: slab: fffffe1fbf316d00 invalid freepointer: ffff07ffcc5b4e80
kmem: maple_node: slab: fffffe1fbea67c00 invalid freepointer: ffff07ffa99f0b80
kmem: maple_node: slab: fffffe1fbe329200 invalid freepointer: ffff07ff8ca48180
kmem: maple_node: slab: fffffe1fbe214200 invalid freepointer: ffff07ff88508d80
kmem: maple_node: slab: fffffe1fbe38e900 invalid freepointer: ffff07ff8e3a4980
kmem: maple_node: slab: fffffe1fbe589100 invalid freepointer: ffff07ff96244780
kmem: maple_node: slab: fffffe1fbe599600 invalid freepointer: ffff07ff96658580
ffff07ff80007a00      256       3534     21760    340    16k  maple_node
kmem: radix_tree_node: slab: fffffe1fbe2f3e00 invalid freepointer: ffff07ff8bcf8b60
ffff07ff80007900      576       8173     12264    219    32k  radix_tree_node
ffff07ff80007800      640         62      2550     50    32k  task_group
ffff07ff80007700     1248        136      3475    139    32k  mm_struct
kmem: vmap_area: slab: fffffe1fbe026240 invalid freepointer: ffff07ff809890f8
ffff07ff80007600       72       6020     11760    210     4k  vmap_area
ffff07ff80007500     8192          0        32      8    32k  kmalloc-cg-8k
ffff07ff80007400     4096        161      1184    148    32k  kmalloc-cg-4k
kmem: kmalloc-cg-2k: slab: fffffe1fbe399800 invalid freepointer: ffff07ff8e660400
ffff07ff80007300     2048        457      2320    145    32k  kmalloc-cg-2k
ffff07ff80007200     1024        369      4416    138    32k  kmalloc-cg-1k
ffff07ff80007100      512        186      8320    130    32k  kmalloc-cg-512
ffff07ff80007000      256        538       896     14    16k  kmalloc-cg-256
ffff07ff80006f00      128        565      3200     50     8k  kmalloc-cg-128
kmem: kmalloc-cg-64: slab: fffffe1fbe366a40 invalid freepointer: ffff07ff8d9a99a0
ffff07ff80006e00       64        664      8512    133     4k  kmalloc-cg-64
ffff07ff80006d00       32       4271     19712    154     4k  kmalloc-cg-32
ffff07ff80006c00       16        620     10240     40     4k  kmalloc-cg-16
ffff07ff80006b00        8        291     23040     45     4k  kmalloc-cg-8
ffff07ff80006a00      192        571      5796    138     8k  kmalloc-cg-192
ffff07ff80006900       96       2567      7224    172     4k  kmalloc-cg-96
ffff07ff80006800     8192          0         0      0    32k  dma-kmalloc-8k
ffff07ff80006700     4096          0         0      0    32k  dma-kmalloc-4k
ffff07ff80006600     2048          0         0      0    32k  dma-kmalloc-2k
ffff07ff80006500     1024          0         0      0    32k  dma-kmalloc-1k
ffff07ff80006400      512          0         0      0    32k  dma-kmalloc-512
ffff07ff80006300      256          0         0      0    16k  dma-kmalloc-256
ffff07ff80006200      128          0         0      0     8k  dma-kmalloc-128
ffff07ff80006100       64          0         0      0     4k  dma-kmalloc-64
ffff07ff80006000       32          0         0      0     4k  dma-kmalloc-32
ffff07ff80005f00       16          0         0      0     4k  dma-kmalloc-16
ffff07ff80005e00        8          0         0      0     4k  dma-kmalloc-8
ffff07ff80005d00      192          0         0      0     8k  dma-kmalloc-192
ffff07ff80005c00       96          0         0      0     4k  dma-kmalloc-96
ffff07ff80005b00     8192          0         0      0    32k  kmalloc-rcl-8k
ffff07ff80005a00     4096          0         0      0    32k  kmalloc-rcl-4k
ffff07ff80005900     2048          0         0      0    32k  kmalloc-rcl-2k
ffff07ff80005800     1024          0         0      0    32k  kmalloc-rcl-1k
ffff07ff80005700      512          0         0      0    32k  kmalloc-rcl-512
ffff07ff80005600      256          0         0      0    16k  kmalloc-rcl-256
ffff07ff80005500      128         10       576      9     8k  kmalloc-rcl-128
ffff07ff80005400       64         90      3008     47     4k  kmalloc-rcl-64
ffff07ff80005300       32          0         0      0     4k  kmalloc-rcl-32
ffff07ff80005200       16          0         0      0     4k  kmalloc-rcl-16
ffff07ff80005100        8          0         0      0     4k  kmalloc-rcl-8
ffff07ff80005000      192          0         0      0     8k  kmalloc-rcl-192
ffff07ff80004f00       96        101      2058     49     4k  kmalloc-rcl-96
ffff07ff80004e00     8192         46       196     49    32k  kmalloc-8k
ffff07ff80004d00     4096        474      1344    168    32k  kmalloc-4k
ffff07ff80004c00     2048       1450      3824    239    32k  kmalloc-2k
kmem: kmalloc-1k: slab: fffffe1fbe51c200 invalid freepointer: ffff07ff94708a00
ffff07ff80004b00     1024       4231      7968    249    32k  kmalloc-1k
ffff07ff80004a00      512      23643     32704    511    32k  kmalloc-512
ffff07ff80004900      256       5202     12352    193    16k  kmalloc-256
ffff07ff80004800      128       3881     11968    187     8k  kmalloc-128
kmem: kmalloc-64: slab: fffffe1fbe095400 invalid freepointer: ffff07ff825502a0
kmem: kmalloc-64: slab: fffffe1fbe097380 invalid freepointer: ffff07ff825ce420
kmem: kmalloc-64: slab: fffffe1fbe10f840 invalid freepointer: ffff07ff843e10e0
kmem: kmalloc-64: slab: fffffe1fbee73f00 invalid freepointer: ffff07ffb9cfc1a0
ffff07ff80004700       64      19811     27648    432     4k  kmalloc-64
kmem: kmalloc-32: slab: fffffe1fbe0ab0c0 invalid freepointer: ffff07ff82ac3150
kmem: kmalloc-32: slab: fffffe1fbf49e640 invalid freepointer: ffff07ffd2799ab0
ffff07ff80004600       32      26379     39680    310     4k  kmalloc-32
ffff07ff80004500       16      40679     69120    270     4k  kmalloc-16
ffff07ff80004400        8      16113     77824    152     4k  kmalloc-8
kmem: kmalloc-192: slab: fffffe1fbebcf680 invalid freepointer: ffff07ffaf3da2a0
ffff07ff80004300      192       2927      8232    196     8k  kmalloc-192
kmem: kmalloc-96: slab: fffffe1fbe0ddd40 invalid freepointer: ffff07ff83775e70
kmem: kmalloc-96: slab: fffffe1fbe23ab40 invalid freepointer: ffff07ff88ead150
kmem: kmalloc-96: slab: fffffe1fbe23ab80 invalid freepointer: ffff07ff88eaeab0
kmem: kmalloc-96: slab: fffffe1fbe13e0c0 invalid freepointer: ffff07ff84f83ab0
kmem: kmalloc-96: slab: fffffe1fbe21cac0 invalid freepointer: ffff07ff8872b870
ffff07ff80004200       96       4273     14658    349     4k  kmalloc-96
ffff07ff80004100       64        268       768     12     4k  kmem_cache_node
ffff07ff80004000      224        267       768     12    16k  kmem_cache

beaker job: https://beaker.engineering.redhat.com/recipes/16323457#tasks
beaker job: https://beaker.engineering.redhat.com/recipes/16337771#tasks

aarch64 - crash on live system

# crash 
 crash 8.0.5-1.el10
Copyright (C) 2002-2024  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011, 2020-2024  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
Copyright (C) 2015, 2021  VMware, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.
 
NOTE: stdin: not a tty

GNU gdb (GDB) 10.2
Copyright (C) 2021 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "aarch64-unknown-linux-gnu".
Type "show configuration" for configuration details.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...

      KERNEL: /usr/lib/debug/lib/modules/6.10.0-0.rc2.10.el10.aarch64/vmlinux
    DUMPFILE: /proc/kcore
        CPUS: 4
        DATE: Thu Jun 13 03:17:21 EDT 2024
      UPTIME: 00:05:02
LOAD AVERAGE: 0.84, 0.63, 0.29
       TASKS: 258
    NODENAME: hpe-apollo-cn99xx-14-vm-11.khw.eng.rdu2.dc.redhat.com
     RELEASE: 6.10.0-0.rc2.10.el10.aarch64
     VERSION: #1 SMP PREEMPT_DYNAMIC Mon Jun 10 23:58:30 UTC 2024
     MACHINE: aarch64  (unknown Mhz)
      MEMORY: 6 GB
         PID: 6644
     COMMAND: "crash"
        TASK: ffff20b29a346900  [THREAD_INFO: ffff20b29a346900]
         CPU: 1
       STATE: TASK_RUNNING (ACTIVE)
crash> swap

swap: invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size
      FILE: memory.c  LINE: 16031  FUNCTION: dump_swap_info()

SWAP_INFO_STRUCT    TYPE       SIZE       USED     PCT  PRI  FILENAME
[/usr/bin/crash] error trace: aaaaaff45580 => aaaaafe312cc => aaaaafe755c0 => aaaaaff3c8b8

swap: invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size
      FILE: memory.c  LINE: 16031  FUNCTION: dump_swap_info()
crash> kmem -i

kmem: invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size
      FILE: memory.c  LINE: 16031  FUNCTION: dump_swap_info()

                 PAGES        TOTAL      PERCENTAGE
    TOTAL MEM  1429763       5.5 GB         ----
         FREE   308338       1.2 GB   21% of TOTAL MEM
         USED  1121425       4.3 GB   78% of TOTAL MEM
       SHARED   141215     551.6 MB    9% of TOTAL MEM
      BUFFERS     1061       4.1 MB    0% of TOTAL MEM
       CACHED   942885       3.6 GB   65% of TOTAL MEM
         SLAB    39356     153.7 MB    2% of TOTAL MEM

   TOTAL HUGE        0            0         ----
    HUGE FREE        0            0    0% of TOTAL HUGE

[/usr/bin/crash] error trace: aaaaafe56998 => aaaaafe5f6c0 => aaaaafe755c0 => aaaaaff3c8b8

kmem: invalid (optional) structure member offsets: swap_info_struct_swap_device or swap_info_struct_old_block_size
      FILE: memory.c  LINE: 16031  FUNCTION: dump_swap_info()

beaker job: https://beaker.engineering.redhat.com/recipes/16337771#tasks

Please provide the package NVR for which bug is seen:
RHEL-10.0-20240605.93
kernel-6.10.0-0.rc2.10.el10
kexec-tools-2.0.28-9.el10
crash 8.0.5-1.el10

How reproducible:
always

Steps to reproduce
install RHEL-10.0-20240605.93+kernel-6.10.0-0.rc2.10.el10
trigger panic to get a vmcore file
crash to against this vmcore file

links to

RHBA-2024:131448 kexec-tools bug fix and enhancement update

Details

Description

Attachments

Issue Links

Activity

People

Dates