虚拟机crash的一个案例,类似参考 vmcore分析案例:"kernel BUG at fs/buffer.c:1270" 使用crash分析
首先获取core对应虚拟机的内核版本
Copy sudo strings -n 50 2017-0411-0913.48-i-23ljwikin.301864.118695.897.core | head -n 30000 | grep -m 1 '^Linux version'
假设内核版本是 2.6.32-573.22.1.el6.x86_64
使用以下命令获取debuginfo包以获得对应的vmlinux
:
Copy export kernel_version=2.6.32-573.22.1.el6.x86_64
wget http://debuginfo.centos.org/6/x86_64/kernel-debuginfo-${kernel_version}.rpm
rpm2cpio kernel-debuginfo-${kernel_version}.rpm | cpio -idv ./usr/lib/debug/lib/modules/${kernel_version}/vmlinux
crash ./usr/lib/debug/lib/modules/${kernel_version}/vmlinux 2017-0411-0913.48-i-23ljwikin.301864.118695.897.core
可以看到导致kernel panic的原因是stack-protector: Kernel stack is corrupted
Copy KERNEL: vmlinux
DUMPFILE: 2017-0409-0418.11-AY1307061945475866c5.155086.33872.487.core
CPUS: 2 [OFFLINE: 1]
DATE: Sun Apr 9 04:18:10 2017
UPTIME: 17:13:25
LOAD AVERAGE: 2.00, 2.10, 2.48
TASKS: 155
NODENAME: kamiconf4
RELEASE: 2.6.32-279.el6.x86_64
VERSION: #1 SMP Fri Jun 22 12:19:21 UTC 2012
MACHINE: x86_64 (2194 Mhz)
MEMORY: 2 GB
PANIC: "Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ffffffffa016087c"
PID: 10522
COMMAND: "AliHids"
TASK: ffff88007c002080 [THREAD_INFO: ffff88000c8f8000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
使用sys
命令也可以显示上述信息
检查历史记录,每次crash的时候,出现的panic都是位于 stack-protector: Kernel stack is corrupted in: ffffffffa016087c
,内存地址都是ffffffffa016087c
使用dmesg
指令查看,可以看到
Not tainted 2.6.32-279.el6.x86_64
表明内核没有强制加载的内核模块(这里采用的是标准的CentOS 6.3系统,内核无污染)
Copy TCP: Peer 112.96.145.174:65008/8080 unexpectedly shrunk window 1283949404:1283949405 (repaired)
Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ffffffffa016087c
Pid: 10522, comm: AliHids Not tainted 2.6.32-279.el6.x86_64 #1
Call Trace:
[<ffffffff814fd11a>] ? panic+0xa0/0x168
[<ffffffff8127cdb0>] ? sprintf+0x40/0x50
[<ffffffff8106b85b>] ? __stack_chk_fail+0x1b/0x30
参考 Kernel Page Error 如果在Oops
部分看到decimal code of the Kernel Page Error
(4位数值,例如Oops: 0002
)
crash Kernel Page Error
注意位0到3是从右到左
0002 (dec) --> 0010 (binary) --> Not instruction fetch|Kernel mode|Write|Invalid access
反向跟踪
我们需要分析进程的执行历史,这里需要使用backtrace
,也就是bt
指令
Copy crash> bt
PID: 10522 TASK: ffff88007c002080 CPU: 0 COMMAND: "AliHids"
#0 [ffff88000c8f9d80] xen_panic_event at ffffffff810033c2
#1 [ffff88000c8f9da0] notifier_call_chain at ffffffff81503325
#2 [ffff88000c8f9de0] atomic_notifier_call_chain at ffffffff8150338a
#3 [ffff88000c8f9df0] panic at ffffffff814fd145
#4 [ffff88000c8f9e70] __stack_chk_fail at ffffffff8106b85b
RIP: 3e79656b2f3c656d RSP: 676e697274733c20 RFLAGS: 2020202020202020
RAX: 2020200a3e79656b RBX: 202020202020200a RCX: 2020202020202020
RDX: 0a3e746369643c20 RSI: 2020202020202020 RDI: 2020202020202020
RBP: 3e746369643c2020 R8: 2f3c676e702e305f R9: 6968736e6169715f
R10: 69746f6169623e79 R11: 656b3c2020202020 R12: 2020202020200a3e
R13: 79656b2f3c73656d R14: 6172663e79656b3c R15: 2020202020202020
ORIG_RAX: 6172663e79656b3c CS: 202020202020200a SS: 382c3233337b7b3e
bt: WARNING: possibly bogus exception frame
这里有一个BUG: 上述采用CentOS 5.11操作系统下编译crash最新版本,来分析bt
的时候,会出现bt: WARNING: possibly bogus exception frame
,参考 Bug 802234 - WARNING: possibly bogus exception frame ,原来这个BUG在CentOS 7上已经得到修复。
转换到CentOS 7平台,同样使用crash ./usr/lib/debug/lib/modules/2.6.32-279.el6.x86_64/vmlinux 2017-0326-0343.02-AY1307061945475866c5.155086.33872.482.core
,就不再出现告警,而输出的正确内容如下
Copy crash> bt
PID: 23983 TASK: ffff88007a9c2080 CPU: 0 COMMAND: "AliHids"
#0 [ffff88007be43d80] xen_panic_event at ffffffff810033c2
#1 [ffff88007be43da0] notifier_call_chain at ffffffff81503325
#2 [ffff88007be43de0] atomic_notifier_call_chain at ffffffff8150338a
#3 [ffff88007be43df0] panic at ffffffff814fd145
#4 [ffff88007be43e70] __stack_chk_fail at ffffffff8106b85b
RIP: 00007f5df958360d RSP: 00007f5df796d958 RFLAGS: 00000206
RAX: 000000000000c747 RBX: ff8b4d7544171b32 RCX: 00007f5df9fc82c0
RDX: 00000000000049d4 RSI: 0000000001ca72f8 RDI: 000000000000000f
RBP: 2f7b07899a13336f R8: 46c34549d6da7435 R9: c3694c0bf4cc3cc9
R10: 40e10a9dac1ffa53 R11: 23d56528e78dedf3 R12: 6a6ecd194c7425a2
R13: 4d852b9f03902eca R14: dd1390feac2b0f52 R15: 0c5079d804d98fa7
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
所以,要正确分析crash core dump,应该在最新的CentOS 7平台上使用crash来完成。否则,会明显看到寄存器内容读取错误,出现诡异的CS: 202020202020200a
。正确的CS
内容在CentOS 7平台可以解析出CS: 0033
Call trace
这里以#
开头的表示call trace,也就是先前crash时执行内核功能的列表,这里可以看出系统宕机前的迹象。
Copy #0 [ffff88007be43d80] xen_panic_event at ffffffff810033c2
#1 [ffff88007be43da0] notifier_call_chain at ffffffff81503325
#2 [ffff88007be43de0] atomic_notifier_call_chain at ffffffff8150338a
#3 [ffff88007be43df0] panic at ffffffff814fd145
#4 [ffff88007be43e70] __stack_chk_fail at ffffffff8106b85b
指令指针
RIP
表示指令指针(instruction point, IP),也就是指向内存地址,表示在内存中程序进程地址。
Copy RIP: 00007f5df958360d RSP: 00007f5df796d958 RFLAGS: 00000206
注意:在32位系统中,指令指针称为EIP
如果有类似 [exception RIP: default_idle+61]
则表示RIP
指向的内核功能是default_idle
,+61
是偏移量(十进制),也就是异常发生的位置。
代码段(Code Segment, CS)寄存器
在Call Trace之后的内容是寄存器内容输出,大多数没有太大用途。不过,有一个CS
(Code Segment)寄存器需要注意
Privilege levels(特权级别)概念
privilege level
是CPU的保护资源的概念。不同执行线程可以有不同的特权级别,以允许访问系统资源,类似内存区域,I/O端口,等等。有4个特权级别,从0到3。0
层是最高权限,也就是内核态;3
层则权限最低,也就是用户态。
大多数现代操作系统,包括linux,不使用中间2层,只使用0
和3
。这里的level
也称为ring
(环)。
当前特权层(Current Privilege Level, CPL)
Code Segment(CS) register是一个指向程序指令设置的段的指针。该寄存器2个最小的重要位指示了CPU的当前特权层(Current Privilege Level, CPL)。这两位代表的数值是0到3。
描述符特权级别(Descriptor Privilege Level, DPL)和 请求特权界别(Requested Privilege Level, RPL)
描述符特权级别(Descriptor Privilege Level, DPL)是被能够访问资源和被定义级别更高级别的权限。这个值在段描述符(Segement Descriptor)中定义。请求特权级别(RPL)是在段选择器(Segment Selector)中定义,至少2位。精确地来说,CPL是不允许超过最大值(RPL, DPL),如果超过,就会导致一般保护错误(General Protection Fault, GPF)。
注意
CPL
如果CPL
是3
的时候发生系统crash,则表示是由于硬件故障导致的,因为系统不可能由于用户态的错误导致crash。
相反,则可能是由于存在bug的系统调用导致的问题。
更多信息,请参考O'Reilly的Understanding Linux Kernel 第二章 "内存地址" 中有关 Segment Selectors, Segment Descriptors, Table Index, Global and Local Descriptor Tables , Current Privilege Level (CPL)等概念。
在这里的案例 CS: 0033
,表示
内核crash时的功能调用
前述案例中crash时执行内核功能的列表
Copy #0 [ffff88000c8f9d80] xen_panic_event at ffffffff810033c2
#1 [ffff88000c8f9da0] notifier_call_chain at ffffffff81503325
#2 [ffff88000c8f9de0] atomic_notifier_call_chain at ffffffff8150338a
#3 [ffff88000c8f9df0] panic at ffffffff814fd145
#4 [ffff88000c8f9e70] __stack_chk_fail at ffffffff8106b85b
可以看到功能调用顺序
Copy xen_panic_event => notifier_call_chain => atomic_notifier_call_chain
panic at ffffffff814fd145
__stack_chk_fail at ffffffff8106b85b
backtrace所有任务
默认情况下,crash智慧显示活跃任务的backtrace,如果要查看所有任务的backtrace,需要使用foreach
dump系统消息缓存
log
命令可以显示系统message buffer。在这个内核log buffer(log_buf
)有可能包含有用的crash信息。不过,如果发生了间歇性的硬件故障或者纯软件bug,这个log
命令可能没有什么帮助。
尝试了发现实际和dmesg
没有差别
显示进程状态信息
ps
命令可以显示进程状态信息:
Copy crash> ps
PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 20423 1 1 ffff88007b2f2080 RU 0.2 243724 3716 pamdicks
...
> 23983 1 0 ffff88007a9c2080 RU 0.4 95416 7756 AliHids
...
这里>
表示active task
注意:一定要使用正确的操作系统和crash,我遇到在CentOS 5上使用crash错误指向了active task
可以看到和bt
显示的一样 PID: 10522 TASK: ffff88007c002080 CPU: 0 COMMAND: "AliHids"
,这里可以从ffff88007c002080
任务地址看到AliHids
分析
完整显示backtrace
Copy crash> bt -f 23983
PID: 23983 TASK: ffff88007a9c2080 CPU: 0 COMMAND: "AliHids"
#0 [ffff88007be43d80] xen_panic_event at ffffffff810033c2
ffff88007be43d88: ffff880000000003 ffffffff8102b610
ffff88007be43d98: ffff88007be43dd8 ffffffff81503325
#1 [ffff88007be43da0] notifier_call_chain at ffffffff81503325
ffff88007be43da8: ffff88007be43db8 ffffffff81793c68
ffff88007be43db8: 0000000000000001 00000000000049d4
ffff88007be43dc8: 0000000001ca72f8 ffff88000f640000
ffff88007be43dd8: ffff88007be43de8 ffffffff8150338a
#2 [ffff88007be43de0] atomic_notifier_call_chain at ffffffff8150338a
ffff88007be43de8: ffff88007be43e68 ffffffff814fd145
#3 [ffff88007be43df0] panic at ffffffff814fd145
ffff88007be43df8: 0000000000000001 00000000000049d4
ffff88007be43e08: ffff880000000010 ffff88007be43e78
ffff88007be43e18: ffff88007be43e28 ffffffff8127cdb0
ffff88007be43e28: ffff880000000018 ffffffffa016087c
ffff88007be43e38: 05b3cfb3cd059c9a 0000000000000000
ffff88007be43e48: 0000000000000004 0000000001ca72f8
ffff88007be43e58: 0000000000000001 ffff88000f6400fd
ffff88007be43e68: ffff88007be43e78 ffffffff8106b85b
#4 [ffff88007be43e70] __stack_chk_fail at ffffffff8106b85b
RIP: 00007f5df958360d RSP: 00007f5df796d958 RFLAGS: 00000206
RAX: 000000000000c747 RBX: ff8b4d7544171b32 RCX: 00007f5df9fc82c0
RDX: 00000000000049d4 RSI: 0000000001ca72f8 RDI: 000000000000000f
RBP: 2f7b07899a13336f R8: 46c34549d6da7435 R9: c3694c0bf4cc3cc9
R10: 40e10a9dac1ffa53 R11: 23d56528e78dedf3 R12: 6a6ecd194c7425a2
R13: 4d852b9f03902eca R14: dd1390feac2b0f52 R15: 0c5079d804d98fa7
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
查看源代码
#4 [ffff88007be43e70] __stack_chk_fail at ffffffff8106b85b
Copy /usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 459
0xffffffff8106b840 <__stack_chk_fail>: push %rbp
0xffffffff8106b841 <__stack_chk_fail+1>: mov %rsp,%rbp
0xffffffff8106b844 <__stack_chk_fail+4>: nopl 0x0(%rax,%rax,1)
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 460
0xffffffff8106b849 <__stack_chk_fail+9>: mov 0x8(%rbp),%rsi
0xffffffff8106b84d <__stack_chk_fail+13>: mov $0xffffffff81793c68,%rdi
0xffffffff8106b854 <__stack_chk_fail+20>: xor %eax,%eax
0xffffffff8106b856 <__stack_chk_fail+22>: callq 0xffffffff814fd07a <panic>
dis -l (function+offset)
-- disassemble
这里可以看到源代码以及所在行
类似倒推,一共有4个函数需要检查
#3 [ffff88007be43df0] panic at ffffffff814fd145
Copy crash> dis -l panic
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 67
0xffffffff814fd07a <panic>: push %rbp
0xffffffff814fd07b <panic+1>: mov %rsp,%rbp
0xffffffff814fd07e <panic+4>: push %rbx
0xffffffff814fd07f <panic+5>: sub $0x68,%rsp
0xffffffff814fd083 <panic+9>: callq 0xffffffff8100adc0 <mcount>
0xffffffff814fd088 <panic+14>: mov %rdi,%rbx
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 83
0xffffffff814fd08b <panic+17>: mov $0xffffffff81e25940,%rdi
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 67
0xffffffff814fd092 <panic+24>: mov %rsi,-0x38(%rbp)
0xffffffff814fd096 <panic+28>: mov %rdx,-0x30(%rbp)
0xffffffff814fd09a <panic+32>: mov %rcx,-0x28(%rbp)
0xffffffff814fd09e <panic+36>: mov %r8,-0x20(%rbp)
0xffffffff814fd0a2 <panic+40>: mov %r9,-0x18(%rbp)
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 83
0xffffffff814fd0a6 <panic+44>: callq 0xffffffff814fff30 <__sched_text_end>
0xffffffff814fd0ab <panic+49>: test %eax,%eax
0xffffffff814fd0ad <panic+51>: jne 0xffffffff814fd0b4 <panic+58>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 84
0xffffffff814fd0af <panic+53>: callq 0xffffffff8106b350 <panic_smp_self_stop>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/include/linux/kernel.h: 358
0xffffffff814fd0b4 <panic+58>: cmpl $0x0,0x59a345(%rip) # 0xffffffff81a97400 <console_printk>
0xffffffff814fd0bb <panic+65>: je 0xffffffff814fd0c7 <panic+77>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/include/linux/kernel.h: 359
0xffffffff814fd0bd <panic+67>: movl $0xf,0x59a339(%rip) # 0xffffffff81a97400 <console_printk>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 87
0xffffffff814fd0c7 <panic+77>: mov $0x1,%edi
0xffffffff814fd0cc <panic+82>: callq 0xffffffff8127f420 <bust_spinlocks>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 88
0xffffffff814fd0d1 <panic+87>: lea 0x10(%rbp),%rax
0xffffffff814fd0d5 <panic+91>: lea -0x60(%rbp),%rcx
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 89
0xffffffff814fd0d9 <panic+95>: mov %rbx,%rdx
0xffffffff814fd0dc <panic+98>: mov $0x400,%esi
0xffffffff814fd0e1 <panic+103>: mov $0xffffffff81e25540,%rdi
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 88
0xffffffff814fd0e8 <panic+110>: movl $0x8,-0x60(%rbp)
0xffffffff814fd0ef <panic+117>: mov %rax,-0x58(%rbp)
0xffffffff814fd0f3 <panic+121>: lea -0x40(%rbp),%rax
0xffffffff814fd0f7 <panic+125>: mov %rax,-0x50(%rbp)
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 89
0xffffffff814fd0fb <panic+129>: callq 0xffffffff8127c780 <vsnprintf>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 91
0xffffffff814fd100 <panic+134>: mov $0xffffffff81e25540,%rsi
0xffffffff814fd107 <panic+141>: mov $0xffffffff81793c40,%rdi
0xffffffff814fd10e <panic+148>: xor %eax,%eax
0xffffffff814fd110 <panic+150>: callq 0xffffffff814fd1e2 <printk>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 93
0xffffffff814fd115 <panic+155>: callq 0xffffffff814fd004 <dump_stack>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 101
0xffffffff814fd11a <panic+160>: xor %edi,%edi
0xffffffff814fd11c <panic+162>: callq 0xffffffff810ba5f0 <crash_kexec>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 103
0xffffffff814fd121 <panic+167>: xor %edi,%edi
0xffffffff814fd123 <panic+169>: callq 0xffffffff8106c930 <kmsg_dump>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/arch/x86/include/asm/smp.h: 85
0xffffffff814fd128 <panic+174>: xor %edi,%edi
0xffffffff814fd12a <panic+176>: callq *0x595e48(%rip) # 0xffffffff81a92f78 <smp_ops+24>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 112
0xffffffff814fd130 <panic+182>: xor %esi,%esi
0xffffffff814fd132 <panic+184>: mov $0xffffffff81e25540,%rdx
0xffffffff814fd139 <panic+191>: mov $0xffffffff81e254c0,%rdi
0xffffffff814fd140 <panic+198>: callq 0xffffffff81503370 <atomic_notifier_call_chain>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 114
0xffffffff814fd145 <panic+203>: xor %edi,%edi
0xffffffff814fd147 <panic+205>: callq 0xffffffff8127f420 <bust_spinlocks>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 116
0xffffffff814fd14c <panic+210>: cmpq $0x0,0x92837c(%rip) # 0xffffffff81e254d0 <panic_blink>
0xffffffff814fd154 <panic+218>: jne 0xffffffff814fd161 <panic+231>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 117
0xffffffff814fd156 <panic+220>: movq $0xffffffff8106b340,0x92836f(%rip) # 0xffffffff81e254d0 <panic_blink>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 119
0xffffffff814fd161 <panic+231>: mov 0x928371(%rip),%esi # 0xffffffff81e254d8 <panic_timeout>
0xffffffff814fd167 <panic+237>: test %esi,%esi
0xffffffff814fd169 <panic+239>: jle 0xffffffff814fd1b8 <panic+318>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 124
0xffffffff814fd16b <panic+241>: mov $0xffffffff8178cf76,%rdi
0xffffffff814fd172 <panic+248>: xor %eax,%eax
0xffffffff814fd174 <panic+250>: xor %ebx,%ebx
0xffffffff814fd176 <panic+252>: callq 0xffffffff814fd1e2 <printk>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 126
0xffffffff814fd17b <panic+257>: jmp 0xffffffff814fd1a2 <panic+296>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 127
0xffffffff814fd17d <panic+259>: callq 0xffffffff810db560 <touch_nmi_watchdog>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 128
0xffffffff814fd182 <panic+264>: mov %rbx,%rdi
0xffffffff814fd185 <panic+267>: callq *0x928345(%rip) # 0xffffffff81e254d0 <panic_blink>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 129
0xffffffff814fd18b <panic+273>: mov $0x418958,%edi
0xffffffff814fd190 <panic+278>: mov %rax,-0x68(%rbp)
0xffffffff814fd194 <panic+282>: callq 0xffffffff8127d440 <__const_udelay>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 130
0xffffffff814fd199 <panic+287>: mov -0x68(%rbp),%rax
0xffffffff814fd19d <panic+291>: lea 0x1(%rbx,%rax,1),%rbx
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 126
0xffffffff814fd1a2 <panic+296>: imul $0x3e8,0x92832c(%rip),%eax # 0xffffffff81e254d8 <panic_timeout>
0xffffffff814fd1ac <panic+306>: cltq
0xffffffff814fd1ae <panic+308>: cmp %rbx,%rax
0xffffffff814fd1b1 <panic+311>: jg 0xffffffff814fd17d <panic+259>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 137
0xffffffff814fd1b3 <panic+313>: callq 0xffffffff8108a770 <emergency_restart>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/arch/x86/include/asm/paravirt.h: 885
0xffffffff814fd1b8 <panic+318>: sti
0xffffffff814fd1b9 <panic+319>: nopw 0x0(%rax,%rax,1)
0xffffffff814fd1bf <panic+325>: xor %ebx,%ebx
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 157
0xffffffff814fd1c1 <panic+327>: callq 0xffffffff810dacf0 <touch_softlockup_watchdog>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 158
0xffffffff814fd1c6 <panic+332>: mov %rbx,%rdi
0xffffffff814fd1c9 <panic+335>: callq *0x928301(%rip) # 0xffffffff81e254d0 <panic_blink>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 159
0xffffffff814fd1cf <panic+341>: mov $0x418958,%edi
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 158
0xffffffff814fd1d4 <panic+346>: add %rax,%rbx
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 160
0xffffffff814fd1d7 <panic+349>: add $0x1,%rbx
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/panic.c: 159
0xffffffff814fd1db <panic+353>: callq 0xffffffff8127d440 <__const_udelay>
0xffffffff814fd1e0 <panic+358>: jmp 0xffffffff814fd1c1 <panic+327>
[ffff88007be43de0] atomic_notifier_call_chain at ffffffff8150338a
Copy crash> dis -l atomic_notifier_call_chain
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/notifier.c: 190
0xffffffff81503370 <atomic_notifier_call_chain>: push %rbp
0xffffffff81503371 <atomic_notifier_call_chain+1>: mov %rsp,%rbp
0xffffffff81503374 <atomic_notifier_call_chain+4>: callq 0xffffffff8100adc0 <mcount>
0xffffffff81503379 <atomic_notifier_call_chain+9>: xor %r8d,%r8d
0xffffffff8150337c <atomic_notifier_call_chain+12>: mov $0xffffffff,%ecx
0xffffffff81503381 <atomic_notifier_call_chain+17>: add $0x8,%rdi
0xffffffff81503385 <atomic_notifier_call_chain+21>: callq 0xffffffff815032d0 <notifier_call_chain>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/notifier.c: 192
0xffffffff8150338a <atomic_notifier_call_chain+26>: leaveq
0xffffffff8150338b <atomic_notifier_call_chain+27>: retq
[ffff88007be43da0] notifier_call_chain at ffffffff81503325
Copy crash> dis -l atomic_notifier_call_chain
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/notifier.c: 190
0xffffffff81503370 <atomic_notifier_call_chain>: push %rbp
0xffffffff81503371 <atomic_notifier_call_chain+1>: mov %rsp,%rbp
0xffffffff81503374 <atomic_notifier_call_chain+4>: callq 0xffffffff8100adc0 <mcount>
0xffffffff81503379 <atomic_notifier_call_chain+9>: xor %r8d,%r8d
0xffffffff8150337c <atomic_notifier_call_chain+12>: mov $0xffffffff,%ecx
0xffffffff81503381 <atomic_notifier_call_chain+17>: add $0x8,%rdi
0xffffffff81503385 <atomic_notifier_call_chain+21>: callq 0xffffffff815032d0 <notifier_call_chain>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/kernel/notifier.c: 192
0xffffffff8150338a <atomic_notifier_call_chain+26>: leaveq
0xffffffff8150338b <atomic_notifier_call_chain+27>: retq
#0 [ffff88007be43d80] xen_panic_event at ffffffff810033c2
Copy crash> dis -l xen_panic_event
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/arch/x86/xen/enlighten.c: 1076
0xffffffff810033a0 <xen_panic_event>: push %rbp
0xffffffff810033a1 <xen_panic_event+1>: mov %rsp,%rbp
0xffffffff810033a4 <xen_panic_event+4>: sub $0x10,%rsp
0xffffffff810033a8 <xen_panic_event+8>: nopl 0x0(%rax,%rax,1)
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/arch/x86/include/asm/xen/hypercall.h: 273
0xffffffff810033ad <xen_panic_event+13>: mov $0x2,%edi
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/arch/x86/xen/enlighten.c: 1077
0xffffffff810033b2 <xen_panic_event+18>: movl $0x3,-0x10(%rbp)
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/arch/x86/include/asm/xen/hypercall.h: 273
0xffffffff810033b9 <xen_panic_event+25>: lea -0x10(%rbp),%rsi
0xffffffff810033bd <xen_panic_event+29>: callq 0xffffffff810013a0 <hypercall_page+928>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/arch/x86/xen/enlighten.c: 1079
0xffffffff810033c2 <xen_panic_event+34>: test %eax,%eax
0xffffffff810033c4 <xen_panic_event+36>: je 0xffffffff810033ca <xen_panic_event+42>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/arch/x86/xen/enlighten.c: 1080
0xffffffff810033c6 <xen_panic_event+38>: ud2
0xffffffff810033c8 <xen_panic_event+40>: jmp 0xffffffff810033c8 <xen_panic_event+40>
/usr/src/debug/kernel-2.6.32-279.el6/linux-2.6.32-279.el6.x86_64/arch/x86/xen/enlighten.c: 1082
0xffffffff810033ca <xen_panic_event+42>: xor %eax,%eax
0xffffffff810033cc <xen_panic_event+44>: leaveq
0xffffffff810033cd <xen_panic_event+45>: retq
扩展ps
和files
可以看到
Copy PID PPID CPU TASK ST %MEM VSZ RSS COMM
> 23983 1 0 ffff88007a9c2080 RU 0.4 95416 7756 AliHids
Copy crash> file ffff88007a9c2080
struct file {
f_u = {
fu_list = {
next = 0x0,
prev = 0xffff88007be42000
},
fu_rcuhead = {
next = 0x0,
func = 0xffff88007be42000
}
},
f_path = {
mnt = 0x40214000000002,
dentry = 0xffffffff00000000
},
f_op = 0x7800000078,
f_lock = {
raw_lock = {
slock = 120
}
},
f_count = {
counter = -2124369120
},
f_flags = 1024,
f_mode = 0,
f_pos = 4194304,
f_owner = {
lock = {
raw_lock = {
lock = 1
}
},
pid = 0x0,
pid_type = PIDTYPE_PID,
uid = 0,
euid = 35743528,
signum = -30720
},
f_cred = 0xffff880037708b00,
f_ra = {
start = 1,
size = 11186381,
async_size = 156034,
ra_pages = 1415622133,
mmap_miss = 3,
prev_pos = 39654903394240
},
f_version = 14245521487,
f_security = 0x0,
private_data = 0x0,
f_ep_links = {
next = 0x23,
prev = 0x0
},
f_mapping = 0x0
}
Copy crash> set ffff88007a9c2080
PID: 23983
COMMAND: "AliHids"
TASK: ffff88007a9c2080 [THREAD_INFO: ffff88007be42000]
CPU: 0
STATE: TASK_RUNNING (PANIC)
内核源代码获取
参考