1.用top命令查看哪個(gè)進(jìn)程占用CPU高
gateway網(wǎng)關(guān)進(jìn)程14094占用CPU高達(dá)891%，這個(gè)數(shù)值是進(jìn)程內(nèi)各個(gè)線程占用CPU的累加值。
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14094 root 15 0 315m 10m 7308 S 891% 2.2 1:49.01 gateway
20642 root 17 0 17784 4148 2220 S 0.5 0.8 2:39.96 microdasys
1679 root 18 0 10984 1856 1556 R 0.3 0.4 0:22.21 sshd
22563 root 18 0 2424 1060 800 R 0.3 0.2 0:00.03 top
1 root 18 0 2156 492 460 S 0.0 0.1 0:01.59 init
2.用top -H -p pid命令查看進(jìn)程內(nèi)各個(gè)線程占用的CPU百分比
top -H -p 14094
top中可以看到有107個(gè)線程，但是下面9個(gè)線程占用CPU很高，下面以線程14086為主，分析其為何high CPU
PID USER PR NI VIRT RES SHR S %CPU MEM TIME+ COMMAND
14086 root 25 0 922m 914m 538m R 101 10.0 21:35.46 gateway
14087 root 25 0 922m 914m 538m R 101 10.0 10:50.22 gateway
14081 root 25 0 922m 914m 538m S 99 10.0 8:57.36 gateway
14082 root 25 0 922m 914m 538m R 99 10.0 11:51.92 gateway
14089 root 25 0 922m 914m 538m R 99 10.0 21:21.77 gateway
14092 root 25 0 922m 914m 538m R 99 10.0 19:55.47 gateway
14094 root 25 0 922m 914m 538m R 99 10.0 21:02.21 gateway
14083 root 25 0 922m 914m 538m R 97 10.0 21:32.39 gateway
14088 root 25 0 922m 914m 538m R 97 10.0 11:23.12 gateway
3.使用gstack命令查看進(jìn)程中各線程的函數(shù)調(diào)用棧
gstack 14094 > gstack.log
在gstack.log中查找線程ID14086，由于函數(shù)棧會(huì)暴露函數(shù)細(xì)節(jié)，因此只顯示了兩個(gè)函數(shù)楨，線程ID14086對(duì)應(yīng)線程號(hào)是37
Thread 37 (Thread 0x4696ab90 (LWP 14086)):
0 0x40000410 in __kernel_vsyscall ()
1 0x40241f33 in poll () from /lib/i686/nosegneg/libc.so.6
4.使用gcore命令轉(zhuǎn)存進(jìn)程映像及內(nèi)存上下文
gcore 14094
該命令生成core文件core.14094
5。用strace命令查看系統(tǒng)調(diào)用和花費(fèi)的時(shí)間
strace -T -r -c -p 14094
-c參數(shù)顯示統(tǒng)計(jì)信息，去掉此參數(shù)可以查看每個(gè)系統(tǒng)調(diào)用話費(fèi)的時(shí)間及返回值。
% time seconds usecs/call calls errors syscall
99.99 22.683879 3385 6702 poll
0.00 0.001132 0 6702 gettimeofday
0.00 0.000127 1 208 208 accept
0.00 0.000022 22 1 read
0.00 0.000000 0 1 write
0.00 0.000000 0 1 close
0.00 0.000000 0 14 time
0.00 0.000000 0 2 stat64
0.00 0.000000 0 4 clock_gettime
0.00 0.000000 0 7 send
0.00 0.000000 0 10 10 recvfrom
100.00 22.685160 13652 218 total
6.用gdb調(diào)試core文件，并線程切換到37號(hào)線程
gcore和實(shí)際的core dump時(shí)產(chǎn)生的core文件幾乎一樣，只是不能用gdb進(jìn)行某些動(dòng)態(tài)調(diào)試
(gdb) gdb gateway core.14094
(gdb) thread 37
[Switching to thread 37 (Thread 0x4696ab90 (LWP 14086))]#0 0x40000410 in __kernel_vsyscall ()
(gdb) where
0 0x40000410 in __kernel_vsyscall ()
1 0x40241f33 in poll () from /lib/i686/nosegneg/libc.so.6
可以根據(jù)詳細(xì)的函數(shù)棧進(jìn)行g(shù)db調(diào)試，打印一些變量值，并結(jié)合源代碼分析為何會(huì)poll調(diào)用占用很高的CPU。
因?yàn)榇a涉及到公司產(chǎn)權(quán)，顧不在此做詳細(xì)分析，需要明白的是分析的流程和使用的命令。
流程為：進(jìn)程ID->線程ID->線程函數(shù)調(diào)用棧->函數(shù)耗時(shí)和調(diào)用統(tǒng)計(jì)->源代碼分析
