本次性能調(diào)優(yōu)項目中由于涉及的環(huán)節(jié)較多，最好能夠?qū)⑸森h(huán)境中的所有內(nèi)容進(jìn)行監(jiān)控，同時考慮最低開銷，這樣就從應(yīng)用服務(wù)器和數(shù)據(jù)庫服務(wù)器兩個服務(wù)器進(jìn)行，以nmon作為監(jiān)控基礎(chǔ)數(shù)據(jù)，同時監(jiān)控JVM和數(shù)據(jù)庫告警和快照。
所有監(jiān)控的內(nèi)容都是手段，只有從海量的監(jiān)控日志中得到規(guī)律性、有意義的數(shù)據(jù)才是性能優(yōu)化的基礎(chǔ)。有了數(shù)據(jù)就是對數(shù)據(jù)的分析，本文將首先介紹需要獲取的數(shù)據(jù)，內(nèi)容也將是我從項目獲取的經(jīng)驗。
基礎(chǔ)環(huán)境：兩臺數(shù)據(jù)庫服務(wù)器，做的數(shù)據(jù)庫集群。
應(yīng)用服務(wù)器 - JVM線程
項目中主要使用tongweb（老系統(tǒng)版本很低），監(jiān)控內(nèi)容類似如下：
監(jiān)控內(nèi)容
...
"2018-01-11T02:25:55.663+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnCreated","10",
"2018-01-11T02:25:55.663+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnAcquired","111292",
"2018-01-11T02:25:55.663+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnNotSuccessfullyMatched","0",
"2018-01-11T02:26:25.670+0800","com.tongtech.tongweb:type=jvm,category=monitor,server=server","UpTime","222520621",
"2018-01-11T02:26:25.670+0800","com.tongtech.tongweb:type=jvm,category=monitor,server=server","HeapSize","2143485952",
"2018-01-11T02:26:25.671+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnUsed","0",
"2018-01-11T02:26:25.671+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnSuccessfullyMatched","0",
"2018-01-11T02:26:25.671+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","WaitQueueLength","0",
"2018-01-11T02:26:25.671+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnDestroyed","0",
"2018-01-11T02:26:25.671+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","ConnRequestWaitTime","4",
"2018-01-11T02:26:25.672+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnFailedValidation","0",
"2018-01-11T02:26:25.672+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnReleased","111292",
"2018-01-11T02:26:25.672+0800","com.tongtech.tongweb:name=***,type=jdbc-connection-pool,category=monitor,server=server","NumConnFree","10",
...
關(guān)注內(nèi)容
tongweb的監(jiān)控數(shù)據(jù)獲取連接池狀態(tài)等信息，我們的方法是通過Excel宏的方式將日志內(nèi)轉(zhuǎn)換成可讀數(shù)據(jù)，并進(jìn)行圖形分析。具體內(nèi)容將單獨(dú)說明。
JVM線程監(jiān)控說明
監(jiān)控意義
通過對tongweb的JVM監(jiān)控，可初步判定性能高峰時間點、連接池是否滿，同時進(jìn)一步判定連接高峰期的性能瓶頸是否出現(xiàn)在應(yīng)用上，這對今后的性能分析尤為重要，可將主要性能問題歸類，減少不必要的工作。
應(yīng)用服務(wù)器 - netstat
在Internet RFC標(biāo)準(zhǔn)中，Netstat的定義是： Netstat是在內(nèi)核中訪問網(wǎng)絡(luò)連接狀態(tài)及其相關(guān)信息的程序，它能提供TCP連接，TCP和UDP監(jiān)聽，進(jìn)程內(nèi)存管理的相關(guān)報告。
監(jiān)控內(nèi)容
以下是在項目中獲取的日志摘取
...
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 0.0.0.0:2049 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:139 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:427 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:427 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:58862 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:2544 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:21 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:631 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:445 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:669 0.0.0.0:* LISTEN
...
應(yīng)用服務(wù)器 - nmon
作為本次性能優(yōu)化主要的分析手段，nmon起著尤為重要的作用，以下是wiki的解釋，有時間可以了解
nmon collects the following operating system statistics:
CPU and CPU threads Utilisation
CPU frequency for servers or virtual machines that can alter their clock rate
GPU stats including utilisation, MHz and temperatures
Physical and Virtual Memory use
Disk read & write and transfers
Disk Groups decided by the user
Swap and Paging
Network read & write and transfers
Local File-systems
Network File-system (NFS)
Top Processes by CPU use, Memory size and I/O rates
Kernel stats including Run Queue, context-switch, fork, Load Average & Uptime
Large and Huge memory pages
Virtual Machine stats (depending on the hardware) - useful for Linux running KVM to host virtual machines
Resources in the Server and virtual machine
總結(jié)其實nmon更像是系統(tǒng)性能開銷的快照，結(jié)合對nmon的分析工具可以很清楚的掌握系統(tǒng)的各項指標(biāo)。
下載分析工具
數(shù)據(jù)庫服務(wù)器 - 告警
了解數(shù)據(jù)庫的告警日志也是掌握當(dāng)前性能的關(guān)鍵環(huán)節(jié)。
日志如下，如出現(xiàn)error可以針對具體情況進(jìn)行分析解決。
2018-01-11-00.36.36.090562+480 I13363168A459 LEVEL: Error
PID : 2228842 TID : 142490 PROC : db2sysc
INSTANCE: db2 NODE : 000 DB : TRADE
EDUID : 142490 EDUNAME: db2agent (**) 0
FUNCTION: DB2 UDB, Query Gateway, sqlqg_fedstp_hook, probe:40
MESSAGE : Unexpected error returned from outer RC=
DATA #1 : Hexdump, 4 bytes
0x07000007053F28D0 : 8126 0012 .&..
數(shù)據(jù)庫服務(wù)器 - 快照
數(shù)據(jù)庫日志快照將作為主要分析依據(jù)，在快照中可以分析數(shù)據(jù)庫時間的開銷情況，如下：
...
Number of automatic storage paths = 1
Automatic storage path = /db2data
Node number = 0
State = In Use
File system ID = 9223372079804448776
Storage path free space (bytes) = 69730709504
File system used space (bytes) = 139648946176
File system total space (bytes) = 209379655680
...
本文只是列出了分析的方法，具體操作有時間我會慢慢總結(jié)。
工具的利用固然重要，但是性能調(diào)優(yōu)并不是僅僅如此，必須步步為營做好長期作戰(zhàn)的準(zhǔn)備。
