zabbix監(jiān)控 zookeeper

線上zookeeper集群越來多,很有必要對其運行狀態(tài)進(jìn)行監(jiān)控,由于線上監(jiān)控大多采用的是zabbix,因此,只需要寫好采集數(shù)據(jù)的腳本即可,不用關(guān)心web端的展示和監(jiān)控報警,比較方便。
1.導(dǎo)入zookeeper模板,模板內(nèi)容在最后,復(fù)制粘貼保存為zookeeper.xml即可
點擊template/import,選擇文件,導(dǎo)入,如果報錯,會提示哪一行有問題,刪除對應(yīng)的那一塊,再導(dǎo)入即可。


Paste_Image.png

2.導(dǎo)入后,在template頁面就可以看到了,然后添加相應(yīng)的host。

Paste_Image.png

3.模板中item的采集數(shù)據(jù)方式為zabbix trapper。也就是通過zabbix_sender來發(fā)送數(shù)據(jù)。關(guān)于zabbix_trapper,請參考https://www.2goo.info/weblog/detail/233434

4.測試數(shù)據(jù)的采集
利用zabbix_sender命令,參數(shù)介紹:
-z: zabbix_server的IP地址
-s: zabbix_agent的HostName名
-k: 模板里key的名字
-o:key對應(yīng)的value
-vv: 開啟debug模式,輸出詳細(xì)信息

bin/zabbix_sender -z 10.10.64.202 -s "oss-mon-kafka-bjc-001" -k "zookeeper.status[zk_num_alive_connections]" -o 100 -vv
Paste_Image.png

發(fā)送成功后,在頁面端點擊monitor/latest data/,然后篩選對應(yīng)的HostName和application,就能查看到對應(yīng)的value


Paste_Image.png

5.采集發(fā)送打通后,接下來就要編寫采集腳本,然后定時發(fā)送。采集腳本采用python,如下:

#!/usr/bin/python  
#coding:utf-8
""" Check Zookeeper Cluster  

zookeeper version should be newer than 3.4.x  

# echo mntr|nc 127.0.0.1 2181  
zk_version  3.4.6-1569965, built on 02/20/2014 09:09 GMT  
zk_avg_latency  0  
zk_max_latency  4  
zk_min_latency  0  
zk_packets_received 84467  
zk_packets_sent 84466  
zk_num_alive_connections    3  
zk_outstanding_requests 0  
zk_server_state follower  
zk_znode_count  17159  
zk_watch_count  2  
zk_ephemerals_count 1  
zk_approximate_data_size    6666471  
zk_open_file_descriptor_count   29  
zk_max_file_descriptor_count    102400  

# echo ruok|nc 127.0.0.1 2181  
imok  

"""  
import sys  
import socket  
import re  
import subprocess  
from StringIO import StringIO  
import os  

#這兩個參數(shù)依據(jù)你安裝zabbix目錄而定。
zabbix_sender = '/usr/bin/zabbix_sender'  
zabbix_conf = '/etc/zabbix/zabbix_agentd.conf'  
send_to_zabbix = 1  

############# get zookeeper server status  
class ZooKeeperServer(object):  

    def __init__(self, host='localhost', port='2181', timeout=1):  
        self._address = (host, int(port))  
        self._timeout = timeout  
        self._result  = {}  

    def _create_socket(self):  
        return socket.socket()  

    def _send_cmd(self, cmd):  
        """ Send a 4letter word command to the server """  
        s = self._create_socket()  
        s.settimeout(self._timeout)  
        s.connect(self._address)  
        s.send(cmd)  
        data = s.recv(2048)  
        s.close()  
        return data  

    def get_stats(self):  
        """ Get ZooKeeper server stats as a map """  
        data_mntr = self._send_cmd('mntr')  
        data_ruok = self._send_cmd('ruok')  
        if data_mntr:  
            result_mntr = self._parse(data_mntr)  
        if data_ruok:  
            result_ruok = self._parse_ruok(data_ruok)  

        self._result = dict(result_mntr.items() + result_ruok.items())  

        if not self._result.has_key('zk_followers') and not self._result.has_key('zk_synced_followers') and not self._result.has_key('zk_pending_syncs'):  

           ##### the tree metrics only exposed on leader role zookeeper server, we just set the followers' to 0  
           leader_only = {'zk_followers':0,'zk_synced_followers':0,'zk_pending_syncs':0}      
           self._result = dict(result_mntr.items() + result_ruok.items() + leader_only.items() )  

        return self._result    

    def _parse(self, data):  
        """ Parse the output from the 'mntr' 4letter word command """  
        h = StringIO(data)  

        result = {}  
        for line in h.readlines():  
            try:  
                key, value = self._parse_line(line)  
                result[key] = value  
            except ValueError:  
                pass # ignore broken lines  

        return result  

    def _parse_ruok(self, data):  
        """ Parse the output from the 'ruok' 4letter word command """  

        h = StringIO(data)  

        result = {}  

        ruok = h.readline()  
        if ruok:  
           result['zk_server_ruok'] = ruok  

        return result  

    def _parse_line(self, line):  
        try:  
            key, value = map(str.strip, line.split('\t'))  
        except ValueError:  
            raise ValueError('Found invalid line: %s' % line)  

        if not key:  
            raise ValueError('The key is mandatory and should not be empty')  

        try:  
            value = int(value)  
        except (TypeError, ValueError):  
            pass  

        return key, value  

    def get_pid(self):  
#  ps -ef|grep java|grep zookeeper|awk '{print $2}'  
         pidarg = '''ps -ef|grep java|grep zookeeper|grep -v grep|awk '{print $2}' '''   
         pidout = subprocess.Popen(pidarg,shell=True,stdout=subprocess.PIPE)  
         pid = pidout.stdout.readline().strip('\n')  
         return pid  


    def send_to_zabbix(self, metric):  
         key = "zookeeper.status[" +  metric + "]"  

         if send_to_zabbix > 0:  
             #print key + ":" + str(self._result[metric])  
             try:  

                subprocess.call([zabbix_sender, "-c", zabbix_conf, "-k", key, "-o", str(self._result[metric]) ], stdout=FNULL, stderr=FNULL, shell=False)  
             except OSError, detail:  
                print "Something went wrong while exectuting zabbix_sender : ", detail  
         else:  
                print "Simulation: the following command would be execucted :\n", zabbix_sender, "-c", zabbix_conf, "-k", key, "-o", self._result[metric], "\n"  

def usage():  
        """Display program usage"""  

        print "\nUsage : ", sys.argv[0], " alive|all"  
        print "Modes : \n\talive : Return pid of running zookeeper\n\tall : Send zookeeper stats as well"  
        sys.exit(1)  

accepted_modes = ['alive', 'all']  

if len(sys.argv) == 2 and sys.argv[1] in accepted_modes:  
        mode = sys.argv[1]  
else:  
        usage()  

zk = ZooKeeperServer()  
#  print zk.get_stats()  
pid = zk.get_pid()  

if pid != "" and  mode == 'all':  
   zk.get_stats()  
   # print zk._result  
   FNULL = open(os.devnull, 'w')  
   for key in zk._result:  
       zk.send_to_zabbix(key)  
   FNULL.close()  
   print pid  

elif pid != "" and mode == "alive":  
    print pid  
else:  
    print 0

執(zhí)行結(jié)果:

#獲取zookeeper的所有key的值
[root@oss-mon-kafka-bjc-001 scripts]# python check_zookeeper.py all
{'zk_followers': 0, 'zk_outstanding_requests': 0, 'zk_approximate_data_size': 50747, 'zk_packets_sent': 31693631, 'zk_pending_syncs': 0, 'zk_avg_latency': 0, 'zk_version': '3.4.6-1569965, built on 02/20/2014 09:09 GMT', 'zk_watch_count': 47, 'zk_packets_received': 31693282, 'zk_open_file_descriptor_count': 33, 'zk_server_ruok': 'imok', 'zk_server_state': 'follower', 'zk_synced_followers': 0, 'zk_max_latency': 1643, 'zk_num_alive_connections': 6, 'zk_min_latency': 0, 'zk_ephemerals_count': 83, 'zk_znode_count': 700, 'zk_max_file_descriptor_count': 655350}
2750
#獲取zookeeper的pid
[root@oss-mon-kafka-bjc-001 scripts]# python check_zookeeper.py alive
2750

6.配置zabbix的采集配置策略
增加腳本可執(zhí)行權(quán)限
chmod +x /usr/local/zabbix-agent/scripts/check_zookeeper.py
zabbix配置文件
vim /etc/zabbix/zabbix_agentd.d/check_zookeeper.conf
增加以下內(nèi)容
UserParameter=zookeeper.status[*],/usr/bin/python /usr/local/zabbix-agent/scripts/check_zookeeper.py $1
重新啟動zabbix-agent服務(wù)
service zabbix-agent restart

7.web端觀察采集數(shù)據(jù)是否正常,如果沒有數(shù)據(jù),則需調(diào)試check_zookeeper.py,看問題在哪。

Paste_Image.png

最后附上zookeeper的監(jiān)控模板,比較長,簡書貌似不支持附件,只能貼上來了。

<?xml version="1.0" encoding="UTF-8"?>  
<zabbix_export>  
    <version>2.0</version>  
    <date>2016-02-27T15:15:09Z</date>  
    <groups>  
        <group>  
            <name>Templates</name>  
        </group>  
    </groups>  
    <templates>  
        <template>  
            <template>Template ZooKeeper</template>  
            <name>Template ZooKeeper</name>  
            <description/>  
            <groups>  
                <group>  
                    <name>Templates</name>  
                </group>  
            </groups>  
            <applications>  
                <application>  
                    <name>ZooKeeper Status</name>  
                </application>  
            </applications>  
            <items>  
                <item>  
                    <name>zookeeper alive connections</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_num_alive_connections]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper approximate data size</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_approximate_data_size]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units>B</units>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper average latency</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_avg_latency]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units>tick</units>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper ephemerals count</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_ephemerals_count]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper leader's followers</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_followers]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper leader's pending syncs</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_pending_syncs]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper leader's synced followers</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_synced_followers]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper max file descriptor count</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_max_file_descriptor_count]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper max latency</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_max_latency]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units>tick</units>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper min latency</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_min_latency]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units>tick</units>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper opened file descriptor count</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_open_file_descriptor_count]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper outstanding requests</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_outstanding_requests]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper packages received</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_packets_received]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper packages sent</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_packets_sent]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper pid</name>  
                    <type>0</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[all]</key>  
                    <delay>30</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper response checking</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_server_ruok]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>1</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper state role</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_server_state]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>1</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper version</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_version]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>1</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper watches count</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_watch_count]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
                <item>  
                    <name>zookeeper znodes count</name>  
                    <type>2</type>  
                    <snmp_community/>  
                    <multiplier>0</multiplier>  
                    <snmp_oid/>  
                    <key>zookeeper.status[zk_znode_count]</key>  
                    <delay>0</delay>  
                    <history>90</history>  
                    <trends>365</trends>  
                    <status>0</status>  
                    <value_type>3</value_type>  
                    <allowed_hosts/>  
                    <units/>  
                    <delta>0</delta>  
                    <snmpv3_contextname/>  
                    <snmpv3_securityname/>  
                    <snmpv3_securitylevel>0</snmpv3_securitylevel>  
                    <snmpv3_authprotocol>0</snmpv3_authprotocol>  
                    <snmpv3_authpassphrase/>  
                    <snmpv3_privprotocol>0</snmpv3_privprotocol>  
                    <snmpv3_privpassphrase/>  
                    <formula>1</formula>  
                    <delay_flex/>  
                    <params/>  
                    <ipmi_sensor/>  
                    <data_type>0</data_type>  
                    <authtype>0</authtype>  
                    <username/>  
                    <password/>  
                    <publickey/>  
                    <privatekey/>  
                    <port/>  
                    <description/>  
                    <inventory_link>0</inventory_link>  
                    <applications>  
                        <application>  
                            <name>ZooKeeper Status</name>  
                        </application>  
                    </applications>  
                    <valuemap/>  
                    <logtimefmt/>  
                </item>  
            </items>  
            <discovery_rules/>  
            <macros/>  
            <templates/>  
            <screens/>  
        </template>  
    </templates>  
    <triggers>  
        <trigger>  
            <expression>{Template ZooKeeper:zookeeper.status[zk_outstanding_requests].last()}>10</expression>  
            <name>big outstanding requests number</name>  
            <url/>  
            <status>0</status>  
            <priority>4</priority>  
            <description/>  
            <type>0</type>  
            <dependencies/>  
        </trigger>  
        <trigger>  
            <expression>{Template ZooKeeper:zookeeper.status[zk_pending_syncs].last()}>10</expression>  
            <name>big pending syncs</name>  
            <url/>  
            <status>0</status>  
            <priority>4</priority>  
            <description/>  
            <type>0</type>  
            <dependencies/>  
        </trigger>  
        <trigger>  
            <expression>{Template ZooKeeper:zookeeper.status[zk_avg_latency].last()}>10</expression>  
            <name>large average latency</name>  
            <url/>  
            <status>0</status>  
            <priority>4</priority>  
            <description/>  
            <type>0</type>  
            <dependencies/>  
        </trigger>  
        <trigger>  
            <expression>{Template ZooKeeper:zookeeper.status[zk_open_file_descriptor_count].last()} > {Template ZooKeeper:zookeeper.status[zk_max_file_descriptor_count].last()}*0.85</expression>  
            <name>large file descriptor used</name>  
            <url/>  
            <status>0</status>  
            <priority>4</priority>  
            <description/>  
            <type>0</type>  
            <dependencies/>  
        </trigger>   
        <trigger>  
            <expression>{Template ZooKeeper:zookeeper.status[all].last()}=0</expression>  
            <name>zookeeper is not running</name>  
            <url/>  
            <status>0</status>  
            <priority>4</priority>  
            <description/>  
            <type>0</type>  
            <dependencies/>  
        </trigger>  
        <trigger>  
            <expression>{Template ZooKeeper:zookeeper.status[zk_server_state].abschange()}>0</expression>  
            <name>zookeeper state role has been changed</name>  
            <url/>  
            <status>0</status>  
            <priority>2</priority>  
            <description/>  
            <type>0</type>  
            <dependencies/>  
        </trigger>  
    </triggers>  
    <graphs>  
        <graph>  
            <name>ZooKeeper Alive Connections</name>  
            <width>900</width>  
            <height>200</height>  
            <yaxismin>0.0000</yaxismin>  
            <yaxismax>100.0000</yaxismax>  
            <show_work_period>1</show_work_period>  
            <show_triggers>1</show_triggers>  
            <type>1</type>  
            <show_legend>1</show_legend>  
            <show_3d>0</show_3d>  
            <percent_left>0.0000</percent_left>  
            <percent_right>0.0000</percent_right>  
            <ymin_type_1>0</ymin_type_1>  
            <ymax_type_1>0</ymax_type_1>  
            <ymin_item_1>0</ymin_item_1>  
            <ymax_item_1>0</ymax_item_1>  
            <graph_items>  
                <graph_item>  
                    <sortorder>0</sortorder>  
                    <drawtype>0</drawtype>  
                    <color>00DDDD</color>  
                    <yaxisside>0</yaxisside>  
                    <calc_fnc>2</calc_fnc>  
                    <type>0</type>  
                    <item>  
                        <host>Template ZooKeeper</host>  
                        <key>zookeeper.status[zk_num_alive_connections]</key>  
                    </item>  
                </graph_item>  
            </graph_items>  
        </graph>  
        <graph>  
            <name>ZooKeeper Data Size</name>  
            <width>900</width>  
            <height>200</height>  
            <yaxismin>0.0000</yaxismin>  
            <yaxismax>100.0000</yaxismax>  
            <show_work_period>1</show_work_period>  
            <show_triggers>1</show_triggers>  
            <type>1</type>  
            <show_legend>1</show_legend>  
            <show_3d>0</show_3d>  
            <percent_left>0.0000</percent_left>  
            <percent_right>0.0000</percent_right>  
            <ymin_type_1>0</ymin_type_1>  
            <ymax_type_1>0</ymax_type_1>  
            <ymin_item_1>0</ymin_item_1>  
            <ymax_item_1>0</ymax_item_1>  
            <graph_items>  
                <graph_item>  
                    <sortorder>0</sortorder>  
                    <drawtype>0</drawtype>  
                    <color>00C800</color>  
                    <yaxisside>0</yaxisside>  
                    <calc_fnc>2</calc_fnc>  
                    <type>0</type>  
                    <item>  
                        <host>Template ZooKeeper</host>  
                        <key>zookeeper.status[zk_approximate_data_size]</key>  
                    </item>  
                </graph_item>  
            </graph_items>  
        </graph>  
        <graph>  
            <name>ZooKeeper Latency</name>  
            <width>900</width>  
            <height>200</height>  
            <yaxismin>0.0000</yaxismin>  
            <yaxismax>100.0000</yaxismax>  
            <show_work_period>1</show_work_period>  
            <show_triggers>1</show_triggers>  
            <type>0</type>  
            <show_legend>1</show_legend>  
            <show_3d>0</show_3d>  
            <percent_left>0.0000</percent_left>  
            <percent_right>0.0000</percent_right>  
            <ymin_type_1>0</ymin_type_1>  
            <ymax_type_1>0</ymax_type_1>  
            <ymin_item_1>0</ymin_item_1>  
            <ymax_item_1>0</ymax_item_1>  
            <graph_items>  
                <graph_item>  
                    <sortorder>0</sortorder>  
                    <drawtype>2</drawtype>  
                    <color>00C800</color>  
                    <yaxisside>0</yaxisside>  
                    <calc_fnc>2</calc_fnc>  
                    <type>0</type>  
                    <item>  
                        <host>Template ZooKeeper</host>  
                        <key>zookeeper.status[zk_avg_latency]</key>  
                    </item>  
                </graph_item>  
                <graph_item>  
                    <sortorder>1</sortorder>  
                    <drawtype>2</drawtype>  
                    <color>C80000</color>  
                    <yaxisside>0</yaxisside>  
                    <calc_fnc>2</calc_fnc>  
                    <type>0</type>  
                    <item>  
                        <host>Template ZooKeeper</host>  
                        <key>zookeeper.status[zk_min_latency]</key>  
                    </item>  
                </graph_item>  
                <graph_item>  
                    <sortorder>2</sortorder>  
                    <drawtype>2</drawtype>  
                    <color>0000C8</color>  
                    <yaxisside>0</yaxisside>  
                    <calc_fnc>2</calc_fnc>  
                    <type>0</type>  
                    <item>  
                        <host>Template ZooKeeper</host>  
                        <key>zookeeper.status[zk_max_latency]</key>  
                    </item>  
                </graph_item>  
            </graph_items>  
        </graph>  
        <graph>  
            <name>ZooKeeper Packages Received/Sent</name>  
            <width>900</width>  
            <height>200</height>  
            <yaxismin>0.0000</yaxismin>  
            <yaxismax>100.0000</yaxismax>  
            <show_work_period>1</show_work_period>  
            <show_triggers>1</show_triggers>  
            <type>1</type>  
            <show_legend>1</show_legend>  
            <show_3d>0</show_3d>  
            <percent_left>0.0000</percent_left>  
            <percent_right>0.0000</percent_right>  
            <ymin_type_1>0</ymin_type_1>  
            <ymax_type_1>0</ymax_type_1>  
            <ymin_item_1>0</ymin_item_1>  
            <ymax_item_1>0</ymax_item_1>  
            <graph_items>  
                <graph_item>  
                    <sortorder>1</sortorder>  
                    <drawtype>0</drawtype>  
                    <color>00C800</color>  
                    <yaxisside>0</yaxisside>  
                    <calc_fnc>2</calc_fnc>  
                    <type>0</type>  
                    <item>  
                        <host>Template ZooKeeper</host>  
                        <key>zookeeper.status[zk_packets_received]</key>  
                    </item>  
                </graph_item>  
                <graph_item>  
                    <sortorder>0</sortorder>  
                    <drawtype>0</drawtype>  
                    <color>FF3333</color>  
                    <yaxisside>0</yaxisside>  
                    <calc_fnc>2</calc_fnc>  
                    <type>0</type>  
                    <item>  
                        <host>Template ZooKeeper</host>  
                        <key>zookeeper.status[zk_packets_sent]</key>  
                    </item>  
                </graph_item>  
            </graph_items>  
        </graph>  
        <graph>  
            <name>ZooKeeper Watches Count</name>  
            <width>900</width>  
            <height>200</height>  
            <yaxismin>0.0000</yaxismin>  
            <yaxismax>100.0000</yaxismax>  
            <show_work_period>1</show_work_period>  
            <show_triggers>1</show_triggers>  
            <type>1</type>  
            <show_legend>1</show_legend>  
            <show_3d>0</show_3d>  
            <percent_left>0.0000</percent_left>  
            <percent_right>0.0000</percent_right>  
            <ymin_type_1>0</ymin_type_1>  
            <ymax_type_1>0</ymax_type_1>  
            <ymin_item_1>0</ymin_item_1>  
            <ymax_item_1>0</ymax_item_1>  
            <graph_items>  
                <graph_item>  
                    <sortorder>0</sortorder>  
                    <drawtype>0</drawtype>  
                    <color>660066</color>  
                    <yaxisside>0</yaxisside>  
                    <calc_fnc>2</calc_fnc>  
                    <type>0</type>  
                    <item>  
                        <host>Template ZooKeeper</host>  
                        <key>zookeeper.status[zk_watch_count]</key>  
                    </item>  
                </graph_item>  
            </graph_items>  
        </graph>  
        <graph>  
            <name>ZooKeeper Znodes Count</name>  
            <width>900</width>  
            <height>200</height>  
            <yaxismin>0.0000</yaxismin>  
            <yaxismax>100.0000</yaxismax>  
            <show_work_period>1</show_work_period>  
            <show_triggers>1</show_triggers>  
            <type>0</type>  
            <show_legend>1</show_legend>  
            <show_3d>0</show_3d>  
            <percent_left>0.0000</percent_left>  
            <percent_right>0.0000</percent_right>  
            <ymin_type_1>0</ymin_type_1>  
            <ymax_type_1>0</ymax_type_1>  
            <ymin_item_1>0</ymin_item_1>  
            <ymax_item_1>0</ymax_item_1>  
            <graph_items>  
                <graph_item>  
                    <sortorder>0</sortorder>  
                    <drawtype>1</drawtype>  
                    <color>FFCCFF</color>  
                    <yaxisside>0</yaxisside>  
                    <calc_fnc>2</calc_fnc>  
                    <type>0</type>  
                    <item>  
                        <host>Template ZooKeeper</host>  
                        <key>zookeeper.status[zk_znode_count]</key>  
                    </item>  
                </graph_item>  
            </graph_items>  
        </graph>  
    </graphs>  
</zabbix_export>  

參考文章
http://blog.csdn.net/reblue520/article/details/52352689

最后編輯于
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成,瀏覽時請結(jié)合常識與多方信息審慎甄別。
平臺聲明:文章內(nèi)容(如有圖片或視頻亦包括在內(nèi))由作者上傳并發(fā)布,文章內(nèi)容僅代表作者本人觀點,簡書系信息發(fā)布平臺,僅提供信息存儲服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容