Multipath 多路徑配置實(shí)踐心得

前言

配置存儲(chǔ)時(shí)一定會(huì)遇到 multipath 多路徑的問題，不同的廠商比如 EMC PowerPath，Veritas VxDMP 等都有獨(dú)立的多路徑軟件，而多路徑軟件的功能也很清晰主要用于IO流量負(fù)載均衡和故障切換恢復(fù)等。在 Linux 環(huán)境中 device-mapper-multipath 是一個(gè)免費(fèi)的通用型多路徑管理軟件，其配置文件也非常簡(jiǎn)單，主要通過修改 /etc/multipath.conf 來調(diào)整。文章提供了《HPE 3PAR Red Hat Enterprise Linux和 Oracle Linux 實(shí)施指南》，也分享了自己配置 multipath 的實(shí)踐過程，希望對(duì)大家有參考價(jià)值。

記錄 3PAR8400 存儲(chǔ) Multipath 多路徑配置

更新歷史

2017年05月04日 - 初稿

閱讀原文 - https://wsgzao.github.io/post/multipath/

擴(kuò)展閱讀

multipath - https://access.redhat.com/labsinfo/multipathhelper
[原]紅旗上使用multipath復(fù)合多條路徑 - http://www.linuxfly.org/post/513/

官方最佳實(shí)踐手冊(cè)

HPE 3PAR Red Hat Enterprise Linux和 Oracle Linux 實(shí)施指南

<iframe src="https://www.slideshare.net/slideshow/embed_code/key/MKQJhstxnv4JFy" width="479" height="511" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe> <div style="margin-bottom:5px"> </div>

什么是multipath

普通的電腦主機(jī)都是一個(gè)硬盤掛接到一個(gè)總線上，這里是一對(duì)一的關(guān)系。而到了有光纖組成的SAN環(huán)境，由于主機(jī)和存儲(chǔ)通過了光纖交換機(jī)連接，這樣的話，就構(gòu)成了多對(duì)多的關(guān)系。也就是說，主機(jī)到存儲(chǔ)可以有多條路徑可以選擇。主機(jī)到存儲(chǔ)之間的IO由多條路徑可以選擇。

既然，每個(gè)主機(jī)到所對(duì)應(yīng)的存儲(chǔ)可以經(jīng)過幾條不同的路徑，如果是同時(shí)使用的話，I/O流量如何分配？其中一條路徑壞掉了，如何處理？還有在操作系統(tǒng)的角度來看，每條路徑，操作系統(tǒng)會(huì)認(rèn)為是一個(gè)實(shí)際存在的物理盤，但實(shí)際上只是通向同一個(gè)物理盤的不同路徑而已，這樣是在使用的時(shí)候，就給用戶帶來了困惑。多路徑軟件就是為了解決上面的問題應(yīng)運(yùn)而生的，多路徑的主要功能就是和存儲(chǔ)設(shè)備一起配合實(shí)現(xiàn)如下功能：

  1. 故障的切換和恢復(fù)
  2. IO流量的負(fù)載均衡
  3. 磁盤的虛擬化

為什么使用multipath

由于多路徑軟件是需要和存儲(chǔ)在一起配合使用的，不同的廠商基于不同的操作系統(tǒng)，都提供了不同的版本。并且有的廠商，軟件和硬件也不是一起賣的，如果要使用多路徑軟件的話，可能還需要向廠商購(gòu)買license才行。比如EMC公司基于Linux下的多路徑軟件，就需要單獨(dú)的購(gòu)買license。其中，EMC提供的就是PowerPath，HDS提供的就是HDLM，Veritas提供的就是VxDMP。當(dāng)然，使用系統(tǒng)自帶的免費(fèi)多路徑軟件包，同時(shí)也是一個(gè)比較通用的包，可以支持大多數(shù)存儲(chǔ)廠商的設(shè)備，即使是一些不是出名的廠商，通過對(duì)配置文件進(jìn)行稍作修改，也是可以支持并運(yùn)行的很好。

比較重要的一點(diǎn)還是聽從原廠工程師的建議根據(jù)實(shí)際的業(yè)務(wù)和存儲(chǔ)策略使用合適的多路徑軟件。

配置multipath

原理看了一堆，實(shí)際配置還是比較簡(jiǎn)單的，配置文件只有一個(gè)：/etc/multipath.conf

#查看主機(jī)或者存儲(chǔ)交換機(jī)上的WWN號(hào)，在存儲(chǔ)上將LUN映射給需要的主機(jī)
cat /sys/class/fc_host/host*/port_name
0x2002d0431efb7f5d
0x2001d0431efb7f5d

#在系統(tǒng)內(nèi)執(zhí)行掃盤命令，沒有命令先安裝sg3_utils
yum install sg3_utils
rescan-scsi-bus.sh

#查看是否映射到對(duì)應(yīng)的
fdisk -l
lsblk

#查看是否安裝了multipath
yum install device-mapper-multipath

rpm -qa | grep device-mapper
device-mapper-libs-1.02.107-5.el7_2.2.x86_64
device-mapper-persistent-data-0.5.5-1.el7.x86_64
device-mapper-multipath-0.4.9-85.el7_2.4.x86_64
device-mapper-1.02.107-5.el7_2.2.x86_64
device-mapper-event-libs-1.02.107-5.el7_2.2.x86_64
device-mapper-event-1.02.107-5.el7_2.2.x86_64
device-mapper-multipath-libs-0.4.9-85.el7_2.4.x86_64


#拷貝默認(rèn)的multipath.conf到/etc目錄下，也可以使用mpathconf命令創(chuàng)建默認(rèn)模板
cp /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf /etc/multipath.conf
mpathconf --enable --with_multipathd y

#查看本地存儲(chǔ)wwid
/lib/udev/scsi_id --whitelisted --device=/dev/sda
36141877030bcca001d9c4d52106b9d90

#查看存儲(chǔ)參數(shù)
cat /sys/block/sdb/device/vendor 
3PARdata
cat /sys/block/sdb/device/model
VV

#添加本地盤到blacklist內(nèi),不同的存儲(chǔ)和系統(tǒng)參考官方的最佳實(shí)踐
vim /etc/multipath.conf

defaults {
polling_interval 10
user_friendly_names no
find_multipaths yes
}
blacklist {
wwid "<wwid_of_the_local_disk>"
}
devices {
device {
vendor "3PARdata"
product "VV"
path_grouping_policy group_by_prio
path_selector "round-robin 0"
path_checker tur
features "0"
hardware_handler "1 alua"
prio alua
failback immediate
rr_weight uniform
no_path_retry 18
rr_min_io_rq 1
detect_prio yes
# fast_io_fail_tmo 10
# dev_loss_tmo 14
}
}

#固定設(shè)備的別名
cat /etc/multipath/bindings 
# Multipath bindings, Version : 1.0
# NOTE: this file is automatically maintained by the multipath program.
# You should not need to edit this file in normal circumstances.
#
# Format:
# alias wwid
#
mpatha 360002ac000000000000000260001a945
mpathb 360002ac000000000000000270001a945
mpathc 360002ac000000000000000280001a945
mpathd 360002ac000000000000000290001a945
mpathe 360002ac0000000000000002a0001a945

#編輯multipath.conf，增加以下字段
vim /etc/multipath.conf

multipaths {
  multipath {
  wwid 360002ac000000000000000260001a945
  alias 3PAR8400_DEV_LUN26
  }
  multipath {
  wwid 360002ac000000000000000270001a945
  alias 3PAR8400_DEV_LUN27
  }
  multipath {
  wwid 360002ac000000000000000280001a945
  alias 3PAR8400_DEV_LUN28
  }
  multipath {
  wwid 360002ac000000000000000290001a945
  alias 3PAR8400_DEV_LUN29
  }
  multipath {
  wwid 360002ac0000000000000002a0001a945
  alias 3PAR8400_DEV_LUN30
  }
}

#清空已有的multipath記錄
multipath -F
#打印診斷信息
multipath -v3
#啟用多路徑守護(hù)程序以在引導(dǎo)時(shí)啟動(dòng)
mpathconf --enable
#啟動(dòng)多路徑服務(wù)
service multipathd start
#如果在啟動(dòng) multipath 守護(hù)程序后更改多路徑配置文件，請(qǐng)運(yùn)行以下命令以使更改生效。
service multipathd reload 
#重啟系統(tǒng)測(cè)試
init 6
#查看多路徑當(dāng)前狀態(tài)
multipath -ll

#Enable extended logging on the qla2xxx driver
chmod u+x /sys/module/qla2xxx/parameters/ql2xextended_error_logging
echo "1" > /sys/module/qla2xxx/parameters/ql2xextended_error_logging
cat /sys/module/qla2xxx/parameters/ql2xextended_error_logging

#重啟lvm服務(wù)是識(shí)別共享存儲(chǔ)vg信息
systemctl restart lvm2-lvmetad.service
pvs

#屏蔽WARNING: duplicate提示信息
vim /etc/lvm/lvm.conf

    # Configuration option devices/filter.
    # Limit the block devices that are used by LVM commands.
    # This is a list of regular expressions used to accept or reject block
    # device path names. Each regex is delimited by a vertical bar '|'
    # (or any character) and is preceded by 'a' to accept the path, or
    # by 'r' to reject the path. The first regex in the list to match the
    # path is used, producing the 'a' or 'r' result for the device.
    # When multiple path names exist for a block device, if any path name
    # matches an 'a' pattern before an 'r' pattern, then the device is
    # accepted. If all the path names match an 'r' pattern first, then the
    # device is rejected. Unmatching path names do not affect the accept
    # or reject decision. If no path names for a device match a pattern,
    # then the device is accepted. Be careful mixing 'a' and 'r' patterns,
    # as the combination might produce unexpected results (test changes.)
    # Run vgscan after changing the filter to regenerate the cache.
    # See the use_lvmetad comment for a special case regarding filters.
    # 
    # Example
    # Accept every block device:
    # filter = [ "a|.*/|" ]
    # Reject the cdrom drive:
    # filter = [ "r|/dev/cdrom|" ]
    # Work with just loopback devices, e.g. for testing:
    # filter = [ "a|loop|", "r|.*|" ]
    # Accept all loop devices and ide drives except hdc:
    # filter = [ "a|loop|", "r|/dev/hdc|", "a|/dev/ide|", "r|.*|" ]
    # Use anchors to be very specific:
    # filter = [ "a|^/dev/hda8$|", "r|.*/|" ]
    # 
    # This configuration option has an automatic default value.
    # filter = [ "a|.*/|" ]
    filter = ["a/sda/","a/mapper/3PAR8400_DEV_LUN26p1/","r/.*/"]
#filter = ["a/sda/","r/.*/"]

    # Configuration option devices/global_filter.
    # Limit the block devices that are used by LVM system components.
    # Because devices/filter may be overridden from the command line, it is
    # not suitable for system-wide device filtering, e.g. udev and lvmetad.
    # Use global_filter to hide devices from these LVM system components.
    # The syntax is the same as devices/filter. Devices rejected by
    # global_filter are not opened by LVM.
    # This configuration option has an automatic default value.
    # global_filter = [ "a|.*/|" ]
    global_filter = ["a/sda/","a/mapper/3PAR8400_DEV_LUN26p1/","r/.*/"]


#一個(gè)簡(jiǎn)單的multipath.conf配置文件
[root@controller01 ~]# cat /etc/multipath.conf
# This is a basic configuration file with some examples, for device mapper
# multipath.
#
# For a complete list of the default configuration values, run either
# multipath -t
# or
# multipathd show config
#
# For a list of configuration options with descriptions, see the multipath.conf
# man page

## By default, devices with vendor = "IBM" and product = "S/390.*" are
## blacklisted. To enable mulitpathing on these devies, uncomment the
## following lines.
#blacklist_exceptions {
#    device {
#        vendor    "IBM"
#        product    "S/390.*"
#    }
#}

## Use user friendly names, instead of using WWIDs as names.
defaults {
    user_friendly_names yes
    find_multipaths yes
}
##
## Here is an example of how to configure some standard options.
##
#
#defaults {
#    polling_interval     10
#    path_selector        "round-robin 0"
#    path_grouping_policy    multibus
#    uid_attribute        ID_SERIAL
#    prio            alua
#    path_checker        readsector0
#    rr_min_io        100
#    max_fds            8192
#    rr_weight        priorities
#    failback        immediate
#    no_path_retry        fail
#    user_friendly_names    yes
#}
##
## The wwid line in the following blacklist section is shown as an example
## of how to blacklist devices by wwid.  The 2 devnode lines are the
## compiled in default blacklist. If you want to blacklist entire types
## of devices, such as all scsi devices, you should use a devnode line.
## However, if you want to blacklist specific devices, you should use
## a wwid line.  Since there is no guarantee that a specific device will
## not change names on reboot (from /dev/sda to /dev/sdb for example)
## devnode lines are not recommended for blacklisting specific devices.
##
#blacklist {
#       wwid 26353900f02796769
#    devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
#    devnode "^hd[a-z]"
#}
#multipaths {
#    multipath {
#        wwid            3600508b4000156d700012000000b0000
#        alias            yellow
#        path_grouping_policy    multibus
#        path_selector        "round-robin 0"
#        failback        manual
#        rr_weight        priorities
#        no_path_retry        5
#    }
#    multipath {
#        wwid            1DEC_____321816758474
#        alias            red
#    }
#}
#devices {
#    device {
#        vendor            "COMPAQ  "
#        product            "HSV110 (C)COMPAQ"
#        path_grouping_policy    multibus
#        path_checker        readsector0
#        path_selector        "round-robin 0"
#        hardware_handler    "0"
#        failback        15
#        rr_weight        priorities
#        no_path_retry        queue
#    }
#    device {
#        vendor            "COMPAQ  "
#        product            "MSA1000         "
#        path_grouping_policy    multibus
#    }
#}
multipaths {
  multipath {
  wwid 360002ac000000000000000260001a945
  alias 3PAR8400_DEV_LUN26
  }
  multipath {
  wwid 360002ac000000000000000270001a945
  alias 3PAR8400_DEV_LUN27
  }
  multipath {
  wwid 360002ac000000000000000280001a945
  alias 3PAR8400_DEV_LUN28
  }
  multipath {
  wwid 360002ac000000000000000290001a945
  alias 3PAR8400_DEV_LUN29
  }
  multipath {
  wwid 360002ac0000000000000002a0001a945
  alias 3PAR8400_DEV_LUN30
  }
  multipath {
  wwid 360002ac0000000000000002b0001a945
  alias 3PAR8400_DEV_LUN31
  }
  multipath {
  wwid 360002ac0000000000000002c0001a945
  alias 3PAR8400_DEV_LUN32
  }
  multipath {
  wwid 360002ac0000000000000002d0001a945
  alias 3PAR8400_DEV_LUN33
  }
  multipath {
  wwid 360002ac0000000000000002e0001a945
  alias 3PAR8400_DEV_LUN34
  }
  multipath {
  wwid 360002ac0000000000000002f0001a945
  alias 3PAR8400_DEV_LUN35
  }
  multipath {
  wwid 360002ac000000000000000320001a945
  alias 3PAR8400_DEV_LUN36
  }
  multipath {
  wwid 360002ac000000000000000330001a945
  alias 3PAR8400_DEV_LUN37
  }
  multipath {
  wwid 360002ac000000000000000340001a945
  alias 3PAR8400_DEV_LUN38
  }
  multipath {
  wwid 360002ac000000000000000350001a945
  alias 3PAR8400_DEV_LUN39
  }
  multipath {
  wwid 360002ac000000000000000360001a945
  alias 3PAR8400_DEV_LUN40
  }
  multipath {
  wwid 360002ac000000000000000370001a945
  alias 3PAR8400_DEV_LUN41
  }
  multipath {
  wwid 360002ac000000000000000380001a945
  alias 3PAR8400_DEV_LUN42
  }
  multipath {
  wwid 360002ac000000000000000390001a945
  alias 3PAR8400_DEV_LUN43
  }
  multipath {
  wwid 360002ac0000000000000003a0001a945
  alias 3PAR8400_DEV_LUN44
  }
  multipath {
  wwid 360002ac0000000000000003b0001a945
  alias 3PAR8400_DEV_LUN45
  }
  multipath {
  wwid 360002ac0000000000000003d0001a945
  alias 3PAR8400_DEV_LUN46
  }
  multipath {
  wwid 360002ac0000000000000003e0001a945
  alias 3PAR8400_DEV_LUN47
  }
  multipath {
  wwid 360002ac0000000000000003f0001a945
  alias 3PAR8400_DEV_LUN48
  }
  multipath {
  wwid 360002ac000000000000000400001a945
  alias 3PAR8400_DEV_LUN49
  }
  multipath {
  wwid 360002ac000000000000000410001a945
  alias 3PAR8400_DEV_LUN50
  }
}
最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

  • Spring Cloud為開發(fā)人員提供了快速構(gòu)建分布式系統(tǒng)中一些常見模式的工具（例如配置管理，服務(wù)發(fā)現(xiàn)，斷路器，智...
    卡卡羅2017閱讀 136,568評(píng)論 19贊 139
  • Ubuntu的發(fā)音 Ubuntu，源于非洲祖魯人和科薩人的語(yǔ)言，發(fā)作 oo-boon-too 的音。了解發(fā)音是有意...
    螢火蟲de夢(mèng)閱讀 100,677評(píng)論 9贊 468
  • ANN [calming down; without looking up] What's that? DOCTO...
    試遣愚衷閱讀 327評(píng)論 0贊 0
  • “一個(gè)人，生活可以變得好，也可以變得壞；可以活得久，也可以活得不久；可以做一個(gè)藝術(shù)家，也可以鋸木頭，沒有多大的區(qū)別...
    系統(tǒng)排列閱讀 344評(píng)論 0贊 0
  • 2016年的夜晚，你刷著微博，屏幕上突然彈出了“演員喬任梁離世”消息。 網(wǎng)絡(luò)上除了粉絲的哀悼，同時(shí)還將“抑郁癥”這...
    秋刀魚醬0v0閱讀 889評(píng)論 0贊 3

友情鏈接更多精彩內(nèi)容