Day32 課堂作業(yè)

1. rsync介紹與實踐

1、什么是rsync?

rsync，Rsync英文全稱為Remote synchronization 縮寫 rsync
是開源、高速的、可實現(xiàn)本地以及遠程，全量以及增量的數(shù)據(jù)復(fù)制（拷貝）工具。

官方鏈接資料：http://www.samba.org/ftp/rsync/rsync.html

2、全量和增量區(qū)別

全量復(fù)制：

[root@nfs01 ~]# cp -a /etc/ /opt/
[root@nfs01 ~]# \cp -a /etc/ /opt
[root@nfs01 ~]# touch /etc/oldboy.txt
[root@nfs01 ~]# \cp -a /etc/ /opt/

增量復(fù)制：只復(fù)制 oldboy.txt

3、rsync的作用

工作中需要定時/實時數(shù)據(jù)備份。本地服務(wù)器目錄、不同機器、
不同機房之間的數(shù)據(jù)備份。都可以用 rsync 完成。

4、rsync功能特性

• 支持拷貝普通文件與特殊文件，如鏈接文件，設(shè)備文件等。
• 支持排除指定文件或目錄同步的功能，相當(dāng)于打包命令 tar 的排除功能。
• 可以做到保持原文件或目錄的權(quán)限、時間、軟硬鏈接、屬主、組等所有屬性均不改變。
• 可實現(xiàn)增量復(fù)制，既只復(fù)制發(fā)生變化的數(shù)據(jù)，因此數(shù)據(jù)傳輸效率很高。
• 可以使用 rcp , rsh ,ssh 等方式來配合進行隧道加密傳輸文件（ rsync 本身不對數(shù)據(jù)加密）。
• 可以通過 socket (進程方式)傳輸文件和數(shù)據(jù)（服務(wù)端和客戶端）。
• 支持匿名或認證（無需系統(tǒng)用戶）進程模式傳輸，安全的進行數(shù)據(jù)備份及鏡像

5、增量復(fù)制原理

使用 quick check 算法，只對增量的部分復(fù)制，根據(jù)大小屬性的變化進行復(fù)制。
版本2.x 比對差異后復(fù)制，版本 3.x 一邊比對一邊復(fù)制。
參考地址：https://coolshell.cn/articles/7425.html

6、rsync三種工作模式介紹

1）本地（ local ）
rsync 就是一個命令

語法：
rsync [OPTION...] SRC... [DEST]
rsync 參數(shù)選項 [源目錄或文件] [目的目錄或文件]

命令操作：
rsync命令
a. 把數(shù)據(jù)從一個地方復(fù)制到另一個地方（僅在一臺機器增量）,相當(dāng)于 cp 。
b. 通過加參數(shù)實現(xiàn)刪除文件和清空文件內(nèi)容的功能，相當(dāng)于 rm 命令。
c. 查看屬性信息功能，相當(dāng)于 ls 。

a. 拷貝實踐
不保持屬性：

[root@backup~]# ls /opt
[root@backup ~]# rsync /etc/hosts /opt
[root@backup ~]# ls /opt
hosts
[root@backup ~]# \cp /etc/hosts /opt

保持屬性：

[root@backup ~]# rsync -zrtopg /etc/hosts /opt/
[root@backup ~]# ls -lhi /etc/hosts /opt/hosts
16829878 -rw-r--r--. 1 root root 332 4月  12 11:24 /etc/hosts
   71373 -rw-r--r--  1 root root 332 4月  12 11:24 /opt/hosts

b. 刪除實踐

刪除文件內(nèi)容：

--------------------------------------------------------------
==>創(chuàng)建一個空文件/null.txt<==
--------------------------------------------------------------
[root@backup ~]# touch /null.txt 
[root@backup ~]# cat /opt/hosts 
127.0.0.1    localhost localhost.localdomain localhost4 localhost4.localdomain4
::1          localhost localhost.localdomain localhost6 localhost6.localdomain6
172.16.1.5 lb01
172.16.1.6 lb02
172.16.1.7 web01
172.16.1.8 web02
172.16.1.9 web03
172.16.1.31 nfs01
172.16.1.41 backup
172.16.1.51 db01 db01.etiantian.org
172.16.1.61 m01

--------------------------------------------------------------
==>讓后面hosts和一樣前面null.txt<==
--------------------------------------------------------------
[root@backup ~]# rsync -r --delete /null.txt /opt/hosts
[root@backup ~]# cat /opt/hosts 

刪除目錄下所有文件：

--------------------------------------------------------------
==>創(chuàng)建一個目錄/null<==
--------------------------------------------------------------
[root@backup ~]# mkdir /null

--------------------------------------------------------------
==>讓后面/opt和前面/null目錄內(nèi)容保持一致<==
--------------------------------------------------------------
[root@backup ~]# rsync -r --delete /null/ /opt/      
[root@backup ~]# ls /opt/

c. 查看屬性

[root@backup ~]# rsync /etc/hosts
-rw-r--r--            332 2019/04/12 11:24:41 hosts

2）遠程Shell模式

借助類似 ssh 隧道傳輸數(shù)據(jù)，適合不同的機器之間復(fù)制。
異地拷貝，相當(dāng)于 scp，區(qū)別：scp 是遠程全量拷貝

有兩種情況
第一種： pull ：（拉） 從遠端拉取到本地。

語法：

rsync [OPTION...] [USER@]HOST:SRC... [DEST]
rsync [參數(shù)選項] [認證用戶]@[主機地址]:[源路徑] [本地路徑]

第二種： push ：（推）從本地推到遠端。

語法：

rsync [OPTION...] SRC... [USER@]HOST:[DEST]
rsync [參數(shù)選項] [本地路徑] [認證用戶]@[主機地址]:[目標(biāo)路徑]

pull 的實踐：

--------------------------------------------------------------
==>兩條命令在目前是等價的<==
--------------------------------------------------------------
rsync -avz root@172.16.1.41:/opt/hosts /opt
rsync -avz -e "ssh -p 22" root@172.16.1.41:/opt/hosts /opt

--------------------------------------------------------------
==>輸入1.41的root密碼即可<==
--------------------------------------------------------------
[root@nfs01 /opt]# rsync -avz -e "ssh -p 22" root@172.16.1.41:/opt/hosts /opt
root@172.16.1.41's password: 
receiving incremental file list

sent 20 bytes  received 44 bytes  11.64 bytes/sec
total size is 332  speedup is 5.19

--------------------------------------------------------------
==>在本地檢查結(jié)果<==
--------------------------------------------------------------
[root@nfs01 ~]# cd /opt/
[root@nfs01 /opt]# ls
hosts
[root@nfs01 /opt]# 

rsync -avz /etc/hosts root@172.16.1.41:/opt/
rsync -avz /etc/hosts -e "ssh -p 22" root@172.16.1.41:/opt/

上述命令是等價的。-e 指定通道 ssh ssh 服務(wù)連接客戶端 -p 22指定 22 端口。

push 實踐：

--------------------------------------------------------------
==>加密傳輸。<==
--------------------------------------------------------------
[root@nfs01 ~]# rsync -avz /etc/hosts root@172.16.1.41:/opt/ 
The authenticity of host '172.16.1.41 (172.16.1.41)' can't be established.
ECDSA key fingerprint is SHA256:qZSBkrmOv7xO/63qOU1uLXkPyNVHdkqvrNAcAmXqNEk.
ECDSA key fingerprint is MD5:23:d0:cb:a9:f4:7c:0b:eb:2d:07:00:e1:a3:12:d8:33.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.1.41' (ECDSA) to the list of known hosts.
root@172.16.1.41's password: 
sending incremental file list
hosts

sent 219 bytes  received 35 bytes  14.51 bytes/sec
total size is 332  speedup is 1.31

--------------------------------------------------------------
==>在backup里檢查<==
--------------------------------------------------------------
[root@backup ~]# cd /opt/
[root@backup /opt]# ls
hosts

--------------------------------------------------------------
==>命令也可以這樣寫<==
--------------------------------------------------------------
[root@nfs01 ~]# rsync -avz /etc/hosts -e "ssh -p 22" root@172.16.1.41:/opt/
root@172.16.1.41's password: 
sending incremental file list

sent 44 bytes  received 12 bytes  22.40 bytes/sec
total size is 332  speedup is 5.93

小結(jié)

• a. 參照物，執(zhí)行命令的機器
• b. root@172.16.1.41 使用的用戶和主機，就用主機和用戶的密碼。

強調(diào)一個重點：適合rsync

null 和 null/ 區(qū)別，

• null 是目錄和目錄下的內(nèi)容，
• null/ 只是目錄下的內(nèi)容，不含本身。

3）rsync守護進程模式

首先要搭建rsync服務(wù)端(要有守護進程)，然后才能在客戶端實現(xiàn)推拉數(shù)據(jù)。
該模式是企業(yè)運維的重要模式（見后續(xù)）
rsync 服務(wù)模式：開啟后臺進程，接受別人訪問。

7、rsync命令參數(shù)

表1：rsync 命令參數(shù)：

提示：企業(yè)常用參數(shù)組合：-avz 或者 -vzrtopg

2. 部署rsync實戰(zhàn)

架構(gòu)前的準(zhǔn)備已在前面介紹了，在此，不在重復(fù)了

rsync 守護進程模式應(yīng)用實踐

（1）以下操作都是在backup服務(wù)器

1）安裝（前期已經(jīng)安裝過了）

--------------------------------------------------------------
==>查看版本<==
--------------------------------------------------------------
[root@backup ~]# rsync --version
rsync  version 3.1.2  protocol version 31

--------------------------------------------------------------
==>若沒有安裝，下面是安裝的命令<==
--------------------------------------------------------------
yum install rsync

2）配置配置文件/etc/rsyncd.conf

--------------------------------------------------------------
==>操作前先備份是個好習(xí)慣<==
--------------------------------------------------------------
cp /etc/rsyncd.conf{,.ori}

--------------------------------------------------------------
==>執(zhí)行以下命令<==
--------------------------------------------------------------
cat>/etc/rsyncd.conf<<EOF
#rsync_config_______________start
#created by oldboy
#site: http://www.oldboyedu.com
uid = rsync
gid = rsync
use chroot = no
fake super = yes
max connections = 200
timeout = 600 
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
ignore errors
read only = false
list = false
hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32
auth users = rsync_backup
secrets file = /etc/rsync.password
[backup]
comment = welcome to oldboyedu backup!
path = /backup/
EOF

--------------------------------------------------------------
==>配置文件說明<==
--------------------------------------------------------------
cat>/etc/rsyncd.conf<<EOF
#rsync_config_______________start ---------->注釋
#created by oldboy ----------------------->注釋
#site: http://www.oldboyedu.com ------->注釋
uid = rsync ------------------------------->管理備份目錄的用戶
gid = rsync ------------------------------>管理備份目錄的用戶組
use chroot = no ----------------------->安全功能，數(shù)據(jù)是否鎖定到備份目錄
fake super = yes ---------------------->使用時，是否不需要切換root用戶
max connections = 200 -------------->并發(fā)連接，同時多少客戶端訪問
timeout = 600 ----------------------->超時時間
pid file = /var/run/rsyncd.pid ------->進程號所在文件
lock file = /var/run/rsync.lock ------->鎖文件
log file = /var/log/rsyncd.log --------->日志文件，查看報錯等
ignore errors ---------------------------->忽略錯誤
read only = false ------------------------>可寫
list = false -------------------------------->不允許列表
hosts allow = 172.16.1.0/24 -------------->哪些主機可以訪問（和下面二選一）
hosts deny = 0.0.0.0/32 ------------------->哪些主機不可以訪問（和上面二選一）
auth users = rsync_backup ---------------->遠程虛擬連接用戶
secrets file = /etc/rsync.password --------->存放密碼的文件，格式：用戶名：密碼 ；權(quán)限：必須600
[backup] --------------------------------------->【模塊名】遠程訪問使用模塊名訪問
comment = welcome to oldboyedu backup!-->注釋
path = /backup/ --------------------------------->服務(wù)端用戶備份的目錄，用戶和組為：rsync rsync
EOF

--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@backup ~]# cat>/etc/rsyncd.conf<<EOF
> #rsync_config_______________start
> #created by oldboy
> #site: http://www.oldboyedu.com
> uid = rsync
> gid = rsync
> use chroot = no
> fake super = yes
> max connections = 200
> timeout = 600 
> pid file = /var/run/rsyncd.pid
> lock file = /var/run/rsync.lock
> log file = /var/log/rsyncd.log
> ignore errors
> read only = false
> list = false
> hosts allow = 172.16.1.0/24
> hosts deny = 0.0.0.0/32
> auth users = rsync_backup
> secrets file = /etc/rsync.password
> [backup]
> comment = welcome to oldboyedu backup!
> path = /backup/
> EOF

--------------------------------------------------------------
==>查看內(nèi)容如下<==
--------------------------------------------------------------
[root@backup ~]# cat /etc/rsyncd.conf
#rsync_config_______________start
#created by oldboy
#site: http://www.oldboyedu.com
uid = rsync
gid = rsync
use chroot = no
fake super = yes
max connections = 200
timeout = 600 
pid file = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
log file = /var/log/rsyncd.log
ignore errors
read only = false
list = false
hosts allow = 172.16.1.0/24
hosts deny = 0.0.0.0/32
auth users = rsync_backup
secrets file = /etc/rsync.password
[backup]
comment = welcome to oldboyedu backup!
path = /backup/

參考資料
man rsync 查命令的參數(shù)
man rsyncd.conf 查配置參數(shù)
https://www.samba.org/ftp/rsync/rsync.html

3）創(chuàng)建用戶和備份目錄

--------------------------------------------------------------
==>命令如下<==
--------------------------------------------------------------
useradd rsync
id rsync
mkdir -p /backup
chown -R rsync.rsync /backup/
ls -ld /backup/
 
--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@backup ~]# useradd rsync
[root@backup ~]# id rsync
uid=1001(rsync) gid=1001(rsync) 組=1001(rsync)
[root@backup ~]# mkdir -p /backup
[root@backup ~]# chown -R rsync.rsync /backup/
[root@backup ~]# ls -ld /backup/
drwxr-xr-x 2 rsync rsync 6 4月  15 12:12 /backup/

4）啟動和檢查

--------------------------------------------------------------
==>CentOS 6及以前啟動方法<==
--------------------------------------------------------------
rsync --daemon

--------------------------------------------------------------
==>CentOS7 啟動方法<==
--------------------------------------------------------------
systemctl start rsyncd

--------------------------------------------------------------
==>設(shè)置開機自啟動<==
--------------------------------------------------------------
systemctl enable rsyncd

--------------------------------------------------------------
==>查看狀態(tài)<==
--------------------------------------------------------------
systemctl status rsyncd

--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@backup ~]# systemctl start rsyncd
[root@backup ~]# systemctl enable rsyncd
[root@backup ~]# systemctl status rsyncd
● rsyncd.service - fast remote file copy program daemon
   Loaded: loaded (/usr/lib/systemd/system/rsyncd.service; enabled; vendor preset: disabled)
   Active: active (running) since 一 2019-04-15 12:21:29 CST; 3h 15min ago
 Main PID: 22814 (rsync)
   CGroup: /system.slice/rsyncd.service
           └─22814 /usr/bin/rsync --daemon --no-detach

4月 15 12:21:29 backup systemd[1]: Started fast remote file copy program daemon.

--------------------------------------------------------------
==>檢查進程<==
--------------------------------------------------------------
[root@backup ~]# ps -ef|grep sync|grep -v grep  
root       7521      1  0 11:39 ?        00:00:00 /usr/bin/rsync --daemon --no-detach

--------------------------------------------------------------
==>檢查端口<==
--------------------------------------------------------------
[root@backup ~]# netstat -lntup|grep 873 
tcp        0      0 0.0.0.0:873             0.0.0.0:*               LISTEN      7521/rsync          
tcp6       0      0 :::873                  :::*                    LISTEN      7521/rsync 

--------------------------------------------------------------
==>檢查端口<==
--------------------------------------------------------------
[root@backup ~]# lsof -i :873 
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
rsync   7521 root    3u  IPv4  41439      0t0  TCP *:rsync (LISTEN)
rsync   7521 root    5u  IPv6  41440      0t0  TCP *:rsync (LISTEN)

面試題：如何查看某端口對應(yīng)什么服務(wù)？
答：netstat -lntup|grep 端口號
或者 lsof -i :端口號

5）配置密碼文件

--------------------------------------------------------------
==>命令如下<==
--------------------------------------------------------------
echo "rsync_backup:oldboy" > /etc/rsync.password
chmod 600 /etc/rsync.password
cat /etc/rsync.password
ls -l /etc/rsync.password
 
--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@backup ~]# echo "rsync_backup:oldboy" > /etc/rsync.password
[root@backup ~]# chmod 600 /etc/rsync.password
[root@backup ~]# cat /etc/rsync.password
rsync_backup:oldboy
[root@backup ~]# ls -l /etc/rsync.password
-rw------- 1 root root 20 4月  15 11:51 /etc/rsync.password

至此，rsync 服務(wù)端配置完成。

（2）以下操作都是在客戶端服務(wù)器

以下方法2選1（推薦第二種）

方法1：認證密碼文件（后面還需要輸入?yún)?shù)，比較長）

--------------------------------------------------------------
==>命令如下<==
--------------------------------------------------------------
echo "oldboy" > /etc/rsync.password
chmod 600 /etc/rsync.password
cat /etc/rsync.password
ls -l /etc/rsync.password

--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@nfs01 ~]# echo "oldboy" > /etc/rsync.password
[root@nfs01 ~]# chmod 600 /etc/rsync.password
[root@nfs01 ~]# cat /etc/rsync.password
oldboy
[root@nfs01 ~]# ls -l /etc/rsync.password
-rw------- 1 root root 7 4月  15 11:55 /etc/rsync.password

方法2：（后面不需要輸入密碼）

--------------------------------------------------------------
==>命令如下<==
--------------------------------------------------------------
echo ' export RSYNC_PASSWORD=oldboy' >>/etc/bashrc 
tail -1 /etc/bashrc 
 . /etc/bashrc 
echo $RSYNC_PASSWORD

--------------------------------------------------------------
==>演示如下<==
--------------------------------------------------------------
[root@nfs01 ~]# echo ' export RSYNC_PASSWORD=oldboy' >>/etc/bashrc 
[root@nfs01 ~]# tail -1 /etc/bashrc 
 export RSYNC_PASSWORD=oldboy
[root@nfs01 ~]# . /etc/bashrc 
[root@nfs01 ~]# echo $RSYNC_PASSWORD
oldboy

至此， nfs01 的 rsync 客戶端配置完成

（3）守護進程模式，客戶端rsync的命令語法：

配置服務(wù)器端守護進程，實現(xiàn)數(shù)據(jù)傳輸

1、服務(wù)器端守護進程。
2、客戶端執(zhí)行命令。

1）、pull ：（拉）從遠端拉取到本地。

語法1（常用）：（注意：與之前的語法有所變化）

rsync [OPTION...] [USER@]HOST::SRC... [DEST]
rsync [參數(shù)選項] [虛擬用戶]@[主機地址]::[模塊名] [本地路徑]

語法2:
rsync [OPTION...] rsync://[USER@]HOST::SRC... [DEST]
rsync [參數(shù)選項] rsync://[虛擬用戶]@[主機地址]/[模塊名] [本地路徑]

2）、push ：（推）從本地推到遠端。

語法1（常用）：

rsync [OPTION...] [DEST] [USER@]HOST::SRC...
rsync [參數(shù)選項] [本地路徑] [虛擬用戶]@[主機地址]::[模塊名]

語法2:

rsync [OPTION...] [DEST] rsync://[USER@]HOST::SRC...
rsync [參數(shù)選項] [本地路徑] rsync://[虛擬用戶]@[主機地址]/[模塊名]

（4）測試成果

以下在實際操作過程報錯

錯誤1：

[root@nfs01 ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password
@ERROR: invalid uid rsync
rsync error: error starting client-server protocol (code 5) at main.c(1648) [sender=3.1.2]

解答：原因是在配置服務(wù)端的時候未創(chuàng)造 rsync 用戶導(dǎo)致的

[root@backup ~]# useradd rsync
[root@backup ~]# id rsync
uid=1001(rsync) gid=1001(rsync) 組=1001(rsync)
[root@backup ~]# mkdir -p /backup
[root@backup ~]# chown -R rsync.rsync /backup/
[root@backup ~]# ls -ld /backup/
drwxr-xr-x 2 rsync rsync 6 4月  15 12:12 /backup/

錯誤2：

[root@nfs01 ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password
sending incremental file list
hosts
rsync: chgrp ".hosts.YDuTjO" (in backup) failed: Operation not permitted (1)

sent 223 bytes  received 124 bytes  694.00 bytes/sec
total size is 332  speedup is 0.96
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1178) [sender=3.1.2]

解答：增加如下參數(shù)到 /etc/rsyncd.conf

fake  super  = yes #不用root用戶也可以存儲文件的完整屬性。
This allows the full attributes of a file to  be  stored  without having to have the daemon actually running as root.
[root@backup ~]# grep fake /etc/rsyncd.conf
fake super = yes

改完配置，要重啟服務(wù)：

[root@backup ~]# systemctl restart rsyncd

然后再測試：成功

--------------------------------------------------------------
==>客戶端操作以下命令<==
--------------------------------------------------------------
[root@nfs01 ~]# rsync -avz /etc/hosts rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password
sending incremental file list
hosts

sent 89 bytes  received 49 bytes  276.00 bytes/sec
total size is 332  speedup is 2.41
[root@nfs01 ~]# rsync -avz /etc rsync_backup@172.16.1.41::backup --password-file=/etc/rsync.password

--------------------------------------------------------------
==>服務(wù)端檢查<==
--------------------------------------------------------------
[root@backup ~]# ls /backup/
etc  hosts

注意： 如果客戶端按照環(huán)境變量的方式配置，則可以忽略 --password-file=/etc/rsync.password 參數(shù)。(前面也提到了，就是這種情況）

--------------------------------------------------------------
==>不需要參數(shù)和密碼也可以<==
--------------------------------------------------------------
[root@nfs01 ~]# rsync -avz /etc rsync_backup@172.16.1.41::backup
sending incremental file list

sent 52,071 bytes  received 644 bytes  105,430.00 bytes/sec
total size is 31,244,350  speedup is 592.70

測試增量

[root@nfs01 ~]# touch /etc/oldboy.txt
[root@nfs01 ~]# rsync -avz /etc rsync_backup@172.16.1.41::backup
sending incremental file list
etc/oldboy.txt

sent 52,098 bytes  received 655 bytes  105,506.00 bytes/sec
total size is 31,244,350  speedup is 592.28

第二種語法測試

[root@nfs01 ~]# rsync -avz /etc rsync://rsync_backup@172.16.1.41/backup
sending incremental file list

sent 52,070 bytes  received 644 bytes  105,428.00 bytes/sec
total size is 31,244,350  speedup is 592.71