Tips
- 整個(gè)環(huán)境是有兩臺(tái)虛擬機(jī)作為節(jié)點(diǎn)構(gòu)建k8s集群，第三臺(tái)虛擬機(jī)作為harbor倉(cāng)庫(kù)組建；具體的kubernetes集群部署之前已經(jīng)完成使用kubeadmin配合calico快速搭建kubernetes集群,此篇文章只介紹如何搭建harbor，并在kubernetes中使用。
- Harbor服務(wù)器的主機(jī)名不要設(shè)置成harbor、docker.io這樣的主機(jī)名，不然可能會(huì)在后期push的有問(wèn)題。
1. 整理部署架構(gòu)
| 角色 | IP | 服務(wù) | VCPU | 內(nèi)存 |
|---|---|---|---|---|
| master | 192.168.199.30 | etcd、kube-apiserver、kube-scheduler、kube-controller-manager、kube-proxy、calico-node | 2 | 2 |
| node1 | 192.168.199.31 | kube-proxy、calico-node | 2 | 2 |
| harbor.registry | 192.168.199.32 | docker-compose、harbor | 2 | 4 |
2. 環(huán)境準(zhǔn)備
(1) 在三臺(tái)主機(jī)上執(zhí)行以下操作
- 編輯
/etc/hosts文件，將主機(jī)映射關(guān)系寫(xiě)入：
192.168.199.31 node1
192.168.199.30 master
192.168.199.30 etcd
192.168.199.30 apiserver.k8s
192.168.199.32 harbor.registry
- 關(guān)閉防火墻
[root@master yum.repos.d]# systemctl stop firewalld
[root@master yum.repos.d]# systemctl disable firewalld
- 關(guān)閉swap
swap，這個(gè)當(dāng)內(nèi)存不足時(shí)，linux會(huì)自動(dòng)使用swap，將部分內(nèi)存數(shù)據(jù)存放到磁盤(pán)中，這個(gè)這樣會(huì)使性能下降，為了性能考慮推薦關(guān)掉
[root@master yum.repos.d]# swapoff -a
- 關(guān)閉selinux
vi /etc/selinux/config，關(guān)閉SELINUX
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
- 配置yum源，此處配置的是阿里的Centos 7的yum源和kubernetes源
Tips：可以在配置之前將/etc/yum.repos.d/ 下的文件都備份到bak目錄下
[root@master ~]# cd /etc/yum.repos.d/ && curl -O http://mirrors.aliyun.com/repo/Centos-7.repo
[root@master yum.repos.d]# vi kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg"
[root@master yum.repos.d]yum clean all
[root@master yum.repos.d]#yum makecache
-
reboot重啟操作系統(tǒng)，使hostname、SELINUX配置生效
3. 安裝docker和docker-compose
3.1 安裝docker
將需要執(zhí)行的步驟都寫(xiě)入了腳本，直接運(yùn)行腳本內(nèi)容即可。 腳本內(nèi)容如下：
#!/bin/bash
# 先定義一下國(guó)內(nèi)的鏡像加速源
export REGISTRY_MIRROR=https://registry.cn-hangzhou.aliyuncs.com
# 安裝 docker
# 參考文檔如下
# https://docs.docker.com/install/linux/docker-ce/centos/
# https://docs.docker.com/install/linux/linux-postinstall/
# 卸載舊版本
yum remove -y docker \
docker-client \
docker-client-latest \
docker-ce-cli \
docker-common \
docker-latest \
docker-latest-logrotate \
docker-logrotate \
docker-selinux \
docker-engine-selinux \
docker-engine
# 設(shè)置 yum repository
yum install -y yum-utils \
device-mapper-persistent-data \
lvm2
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
# 安裝并啟動(dòng) docker
yum install -y docker-ce-19.03.11 docker-ce-cli-19.03.11 containerd.io-1.2.13
mkdir /etc/docker || true
cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors": ["${REGISTRY_MIRROR}"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
EOF
mkdir -p /etc/systemd/system/docker.service.d
# Restart Docker
systemctl daemon-reload
systemctl enable docker
systemctl restart docker
docker version
3.2 安裝docker-compose
下載軟件：docker-compose軟件是一個(gè)可執(zhí)行的二進(jìn)制文件，在要安裝Harbor私有鏡像倉(cāng)庫(kù)的虛擬機(jī)中執(zhí)行以下命令即可從官網(wǎng)中下載docker-compose軟件，下載速度很比較慢，如果失敗，則需要多試幾次。
curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
設(shè)置權(quán)限：docker-compose可執(zhí)行文件下載到了/usr/local/bin目錄下，但是并不具有可執(zhí)行的權(quán)限，所以要設(shè)置可執(zhí)行的權(quán)限。在要安裝Harbor私有鏡像倉(cāng)庫(kù)的虛擬機(jī)中執(zhí)行以下命令即可設(shè)置可執(zhí)行權(quán)限。
sudo chmod +x /usr/local/bin/docker-compose
查看版本：給docker-compose可執(zhí)行文件設(shè)置了可執(zhí)行權(quán)限之后，即可執(zhí)行以下命令以查看安裝的docker-compose的版本。
docker-compose --version
4. 安裝harbor
官方文檔：Harbor的GitHub官方倉(cāng)庫(kù)、Harbor的官方下載地址、Harbor的官方安裝文檔
個(gè)人建議直接到官方下載地址，下載到本地之后再上傳到服務(wù)器上， 比較快。
此次下載的版本為v1.10.6的版本。Harbor的安裝分為離線和在線兩種安裝方式，此次選擇離線的部署方式，文件名為harbor-offline-installer-v1.10.6.tgz。上傳到服務(wù)器之后，執(zhí)行以下操作：
- 解壓文件夾并進(jìn)入
[root@harbor ~]# tar -zvxf harbor-offline-installer-v1.10.6.tgz && cd ./harbor
進(jìn)入文件夾之后，修改配置文件harbor.yml。主要注意以下內(nèi)容即可：
harbor.yml配置內(nèi)容
- 執(zhí)行
[root@harbor harbor]# bash install.sh直接安裝即可，運(yùn)行結(jié)束執(zhí)行docker-compose ps查看部署的容器及結(jié)果
[root@harbor harbor]# bash install.sh
[Step 0]: checking if docker is installed ...
Note: docker version: 19.03.11
[Step 1]: checking docker-compose is installed ...
Note: docker-compose version: 1.18.0
[Step 2]: loading Harbor images ...
517216c3ed59: Loading layer [==================================================>] 34.5MB/34.5MB
b9984883b90b: Loading layer [==================================================>] 331.5MB/331.5MB
979e8823fd48: Loading layer [==================================================>] 135.2kB/135.2kB
Loaded image: goharbor/harbor-migrator:v1.10.6
9af36121a2d3: Loading layer [==================================================>] 12.29MB/12.29MB
7469a23fd831: Loading layer [==================================================>] 42.51MB/42.51MB
78b8a6419717: Loading layer [==================================================>] 5.632kB/5.632kB
d82d965b4ef8: Loading layer [==================================================>] 40.45kB/40.45kB
df344cc92b3d: Loading layer [==================================================>] 42.51MB/42.51MB
986058fa362a: Loading layer [==================================================>] 2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v1.10.6
e57060edfa4f: Loading layer [==================================================>] 63.67MB/63.67MB
6b4f0118d2e7: Loading layer [==================================================>] 75.99MB/75.99MB
29a495a70a49: Loading layer [==================================================>] 5.632kB/5.632kB
79b703e7a4e5: Loading layer [==================================================>] 2.56kB/2.56kB
e867025af608: Loading layer [==================================================>] 2.56kB/2.56kB
808ee6ef9f3e: Loading layer [==================================================>] 2.56kB/2.56kB
9dd136dbdbb8: Loading layer [==================================================>] 2.56kB/2.56kB
07a93256363b: Loading layer [==================================================>] 10.75kB/10.75kB
Loaded image: goharbor/harbor-db:v1.10.6
d88d1d4c1e75: Loading layer [==================================================>] 8.515MB/8.515MB
4af0316b2679: Loading layer [==================================================>] 3.584kB/3.584kB
7ffcfabe0b2f: Loading layer [==================================================>] 20.51MB/20.51MB
b2c9e51892ed: Loading layer [==================================================>] 3.072kB/3.072kB
69f04e0492e5: Loading layer [==================================================>] 8.662MB/8.662MB
0befdda31216: Loading layer [==================================================>] 29.99MB/29.99MB
Loaded image: goharbor/harbor-registryctl:v1.10.6
0285f3f1d24f: Loading layer [==================================================>] 10.36MB/10.36MB
Loaded image: goharbor/nginx-photon:v1.10.6
d603a9558e8c: Loading layer [==================================================>] 115.4MB/115.4MB
b23664baab69: Loading layer [==================================================>] 12.15MB/12.15MB
e785164697c1: Loading layer [==================================================>] 3.072kB/3.072kB
ddc72a1554c7: Loading layer [==================================================>] 49.15kB/49.15kB
1bbe40c913b2: Loading layer [==================================================>] 3.584kB/3.584kB
8292a27c053f: Loading layer [==================================================>] 13.03MB/13.03MB
Loaded image: goharbor/clair-photon:v1.10.6
55cfd42af466: Loading layer [==================================================>] 8.515MB/8.515MB
2ef32a6672e4: Loading layer [==================================================>] 9.71MB/9.71MB
516c0b3f7b7d: Loading layer [==================================================>] 9.71MB/9.71MB
Loaded image: goharbor/clair-adapter-photon:v1.10.6
24717191d0eb: Loading layer [==================================================>] 10.36MB/10.36MB
e724890b1092: Loading layer [==================================================>] 7.697MB/7.697MB
a6186df96eea: Loading layer [==================================================>] 223.2kB/223.2kB
9dabcd61b9a2: Loading layer [==================================================>] 195.1kB/195.1kB
70a9eabcf0cf: Loading layer [==================================================>] 15.36kB/15.36kB
6327d3371dbe: Loading layer [==================================================>] 3.584kB/3.584kB
Loaded image: goharbor/harbor-portal:v1.10.6
cbb4f0baa930: Loading layer [==================================================>] 74.79MB/74.79MB
9cff4188aa57: Loading layer [==================================================>] 3.584kB/3.584kB
c2b996a82566: Loading layer [==================================================>] 3.072kB/3.072kB
9829c3787744: Loading layer [==================================================>] 2.56kB/2.56kB
67ddfe98d90a: Loading layer [==================================================>] 3.072kB/3.072kB
ff8b78cb63e7: Loading layer [==================================================>] 3.584kB/3.584kB
854dabac3758: Loading layer [==================================================>] 12.29kB/12.29kB
783d10970f24: Loading layer [==================================================>] 5.632kB/5.632kB
Loaded image: goharbor/harbor-log:v1.10.6
13fd12d41fea: Loading layer [==================================================>] 8.515MB/8.515MB
7f93018d3462: Loading layer [==================================================>] 3.584kB/3.584kB
59f5dbf0affb: Loading layer [==================================================>] 3.072kB/3.072kB
9311a8fa1514: Loading layer [==================================================>] 20.51MB/20.51MB
614eb395ab40: Loading layer [==================================================>] 21.33MB/21.33MB
Loaded image: goharbor/registry-photon:v1.10.6
57010d02c56f: Loading layer [==================================================>] 8.509MB/8.509MB
3742ae96b00e: Loading layer [==================================================>] 6.239MB/6.239MB
216c0af522f9: Loading layer [==================================================>] 14.43MB/14.43MB
09f6f8766ff3: Loading layer [==================================================>] 27.97MB/27.97MB
bffe3cc7a54c: Loading layer [==================================================>] 22.02kB/22.02kB
a60206e2897f: Loading layer [==================================================>] 48.64MB/48.64MB
Loaded image: goharbor/notary-signer-photon:v1.10.6
bbb3a4666fca: Loading layer [==================================================>] 12.29MB/12.29MB
3e7661db06ab: Loading layer [==================================================>] 49.38MB/49.38MB
Loaded image: goharbor/harbor-jobservice:v1.10.6
762f7fd02c27: Loading layer [==================================================>] 98.91MB/98.91MB
738675c5bb2a: Loading layer [==================================================>] 3.072kB/3.072kB
957f6ceb4893: Loading layer [==================================================>] 59.9kB/59.9kB
2ce8a82769c7: Loading layer [==================================================>] 61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v1.10.6
842f97800f3d: Loading layer [==================================================>] 94.53MB/94.53MB
e0d5dfe658d5: Loading layer [==================================================>] 49.92MB/49.92MB
f2d4a8662d68: Loading layer [==================================================>] 2.56kB/2.56kB
5f622d209603: Loading layer [==================================================>] 1.536kB/1.536kB
01094f024495: Loading layer [==================================================>] 157.2kB/157.2kB
f1b9e5269785: Loading layer [==================================================>] 3.017MB/3.017MB
Loaded image: goharbor/prepare:v1.10.6
1f0f4177bbb7: Loading layer [==================================================>] 15.84MB/15.84MB
baf2ca0b445a: Loading layer [==================================================>] 27.97MB/27.97MB
515bf108ac7d: Loading layer [==================================================>] 22.02kB/22.02kB
c2146de85e39: Loading layer [==================================================>] 50.05MB/50.05MB
Loaded image: goharbor/notary-server-photon:v1.10.6
fe546267e7cd: Loading layer [==================================================>] 8.514MB/8.514MB
7de41eeff949: Loading layer [==================================================>] 67.49MB/67.49MB
3204c9160f30: Loading layer [==================================================>] 3.072kB/3.072kB
cb68e6e1c61e: Loading layer [==================================================>] 3.584kB/3.584kB
3b277bfd90f8: Loading layer [==================================================>] 68.32MB/68.32MB
Loaded image: goharbor/chartmuseum-photon:v1.10.6
[Step 3]: preparing environment ...
[Step 4]: preparing harbor configs ...
prepare base dir is set to /root/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /secret/keys/secretkey
Creating harbor-log ... done
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir
Creating registry ... done
Creating harbor-core ... done
Creating network "harbor_harbor" with the default driver
Creating nginx ... done
Creating redis ...
Creating registryctl ...
Creating harbor-portal ...
Creating harbor-db ...
Creating registry ...
Creating harbor-core ...
Creating harbor-jobservice ...
Creating nginx ...
? ----Harbor has been installed and started successfully.----
[root@harbor harbor]# docker-compose ps
Name Command State Ports
--------------------------------------------------------------------------------------
harbor-core /harbor/harbor_core Up
harbor-db /docker-entrypoint.sh Up 5432/tcp
harbor-jobservice /harbor/harbor_jobservice ... Up
harbor-log /bin/sh -c /usr/local/bin/ ... Up 127.0.0.1:1514->10514/tcp
harbor-portal nginx -g daemon off; Up 8080/tcp
nginx nginx -g daemon off; Up 0.0.0.0:80->8080/tcp
redis redis-server /etc/redis.conf Up 6379/tcp
registry /home/harbor/entrypoint.sh Up 5000/tcp
registryctl /home/harbor/start.sh Up
- 修改兩個(gè)節(jié)點(diǎn)的daemon.json配置，添加私有鏡像倉(cāng)庫(kù)地址，并重啟docker服務(wù)。Tips：所有節(jié)點(diǎn)都要配置！
[root@harbor docker]# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://registry.cn-hangzhou.aliyuncs.com"],
"insecure-registries": ["harbor.registry"],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2",
"storage-opts": [
"overlay2.override_kernel_check=true"
]
}
[root@harbor docker]# systemctl daemon-reload
[root@harbor docker]# systemctl restart docker
如果在重啟docker的時(shí)候遇到報(bào)錯(cuò)，請(qǐng)仔細(xì)檢查daemon.json的配置，確認(rèn)無(wú)誤之后reload配置文件再次重啟docker嘗試。、
- 每個(gè)節(jié)點(diǎn)使用
docker login harbor.registry命令登陸harbor，之后就可以進(jìn)行pull、tag和push的操作了。
[root@harbor harbor]# docker login harbor.registry
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@harbor harbor]#
Tips：如果在login的時(shí)候遇到報(bào)錯(cuò)
Error response from daemon: Get harbor.registry/v2/: dial tcp 192.168.199.32:80: connect: connection refused，可能存在以下兩種情況：
（1）檢查/etc/docker/daemon.json中配置的"insecure-registries"參數(shù)是否是正確的harbor主機(jī)名或IP地址。
（2）harbor服務(wù)異常，切回harbor目錄并通過(guò)docker-compose ps查看各harbor容器狀態(tài)，如異常情況執(zhí)行docker-compose restart即可。注：操作必須要在harbor目錄才行。
（3）查看harbar的harbor.yml中的hostname配置的對(duì)不對(duì)
harbor異常
docker-compose restart
在master節(jié)點(diǎn)上的login
至此，所有的harbor搭建階段已經(jīng)完成，最后看一下harbor的頁(yè)面訪問(wèn)效果。
harbor登陸界面
登陸首頁(yè)
5. 測(cè)試
使用之前部署的kubernetes集群的master作為測(cè)試節(jié)點(diǎn)，向harbor推送鏡像，并使用harbor節(jié)點(diǎn)pull到本地。
-
在harbor頁(yè)面新建項(xiàng)目
新建項(xiàng)目 在master節(jié)點(diǎn)登陸harbor
[root@master ~]# docker login harbor.registry
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[root@master ~]#
- 使用
docker image ls查看自身鏡像，此次挑選calico/node鏡像作為試驗(yàn)。
docker image ls - 使用
docker tag打完鏡像標(biāo)簽， 并使用docker push推送到harbor服務(wù)器上
[root@master ~]# docker tag calico/node:v3.13.1 harbor.registry/calico/node:v3.13.1
[root@master ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.aliyuncs.com/k8sxio/kube-proxy v1.19.2 d373dd5a8593 2 months ago 118MB
registry.aliyuncs.com/k8sxio/kube-apiserver v1.19.2 607331163122 2 months ago 119MB
registry.aliyuncs.com/k8sxio/kube-controller-manager v1.19.2 8603821e1a7a 2 months ago 111MB
registry.aliyuncs.com/k8sxio/kube-scheduler v1.19.2 2f32d66b884f 2 months ago 45.7MB
registry.aliyuncs.com/k8sxio/etcd 3.4.13-0 0369cf4303ff 2 months ago 253MB
registry.aliyuncs.com/k8sxio/coredns 1.7.0 bfe3a36ebd25 5 months ago 45.2MB
calico/node v3.13.1 2e5029b93d4a 8 months ago 260MB
harbor/calico/node v3.13.1 2e5029b93d4a 8 months ago 260MB
harbor.registry/calico/node v3.13.1 2e5029b93d4a 8 months ago 260MB
calico/pod2daemon-flexvol v3.13.1 e8c600448aae 8 months ago 111MB
calico/cni v3.13.1 6912ec2cfae6 8 months ago 207MB
calico/kube-controllers v3.13.1 3971f13f2c6c 8 months ago 56.6MB
registry.aliyuncs.com/k8sxio/pause 3.2 80d28bedfe5d 9 months ago 683kB
[root@master ~]# docker push harbor.registry/calico/node:v3.13.1
The push refers to repository [harbor.registry/calico/node]
12fd6ed06c05: Pushed
fbde6cdd36b6: Pushed
cd59206a737d: Pushed
2e8e06e347e4: Pushed
0c9d8b0f21a3: Pushed
6861e9e920e0: Pushed
c5e173042e71: Pushed
8c2f5310e5b2: Pushed
783d3f7a3bc6: Pushed
676f013efbd6: Pushed
224fce06dca9: Pushed
27cd2023d60a: Pushed
4b52dfd1f9d9: Pushed
v3.13.1: digest: sha256:cbd5bf2ed8cb93595d358b6f23d3937da1620e8a600c93efbb29f689790b882b size: 3042
-
查看harbor的calico項(xiàng)目下的鏡像
web頁(yè)面的calico項(xiàng)目
Tips:
打標(biāo)簽：docekr tag 鏡像名稱(chēng):標(biāo)簽 你的IP:端口/harbor項(xiàng)目名稱(chēng)/鏡像名稱(chēng):標(biāo)簽 ，如：docker tag calico/node:v3.13.1 harbor.registry/calico/node:v3.13.1
推送：docker push 你的IP:端口/harbor項(xiàng)目名稱(chēng)/鏡像名稱(chēng):標(biāo)簽，如：docker push harbor.registry/calico/node:v3.13.1
刪除鏡像：docker rmi image_name:tag，如：docker rmi harbor/calico/node:v3.13.1
- 在harbar.registry的節(jié)點(diǎn)上作為客戶端，pull下剛才上傳的鏡像
[root@harbor harbor]# docker pull harbor.registry/calico/node:v3.13.1
v3.13.1: Pulling from calico/node
7fe5fcc0340e: Pull complete
086296bbdfc7: Pull complete
352a99f50574: Pull complete
8c35de76d622: Pull complete
90d1c845595c: Pull complete
a0138100ff24: Pull complete
d424c9255a01: Pull complete
9251f9ad4cf0: Pull complete
1895fa439d94: Pull complete
64b522df02c8: Pull complete
058b195a5d5d: Pull complete
996a1138b8e4: Pull complete
d2c562af5e2a: Pull complete
Digest: sha256:cbd5bf2ed8cb93595d358b6f23d3937da1620e8a600c93efbb29f689790b882b
Status: Downloaded newer image for harbor.registry/calico/node:v3.13.1
harbor.registry/calico/node:v3.13.1
[root@harbor harbor]#
