[TOC]

1、strongswan+vpp簡介

strongswan與vpp如何結合

我們使用的是VPP 20.01 版本 + strongswan 5.8.3版本編譯。
目前strongSwan+vpp的方案主要是使用strongswan的插件機制，替換strongswan的兩個默認插件

• 1、socket-default 該插件是IKE報文的socket backend。
• 2、kernel-netlink 該插件是IPSEC 數(shù)通backend

將默認的socket-default連接替換為VPP的punt socket方式，punt socket會將ike協(xié)議報文通過VPP上送到strongswan中，strongswan也會將回應的報文通過punt socket傳輸回vpp，IKE的協(xié)商層面是通過strongswan完成。
ike協(xié)商完成之后，strongswan通過vpp的C語言 API向VPP下發(fā)IPSEC的配置，sa spd 路由等等，下發(fā)完成配置之后，VPP的IPSEC隧道就完成了建立。
用strongswan替換VPP自身的IKE功能，是因為VPP本身的IKE只支持IKE V2而且功能的豐富度不如strongswan完善。

已有的開源項目簡介

作者matfabia

https://github.com/matfabia/strongswan/tree/vpp
該項目是strongswan+vpp這個方案的最初的項目，確定了strongswan與vpp結合的大致方向，完成度也比較高，后續(xù)的其他開源項目都是在此基礎上修修補補。
該項目在上傳最初代碼后，就停止更新了，代碼基于的VPP版本應該是v18.01左右

作者mestery

https://github.com/mestery/strongswan
該項目是基于上面原始項目進行修改，支持VPP 的1810版本，進行了小幅度的API適配整體和上面項目相差不大。隨后該項目也停止更新了，但是該項目中有一個pull request比較關鍵，在4500 UDP端口上支持了NAT-T IKE，增加了一些VPP新支持的加密算法例如GCM的支持。但是該pull request并沒有合入到該項目，因為作者可能已經忘記這個項目了。

作者rayshi-10

https://github.com/rayshi-10/Strongswan-Vpp2001
該項目是基于第二個項目做的，而且把第二個項目中pull request合入了進來，支持了VPP后來加入的更多加密和認證算法，而且支持NAT-T IKE。然后支持了VPP v20.01版本。這個版本的代碼修改量還是比較大的。因為VPP v20.01版本API和數(shù)據(jù)結構的改動是相當大的，大部分原有的IPSEC配置API都發(fā)生的變化，進行了多次重構，但是而且設置還刪除了一些配置屬性，導致原有的流程可以需要改動比較大才能適配。
v20.01的VPP ipv4 ipv6的配置需要顯示下發(fā)兩條，而以前的版本是使用any屬性標志下發(fā)一條就可以了。這部分的改動需要特別關注下，該項目目測這部分可能會有BUG?？梢蕴貏e關注下該項目的manage_policy函數(shù)，例如下面的部分，is_anyaddr的情況只下發(fā)了一條policy?？赡軙霈F(xiàn)問題

if (src->is_anyaddr(src) && dst->is_anyaddr(dst))
    {
        memset(mp->entry.local_address_stop.un.ip6, 0xFF, 16);
        memset(mp->entry.remote_address_stop.un.ip6, 0xFF, 16);
    } 

該項目基于的strongswan版本是5.6.x版本的，不是最新的release。

2、基于rayshi-10的代碼和strongswan最新release5.8.3進行修改

下載源碼

首先下載strongswan主線代碼，切換到5.8.3分支

git clone https://github.com/strongswan/strongswan.git
git checkout 5.8.3

然后下載rayshi-10 strongswan + vpp 20.01代碼

git clone https://github.com/rayshi-10/Strongswan-Vpp2001.git

替換文件

然后將該項目的

src/libcharon/plugins/kernel_vpp/
src/libcharon/plugins/socket_vpp/

兩個目錄替換到strongswan 5.8.3對應目錄下，然后將該項目configure.ac目錄下kernel-vpp socket-vpp相關的內存，添加到strongswan 5.8.3對應的文件里。
注，該項目的configure.ac里面缺少下面兩條配置

ADD_PLUGIN([kernel-vpp],           [c charon])
ADD_PLUGIN([socket-vpp],           [c charon])

需要將這兩條配置自行添加到configure.ac中的合適位置，
例如向下面的方式添加

ADD_PLUGIN([kernel-iph],           [c charon])
ADD_PLUGIN([kernel-vpp],           [c charon])
ADD_PLUGIN([kernel-pfkey],         [c charon starter nm cmd])
ADD_PLUGIN([kernel-pfroute],       [c charon starter nm cmd])
ADD_PLUGIN([kernel-netlink],       [c charon starter nm cmd])
ADD_PLUGIN([resolve],              [c charon cmd])
ADD_PLUGIN([save-keys],            [c])
ADD_PLUGIN([socket-default],       [c charon nm cmd])
ADD_PLUGIN([socket-dynamic],       [c charon cmd])
ADD_PLUGIN([socket-win],           [c charon])
ADD_PLUGIN([socket-vpp],           [c charon])
ADD_PLUGIN([bypass-lan],           [c charon nm cmd])

注意dnssec_status_t的修改

dnssec_status_t枚舉變量在strongswan vpp中進行了重命名，將這個枚舉中的變量全都加了DNSS前綴，可能是因為這個枚舉里面的變量和VPP里面的內容重名了，我們在替換時，如果編譯失敗了，可能是忘記重命名該名稱導致
重命名后的效果如下

enum dnssec_status_t {
    /**
     * The validating resolver has a trust anchor, has a chain of
     * trust, and is able to verify all the signatures in the response.
     * [RFC4033]
     */
    DNSS_SECURE,
    /**
     * The validating resolver has a trust anchor, a chain of
     * trust, and, at some delegation point, signed proof of the
     * non-existence of a DS record.  This indicates that subsequent
     * branches in the tree are provably insecure.  A validating resolver
     * may have a local policy to mark parts of the domain space as
     * insecure. [RFC4033]
     */
    DNSS_INSECURE,
    /**
     * The validating resolver has a trust anchor and a secure
     * delegation indicating that subsidiary data is signed, but the
     * response fails to validate for some reason: missing signatures,
     * expired signatures, signatures with unsupported algorithms, data
     * missing that the relevant NSEC RR says should be present, and so
     * forth. [RFC4033]
     */
    DNSS_BOGUS,
    /**
     * There is no trust anchor that would indicate that a
     * specific portion of the tree is secure.  This is the default
     * operation mode. [RFC4033]
     */
    DNSS_INDETERMINATE,
};

修改PUNT read socket path

在src/libcharon/plugins/socket_vpp/socket_vpp_socket.c中該項目中vpp的punt read path是/tmp目錄，該地址可以自行設定，例如我將該地址進行了下面的修改，和VPP其他unix socket放置在同一目錄

#define READ_PATH "/var/run/vpp/ike-punt-read.sock"

3、編譯項目

下載依賴

我的系統(tǒng)是Centos7，使用下面的命令下載編譯中的依賴項

yum install gperf
yum install python3
yum install gmp
yum install gmp-devel

編譯vpp

git clone https://github.com/FDio/vpp.git
git checkout v20.01
make install-dep
make build-releasels

將編譯好的VPP安裝到系統(tǒng)中

cp build-root/install-vpp-native/vpp/include/* /usr/include/ -r
cp build-root/install-vpp-native/vpp/lib/* /lib64/ -r
cp build-root/install-vpp-native/vpp/lib/vpp_plugins /lib/ -r
cp build-root/install-vpp-native/vpp/bin/vpp /usr/bin/
cp build-root/install-vpp-native/vpp/bin/vppctl /usr/bin/

編譯strongswan

預處理

最新版本的strongswan在centos下可能編譯不過，pkgconfig版本低，缺少PKG_CHECK_VAR
需要在configure.ac前面添加下面的定義

# backwards compat with older pkg-config
# - pull in AC_DEFUN from pkg.m4
m4_ifndef([PKG_CHECK_VAR], [
# PKG_CHECK_VAR(VARIABLE, MODULE, CONFIG-VARIABLE,
# [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND])
# -------------------------------------------
# Retrieves the value of the pkg-config variable for the given module.
AC_DEFUN([PKG_CHECK_VAR],
[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl
AC_ARG_VAR([$1], [value of $3 for $2, overriding pkg-config])dnl
_PKG_CONFIG([$1], [variable="][$3]["], [$2])
AS_VAR_COPY([$1], [pkg_cv_][$1])
AS_VAR_IF([$1], [""], [$5], [$4])dnl
])# PKG_CHECK_VAR
])

執(zhí)行編譯

./autogen.sh
./configure --enable-socket-vpp --enable-kernel-vpp --enable-libipsec --enable-openssl
make -j 8

將編譯好的strongswan安裝到系統(tǒng)中

make install

安裝好的默認目錄是/usr/local/，主要文件和目錄如下所示

/usr/local/bin/pki
/usr/local/sbin/ipsec
/usr/local/sbin/swanctl
/usr/local/sbin/etc/

4、運行測試

測試拓撲

目前我測試了strongswan官方文檔中拓撲的Roadwarrior Case和Site-to-Site Case，這兩種拓撲是移動客戶端場景和網關-網關的部署場景。 這里要注意，strongswan的配置方式可能發(fā)生了比較大的變化，之前是通過配置ipsec目錄下的內容去實現(xiàn)的，而最新的版本一般是用過配置swanctl相關的內容去完成。
官方項目中的README里面的內容是有些錯誤的，不要去看那部分的內容，例如Roadwarrior Case中網關和客戶端的配置中都寫的是：

local_ts  = 10.1.0.0/16

而顯然配置應該是服務器是local_ts = 10.1.0.0/16而客戶端應該是remote_ts = 10.1.0.0/16
這里應該去參考https://www.strongswan.org/testresults.html該頁面中swanctl的部分，這個頁面是測試例和執(zhí)行的情況，是隨著版本更新的，而且每發(fā)布一個版本會自動跑一遍這些部署場景，這里面有非常詳細的每個階段swanctl的配置和狀態(tài)，學習新的部署場景和配置應該從這個頁面去學習。

搭建方式

我是通過兩個qemu-kvm的虛擬機跑了兩個vpp + strongswan的VM環(huán)境，兩者的接口使用brctl通過內核橋連通。
strongswan+vpp的啟動順序，需要首先啟動VPP，然后配置好接口之后，啟動strongswan，然后啟動協(xié)商隧道。

vpp啟動配置

startup.conf

statseg
{
  default
  per-node-counters on
}
socksvr
{
    socket-name /var/run/vpp/vpp-api.sock
}
unix
{
  cli-listen /run/vpp/cli.sock
  log /tmp/vpe.log
  nodaemon
  coredump-size 1M
}
punt 
{ 
    socket /var/run/vpp/ike-punt-write.sock
}
api-trace { on }
heapsize 4G
buffers
{
    buffers-per-numa 40000
}
plugins
{
  plugin dpdk_plugin.so
  {
    enable
  }
}
cpu
{
  # Dynamic Create Option
  main-core 0
  corelist-workers 1-7
}
dpdk
{
  log-level debug
  huge-dir /dev/hugepages
  no-tx-checksum-offload
  dev 0000:00:06.0 { name G1/1 }
  dev 0000:00:07.0 { name G1/2 }
  
}

上面的配置CPU部分需要根據(jù)自己的環(huán)境編寫，綁定工作線程和主線程到某些CPU核。dpdk部分的接口PCI號，也需要根據(jù)實際的情況填寫，上面的配置ipsec加解密使用了openssl的能力，沒有使用dpdk的加解密套件，使用dpdk加解密套件請看最后一節(jié)。
上面配置中比較重要的一點是punt這一部分，該配置必須填寫。strongswan使用到了兩個punt socket，其中一個是VPP startup.conf中指定，是write socket，strongswan寫報文使用該unix socket。還有一個是punt讀接口，該unix socket在strongswan的socket-vpp插件中啟動時，動態(tài)向vpp注冊，接口的路徑在代碼中寫死。上面已經說過這個問題了。

vpp運行配置

VPP成功啟動后，需要配置接口的IP信息，這一部分信息就根據(jù)上面提到的測試例頁面中的拓撲來配置就可以。
下面是我的環(huán)境中site-to-site中配置CLI命令
moon的配置

vppctl set int state G1/1 up
vppctl set int state G1/2 up
vppctl set int state local0 up
vppctl set int ip addr G1/1 192.168.0.1/24
vppctl set int ip addr G1/2 10.1.0.1/16

sun的配置

vppctl set int state G1/1 up
vppctl set int state G1/2 up
vppctl set int state local0 up
vppctl set int ip addr G1/1 192.168.0.2/24
vppctl set int ip addr G1/2 10.2.0.1/16

制作證書

使用證書認證的話，就需要自己制作證書了，這一部分是比較復雜的，因為strongswan的證書驗證比較嚴格，而且根據(jù)網上教程制作出來的證書大多都是認證不通過的。我這里編寫了一個車腳本去制作這兩個場景的證書。使用了strongswan的pki這個工具，在制作證書之前，需要先下載一個程序，如果不下載該程序的話，可能會生成密鑰非常慢，長達幾十分鐘。

yum install haveged
systemctl start haveged

證書的制作要點就是配置中的id和證書中的CN以及SAN選項要一致

Roadwarrior Case

10.1.0.0/16 -- | 192.168.0.1 | === | x.x.x.x |
  moon-net          moon              carol

該腳本是跑在Roadwarrior Case中的移動客戶端carol上面的，在此場景中我們carol的管理IP是192.168.199.102，moon的管理IP是192.168.199.101，證書生成之后自動拷貝的，也可以手動拷貝到對應的目錄

#!/bin/bash

INSTALLDIR="/usr/local"
CONFIGPATH="$INSTALLDIR/etc"
SERVER_HOST=moon.strongswan.org
SERVER_IP=192.168.0.1
CLIENT_HOST=carol@strongswan.org
CLIENT_IP=192.168.0.2

# remove old files
rm -rf cert > /dev/null 2>&1
mkdir cert && cd cert

# create CA certificate
echo -e "\033[32mCreate CA certificate...\033[0m"
pki --gen --outform pem > ca.key.pem
pki --self --in ca.key.pem --dn "C=CN, O=StrongSwan, CN=StrongSwan CA" --ca --outform pem > ca.cert.pem

# create server certificate
echo -e "\033[32mCreate server certificate...\033[0m"
pki --gen --outform pem > server.key.pem
pki --pub --in server.key.pem | ipsec pki --issue --cacert ca.cert.pem \
  --cakey ca.key.pem --dn "C=CN, O=StrongSwan, CN=$SERVER_HOST" \
  --san "$SERVER_HOST" --san="$SERVER_IP" --flag serverAuth --flag ikeIntermediate \
  --outform pem > server.cert.pem

# create client certificate
echo -e "\033[32mCreate client certificate...\033[0m"
pki --gen --outform pem > client.key.pem
pki --pub --in client.key.pem | ipsec pki --issue --cacert ca.cert.pem \
  --cakey ca.key.pem --dn "C=CN, O=StrongSwan, CN=carol@strongswan.org" \
  --san "$CLIENT_HOST" --san="CLIENT_IP" \
  --outform pem > client.cert.pem

echo -e "\033[32mInstall certificate...\033[0m"
cp ca.cert.pem $CONFIGPATH/swanctl/x509ca/strongswanCert.pem 
cp client.cert.pem $CONFIGPATH/swanctl/x509/carolCert.pem 
cp client.key.pem $CONFIGPATH/swanctl/private/carolKey.pem 

sshpass -pnsfocus scp ca.cert.pem 192.168.199.101:$CONFIGPATH/swanctl/x509ca/strongswanCert.pem 
sshpass -pnsfocus scp server.cert.pem 192.168.199.101:$CONFIGPATH/swanctl/x509/moonCert.pem
sshpass -pnsfocus scp server.key.pem 192.168.199.101:$CONFIGPATH/swanctl/private/moonKey.pem

site-to-site case

10.1.0.0/16 -- | 192.168.0.1 | === | 192.168.0.2 | -- 10.2.0.0/16
  moon-net          moon                 sun           sun-net

#!/bin/bash

INSTALLDIR="/usr/local"
CONFIGPATH="$INSTALLDIR/etc"
SERVER_HOST=moon.strongswan.org
SERVER_IP=192.168.0.1
CLIENT_HOST=sun.strongswan.org
CLIENT_IP=192.168.0.2

# remove old files
rm -rf cert > /dev/null 2>&1
mkdir cert && cd cert

# create CA certificate
echo -e "\033[32mCreate CA certificate...\033[0m"
pki --gen --outform pem > ca.key.pem
pki --self --in ca.key.pem --dn "C=CN, O=StrongSwan, CN=StrongSwan CA" --ca --outform pem > ca.cert.pem

# create server certificate
echo -e "\033[32mCreate server certificate...\033[0m"
pki --gen --outform pem > server.key.pem
pki --pub --in server.key.pem | ipsec pki --issue --cacert ca.cert.pem \
  --cakey ca.key.pem --dn "C=CN, O=StrongSwan, CN=$SERVER_HOST" \
  --san "$SERVER_HOST" --san="$SERVER_IP" --flag serverAuth --flag ikeIntermediate \
  --outform pem > server.cert.pem

# create client certificate
echo -e "\033[32mCreate client certificate...\033[0m"
pki --gen --outform pem > client.key.pem
pki --pub --in client.key.pem | ipsec pki --issue --cacert ca.cert.pem \
  --cakey ca.key.pem --dn "C=CN, O=StrongSwan, CN=$CLIENT_HOST" \
  --san "$CLIENT_HOST" --san="CLIENT_IP" \
  --outform pem > client.cert.pem


echo -e "\033[32mInstall certificate...\033[0m"
cp ca.cert.pem $CONFIGPATH/swanctl/x509ca/strongswanCert.pem 
cp client.cert.pem $CONFIGPATH/swanctl/x509/carolCert.pem 
cp client.key.pem $CONFIGPATH/swanctl/private/carolKey.pem 
sshpass -pnsfocus scp ca.cert.pem 192.168.199.101:$CONFIGPATH/swanctl/x509ca/strongswanCert.pem 
sshpass -pnsfocus scp server.cert.pem 192.168.199.101:$CONFIGPATH/swanctl/x509/moonCert.pem
sshpass -pnsfocus scp server.key.pem 192.168.199.101:$CONFIGPATH/swanctl/private/moonKey.pem

配置strongswan

開啟vpp插件

在進行配置之前，需要先啟用我們的kernel-vpp和socket-vpp插件。首先我們將

/usr/local/etc/strongswan.d/charon/kernel-netlink.conf 
/usr/local/etc/strongswan.d/charon/socket-default.conf 

兩個默認插件的內容修改一下將默認加載變成不加載
load = no
然后將我們新增的兩個插件加載狀態(tài)變?yōu)閥es

/usr/local/etc/strongswan.d/charon/socket-vpp.conf 
/usr/local/etc/strongswan.d/charon/kernel-vpp.conf

修改為load = yes
strongswan新版本，我們配置的內容主要是/usr/local/etc/swanctl/swanctl.conf文件，具體的場景和配置可以參考上面給出的官方測試例的配置。

Roadwarrior Case配置

10.1.0.0/16 -- | 192.168.0.1 | === | x.x.x.x |
  moon-net          moon              carol

網關 moon端配置

證書位置：

/usr/local/etc/swanctl/x509ca/strongswanCert.pem
/usr/local/etc/swanctl/x509/moonCert.pem
/usr/local/etc/swanctl/private/moonKey.pem

swanctl配置

/usr/local/etc/swanctl/swanctl.conf:
connections {
    rw {
        local {
            auth = pubkey
            certs = moonCert.pem
            id = moon.strongswan.org
        }
        remote {
            auth = pubkey
        }
        children {
            net-net {
                local_ts  = 10.1.0.0/16
            }
        }
     version = 2
     proposals = aes128-sha256-curve25519
    }
}

vpp 運行配置：

vppctl set int state G1/1 up
vppctl set int state G1/2 up
vppctl set int state local0 up
vppctl set int ip addr G1/1 192.168.0.2/24
vppctl set int ip addr G1/2 10.2.0.1/16

移動端carol配置

證書位置：

/usr/local/etc/swanctl/x509ca/strongswanCert.pem
/usr/local/etc/swanctl/x509/carolCert.pem
/usr/local/etc/swanctl/private/carolKey.pem

swanctl配置：

/usr/local/etc/swanctl/swanctl.conf:
connections {
    home {
        remote_addrs = moon.strongswan.org
        local_addrs = 192.168.0.2

        local {
            auth = pubkey
            certs = carolCert.pem
            id = carol@strongswan.org
        }
        remote {
            auth = pubkey
            id = moon.strongswan.org
        }
        children {
            home {
                remote_ts  = 10.1.0.0/16
                start_action = start
            }
        }
        version = 2
        proposals = aes128-sha256-curve25519
    }
}

vpp運行配置：

vppctl set int state G1/1 up
vppctl set int state G1/2 up
vppctl set int state local0 up
vppctl set int ip addr G1/1 192.168.0.2/24

site-to-site配置

10.1.0.0/16 -- | 192.168.0.1 | === | 192.168.0.2 | -- 10.2.0.0/16
  moon-net          moon                 sun           sun-net

site1 moon配置

證書位置

/usr/local/etc/swanctl/x509ca/strongswanCert.pem
/usr/local/etc/swanctl/x509/moonCert.pem
/usr/local/etc/swanctl/private/moonKey.pem

swanctl配置

/usr/local/etc/swanctl/swanctl.conf:
connections {

   gw-gw {
      local_addrs  = 192.168.0.1
      remote_addrs = 192.168.0.2

      local {
         auth = pubkey
         certs = moonCert.pem
         id = moon.strongswan.org
      }
      remote {
         auth = pubkey
         id = sun.strongswan.org
      }
      children {
         net-net {
            local_ts  = 10.1.0.0/16
            remote_ts = 10.2.0.0/16
            
            rekey_time = 5400
            rekey_bytes = 500000000
            rekey_packets = 1000000
            esp_proposals = aes128gcm128-sha256
         }
      }
      version = 2
      mobike = no
      reauth_time = 10800
      proposals = aes128-sha256-x25519
   }
}

vpp配置

vppctl set int state G1/1 up
vppctl set int state G1/2 up
vppctl set int state local0 up
vppctl set int ip addr G1/1 192.168.0.1/24
vppctl set int ip addr G1/2 10.1.0.1/16

site2 sun配置

證書位置：

/usr/local/etc/swanctl/x509ca/strongswanCert.pem
/usr/local/etc/swanctl/x509/carolCert.pem
/usr/local/etc/swanctl/private/carolKey.pem

swanctl配置

/usr/local/etc/swanctl/swanctl.conf:
connections {

   gw-gw {
      local_addrs  = 192.168.0.2
      remote_addrs = 192.168.0.1

      local {
         auth = pubkey
         certs = carolCert.pem
         id = sun.strongswan.org
      }
      remote {
         auth = pubkey
         id = moon.strongswan.org
      }
      children {
         net-net {
            local_ts  = 10.2.0.0/16
            remote_ts = 10.1.0.0/16

            rekey_time = 5400
            rekey_bytes = 500000000
            rekey_packets = 1000000
            esp_proposals = aes128gcm128-sha256
         }
      }
      version = 2
      mobike = no
      reauth_time = 10800
      proposals = aes128-sha256-x25519
   }
}

vpp配置

vppctl set int state G1/1 up
vppctl set int state G1/2 up
vppctl set int state local0 up
vppctl set int ip addr G1/1 192.168.0.2/24
vppctl set int ip addr G1/2 10.2.0.1/16

開始運行

首先啟動VPP，配置好strongswan的配置和VPP的配置，然后兩端都使用systemctl start strongswan-starter.service啟動strongswan
可以使用swanctl --stats命令查看一下vpp的插件加載是否正確，在/var/log/messages文件中查看日志是否有報錯等等。
然后查看一下VPP端，strongswan是否已經建立好了連接，如果建立成功之后，vpp中應該會有如下的顯示

[root@localhost home]# vppctl show api clients
Shared memory clients
                Name      PID   Queue Length           Queue VA Health
          strongswan    14102              0 0x00000001301ce9c0 OK
[root@localhost home]# vppctl show udp punt
IPV4 UDP ports punt : 500, 4500
IPV6 UDP ports punt : 500, 4500

在兩端執(zhí)行swanctl --load-all加載所有的配置和證書。
在協(xié)商的發(fā)起端執(zhí)行初始化命令，這個net-net是根據(jù)當前的swanctl.conf配置文件中children字段里面的內容填寫的。

swanctl --initiate --child net-net
或者
swanctl --initiate --child home

查看日志/var/log/messages是否成功，如果有類似如下的顯示，則證明IKE協(xié)商成功了

Apr  8 18:05:06 localhost ipsec: 08[CFG] initiating 'home'
Apr  8 18:05:06 localhost ipsec: 08[IKE] initiating IKE_SA home[1] to 192.168.0.1
Apr  8 18:05:06 localhost ipsec: 08[ENC] generating IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) N(HASH_ALG) N(REDIR_SUP) ]
Apr  8 18:05:06 localhost ipsec: 08[NET] sending packet: from 192.168.0.2[500] to 192.168.0.1[500] (240 bytes)
Apr  8 18:05:06 localhost ipsec: 05[NET] sending vpp packet: from 192.168.0.2[500] to 192.168.0.1[500] by sock 8
Apr  8 18:05:06 localhost charon: 16[ENC] parsed IKE_AUTH response 1 [ IDr CERT AUTH SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) ]
Apr  8 18:05:06 localhost ipsec: 10[NET] received packet: from 192.168.0.1[500] to 192.168.0.2[500] (273 bytes)
Apr  8 18:05:06 localhost ipsec: 10[ENC] parsed IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(HASH_ALG) N(CHDLESS_SUP) N(MULT_AUTH) ]
Apr  8 18:05:06 localhost ipsec: 10[CFG] selected proposal: IKE:AES_CBC_128/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/CURVE_25519
Apr  8 18:05:06 localhost ipsec: 10[IKE] received cert request for "C=CN, O=StrongSwan, CN=StrongSwan CA"
Apr  8 18:05:06 localhost ipsec: 10[IKE] sending cert request for "C=CN, O=StrongSwan, CN=StrongSwan CA"
Apr  8 18:05:06 localhost ipsec: 10[IKE] authentication of 'carol@strongswan.org' (myself) with RSA_EMSA_PKCS1_SHA2_256 successful
Apr  8 18:05:06 localhost ipsec: 10[IKE] sending end entity cert "C=CN, O=StrongSwan, CN=carol@strongswan.org"
Apr  8 18:05:06 localhost ipsec: 10[IKE] establishing CHILD_SA home{1}
Apr  8 18:05:06 localhost ipsec: 10[ENC] generating IKE_AUTH request 1 [ IDi CERT N(INIT_CONTACT) CERTREQ IDr AUTH SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) N(MULT_AUTH) N(EAP_ONLY) N(MSG_ID_SYN_SUP) ]
Apr  8 18:05:06 localhost ipsec: 10[ENC] splitting IKE message (1488 bytes) into 2 fragments
Apr  8 18:05:06 localhost ipsec: 10[ENC] generating IKE_AUTH request 1 [ EF(1/2) ]
Apr  8 18:05:06 localhost ipsec: 10[ENC] generating IKE_AUTH request 1 [ EF(2/2) ]
Apr  8 18:05:06 localhost ipsec: 10[NET] sending packet: from 192.168.0.2[500] to 192.168.0.1[4500] (1252 bytes)
Apr  8 18:05:06 localhost ipsec: 10[NET] sending packet: from 192.168.0.2[500] to 192.168.0.1[4500] (308 bytes)
Apr  8 18:05:06 localhost ipsec: 05[NET] sending vpp packet: from 192.168.0.2[500] to 192.168.0.1[4500] by sock 8
Apr  8 18:05:06 localhost ipsec: 05[NET] sending vpp packet: from 192.168.0.2[500] to 192.168.0.1[4500] by sock 8
Apr  8 18:05:06 localhost ipsec: 11[NET] received packet: from 192.168.0.1[4500] to 192.168.0.2[500] (1252 bytes)
Apr  8 18:05:06 localhost ipsec: 11[ENC] parsed IKE_AUTH response 1 [ EF(1/2) ]
Apr  8 18:05:06 localhost ipsec: 11[ENC] received fragment #1 of 2, waiting for complete IKE message
Apr  8 18:05:06 localhost ipsec: 16[NET] received packet: from 192.168.0.1[4500] to 192.168.0.2[500] (164 bytes)
Apr  8 18:05:06 localhost ipsec: 16[ENC] parsed IKE_AUTH response 1 [ EF(2/2) ]
Apr  8 18:05:06 localhost ipsec: 16[ENC] received fragment #2 of 2, reassembled fragmented IKE message (1344 bytes)
Apr  8 18:05:06 localhost ipsec: 16[ENC] parsed IKE_AUTH response 1 [ IDr CERT AUTH SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) ]
Apr  8 18:05:06 localhost ipsec: 16[IKE] received end entity cert "C=CN, O=StrongSwan, CN="
Apr  8 18:05:06 localhost ipsec: 16[CFG]   using certificate "C=CN, O=StrongSwan, CN="
Apr  8 18:05:06 localhost ipsec: 16[CFG]   using trusted ca certificate "C=CN, O=StrongSwan, CN=StrongSwan CA"
Apr  8 18:05:06 localhost charon: 16[IKE] received end entity cert "C=CN, O=StrongSwan, CN="
Apr  8 18:05:06 localhost ipsec: 16[CFG] checking certificate status of "C=CN, O=StrongSwan, CN="
Apr  8 18:05:06 localhost ipsec: 16[CFG] certificate status is not available
Apr  8 18:05:06 localhost charon: 16[CFG]   using certificate "C=CN, O=StrongSwan, CN="
Apr  8 18:05:06 localhost charon: 16[CFG]   using trusted ca certificate "C=CN, O=StrongSwan, CN=StrongSwan CA"
Apr  8 18:05:06 localhost charon: 16[CFG] checking certificate status of "C=CN, O=StrongSwan, CN="
Apr  8 18:05:06 localhost charon: 16[CFG] certificate status is not available
Apr  8 18:05:06 localhost charon: 16[CFG]   reached self-signed root ca with a path length of 0
Apr  8 18:05:06 localhost charon: 16[IKE] authentication of 'moon.strongswan.org' with RSA_EMSA_PKCS1_SHA2_256 successful
Apr  8 18:05:06 localhost charon: 16[IKE] IKE_SA home[1] established between 192.168.0.2[carol@strongswan.org]...192.168.0.1[moon.strongswan.org]
Apr  8 18:05:06 localhost charon: 16[IKE] scheduling rekeying in 14049s
Apr  8 18:05:06 localhost charon: 16[IKE] maximum IKE_SA lifetime 15489s
Apr  8 18:05:06 localhost charon: 16[CFG] selected proposal: ESP:AES_CBC_128/HMAC_SHA2_256_128/NO_EXT_SEQ
Apr  8 18:05:06 localhost charon: 16[KNL] firstly created, spd for G1/1 found sw_if_index is 1
Apr  8 18:05:07 localhost charon: 16[IKE] CHILD_SA home{1} established with SPIs c079e4f7_i c8e6f92e_o and TS 192.168.0.2/32 === 10.1.0.0/16
Apr  8 18:05:07 localhost charon: 16[IKE] peer supports MOBIKE

此時如果你查看vpp的配置應該可以看到ipsec相關的內容

[root@localhost home]# vppctl show ipsec all
[0] sa 1 (0x1) spi 4158945728 (0xf7e479c0) protocol:esp flags:[tunnel ]
[1] sa 2 (0x2) spi 788129480 (0x2ef9e6c8) protocol:esp flags:[tunnel ]
spd 1
 ip4-outbound:
   [1] priority 2147483647 action bypass type ip4-outbound protocol IPSEC_AH
     local addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     remote addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     packets 0 bytes 0
   [3] priority 2147483647 action bypass type ip4-outbound protocol IPSEC_ESP
     local addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     remote addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     packets 0 bytes 0
   [5] priority 2147483647 action bypass type ip4-outbound protocol UDP
     local addr range 0.0.0.0 - 255.255.255.255 port range 500 - 500
     remote addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     packets 0 bytes 0
   [7] priority 2147483647 action bypass type ip4-outbound protocol UDP
     local addr range 0.0.0.0 - 255.255.255.255 port range 4500 - 4500
     remote addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     packets 0 bytes 0
   [10] priority 2147480764 action protect type ip4-outbound protocol any sa 2
     local addr range 192.168.0.2 - 192.168.0.2 port range 0 - 65535
     remote addr range 10.1.0.0 - 10.1.255.255 port range 0 - 65535
     packets 0 bytes 0
 ip6-outbound:
 ip4-inbound-protect:
   [8] priority 2147480764 action protect type ip4-inbound-protect protocol any sa 1
     local addr range 192.168.0.2 - 192.168.0.2 port range 0 - 65535
     remote addr range 10.1.0.0 - 10.1.255.255 port range 0 - 65535
     packets 0 bytes 0
   [9] priority 2147480764 action protect type ip4-inbound-protect protocol any sa 1
     local addr range 192.168.0.2 - 192.168.0.2 port range 0 - 65535
     remote addr range 10.1.0.0 - 10.1.255.255 port range 0 - 65535
     packets 0 bytes 0
 ip6-inbound-protect:
 ip4-inbound-bypass:
   [0] priority 2147483647 action bypass type ip4-inbound-bypass protocol IPSEC_AH
     local addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     remote addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     packets 0 bytes 0
   [2] priority 2147483647 action bypass type ip4-inbound-bypass protocol IPSEC_ESP
     local addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     remote addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     packets 0 bytes 0
   [4] priority 2147483647 action bypass type ip4-inbound-bypass protocol UDP
     local addr range 0.0.0.0 - 255.255.255.255 port range 500 - 500
     remote addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     packets 0 bytes 0
   [6] priority 2147483647 action bypass type ip4-inbound-bypass protocol UDP
     local addr range 0.0.0.0 - 255.255.255.255 port range 4500 - 4500
     remote addr range 0.0.0.0 - 255.255.255.255 port range 0 - 65535
     packets 0 bytes 0
 ip6-inbound-bypass:
SPD Bindings:
  1 -> G1/1
[root@localhost home]# vppctl show ipsec sa detail
[0] sa 1 (0x1) spi 4158945728 (0xf7e479c0) protocol:esp flags:[tunnel ]
   locks 3
   salt 0x0
   thread-indices [encrypt:-1 decrypt:-1]
   seq 0 seq-hi 0
   last-seq 0 last-seq-hi 0 window 0000000000000000000000000000000000000000000000000000000000000000
   crypto alg aes-cbc-128 key [redacted]
   integrity alg sha-256-128 key [redacted]
   packets 0 bytes 0
   table-ID 0 tunnel src 192.168.0.1 dst 192.168.0.2
    resovle via fib-entry: 10
    stacked on:
      [@3]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:13 to:[3:1773]]
        [0] [@2]: dpo-receive: 192.168.0.2 on G1/1
[1] sa 2 (0x2) spi 788129480 (0x2ef9e6c8) protocol:esp flags:[tunnel ]
   locks 2
   salt 0x0
   thread-indices [encrypt:-1 decrypt:-1]
   seq 0 seq-hi 0
   last-seq 0 last-seq-hi 0 window 0000000000000000000000000000000000000000000000000000000000000000
   crypto alg aes-cbc-128 key [redacted]
   integrity alg sha-256-128 key [redacted]
   packets 0 bytes 0
   table-ID 0 tunnel src 192.168.0.2 dst 192.168.0.1
    resovle via fib-entry: 15
    stacked on:
      [@3]: dpo-load-balance: [proto:ip4 index:17 buckets:1 uRPF:18 to:[3:1884]]
        [0] [@5]: ipv4 via 192.168.0.1 G1/1: mtu:9000 525400023faf5254008ce41c0800
        

查看vpp的fib表，也動態(tài)添加了對應的路由進去，例如在Roadwarrior場景中，客戶端carol中自動添加的10.1.0.0/16的路由，通過192.168.0.1 G1/1出去

[root@localhost home]# vppctl show ip fib
ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto ] epoch:0 flags:none locks:[adjacency:1, default-route:1, nat-hi:2, ]
0.0.0.0/0
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:1 buckets:1 uRPF:0 to:[0:0]]
    [0] [@0]: dpo-drop ip4
0.0.0.0/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:2 buckets:1 uRPF:1 to:[0:0]]
    [0] [@0]: dpo-drop ip4
10.1.0.0/16
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:18 buckets:1 uRPF:20 to:[0:0]]
    [0] [@5]: ipv4 via 192.168.0.1 G1/1: mtu:9000 525400023faf5254008ce41c0800
10.2.0.0/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:14 buckets:1 uRPF:15 to:[0:0]]
    [0] [@0]: dpo-drop ip4
10.2.0.0/16
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:13 buckets:1 uRPF:14 to:[0:0]]
    [0] [@4]: ipv4-glean: G1/2: mtu:9000 ffffffffffff52540008713e0806
10.2.0.1/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:16 buckets:1 uRPF:19 to:[0:0]]
    [0] [@2]: dpo-receive: 10.2.0.1 on G1/2
10.2.255.255/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:15 buckets:1 uRPF:17 to:[0:0]]
    [0] [@0]: dpo-drop ip4
192.168.0.0/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:10 buckets:1 uRPF:9 to:[0:0]]
    [0] [@0]: dpo-drop ip4
192.168.0.1/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:17 buckets:1 uRPF:18 to:[3:1884]]
    [0] [@5]: ipv4 via 192.168.0.1 G1/1: mtu:9000 525400023faf5254008ce41c0800
192.168.0.0/24
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:8 to:[1:108]]
    [0] [@4]: ipv4-glean: G1/1: mtu:9000 ffffffffffff5254008ce41c0806
192.168.0.2/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:12 buckets:1 uRPF:13 to:[3:1773]]
    [0] [@2]: dpo-receive: 192.168.0.2 on G1/1
192.168.0.255/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:11 buckets:1 uRPF:11 to:[0:0]]
    [0] [@0]: dpo-drop ip4
224.0.0.0/4
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:4 buckets:1 uRPF:3 to:[0:0]]
    [0] [@0]: dpo-drop ip4
240.0.0.0/4
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:3 buckets:1 uRPF:2 to:[0:0]]
    [0] [@0]: dpo-drop ip4
255.255.255.255/32
  unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:5 buckets:1 uRPF:4 to:[0:0]]
    [0] [@0]: dpo-drop ip4

如果是Roadwarrior場景我們在移動客戶端carol ping 10.1.0.1可以成功ping通

vpp# ping 10.1.0.1
116 bytes from 10.1.0.1: icmp_seq=1 ttl=64 time=7.7229 ms
116 bytes from 10.1.0.1: icmp_seq=2 ttl=64 time=1.2671 ms
116 bytes from 10.1.0.1: icmp_seq=3 ttl=64 time=4.2904 ms
116 bytes from 10.1.0.1: icmp_seq=4 ttl=64 time=8.3667 ms
116 bytes from 10.1.0.1: icmp_seq=5 ttl=64 time=1.3370 ms

此時如果在中間抓包的話，應該看到的是ESP報文，我們查看vpp的節(jié)點統(tǒng)計，數(shù)據(jù)如下，說明IPSEC隧道建立成功了

vpp# show errors  
   Count                    Node                  Reason
         5          ipsec4-output-feature         IPSec policy protect
         5              esp4-encrypt              ESP pkts received
         5              esp4-decrypt              ESP pkts received
         5          ipsec4-input-feature          IPSEC pkts received
         5          ipsec4-input-feature          IPSEC pkts matched
         5          ipsec4-output-feature         IPSec policy bypass
         5              esp4-encrypt              ESP pkts received

同理，site-to-site的場景也可以通過這個方法來進行驗證。更多的場景應用可以通過上面說的官方測試例的網站來進行搭建

使用DPDK加解密套件

上面的VPP startup.conf配置文件使用了openssl的加解密套件，但是如果想要更高的處理性能的話，需要使用DPDK的加解密套件。添加方式是將上面的startup.conf文件dpdk部分修改為下面的配置添加dpdk加密虛擬設備vdev crypto_aesni_mb

dpdk
{
  log-level debug
  huge-dir /dev/hugepages
  vdev crypto_aesni_mb
  dev 0000:00:06.0 { name G1/1 }
  dev 0000:00:07.0 { name G1/2 }
}

添加完成之后，可以使用下面的命令查看

show dpdk crypto devices
show dpdk crypto
show ipsec backends 

查看DPDK解加密設備是否加載

vpp# show dpdk crypto devices 
crypto_aesni_mb          crypto_aesni_mb     up        
  numa_node 0, max_queues 8
  SYMMETRIC_CRYPTO, SYM_OPERATION_CHAINING, CPU_AVX2, CPU_AESNI, OOP_LB_IN_LB_OUT
  Cipher: aes-cbc-128, aes-cbc-192, aes-cbc-256, aes-ctr-128, aes-ctr-192, aes-ctr-256, aes-gcm-128, aes-gcm-192, aes-gcm-256
  Auth: md5-96, sha1-96, sha-256-96, sha-256-128, sha-384-192, sha-512-256
  enqueue 0          dequeue 0          enqueue_err 0          dequeue_err 0          
  free_resources 1 :
                     thr_id  -1 qp  7 enc_inflight 0, dec_inflights 0
  used_resources 7 :
                     thr_id   1 qp  0 enc_inflight 0, dec_inflights 0
                     thr_id   2 qp  1 enc_inflight 0, dec_inflights 0
                     thr_id   3 qp  2 enc_inflight 0, dec_inflights 0
                     thr_id   4 qp  3 enc_inflight 0, dec_inflights 0
                     thr_id   5 qp  4 enc_inflight 0, dec_inflights 0
                     thr_id   6 qp  5 enc_inflight 0, dec_inflights 0
                     thr_id   7 qp  6 enc_inflight 0, dec_inflights 0

show dpdk crypto placement查看dpdk加解密設備綁定的線程，此處需要注意一點，如果VPP采用多線程工作模式的話，dpdk的加密設備是不會綁定在vpp_main主線程的，而且加解密處理入口節(jié)點dpdk-crypto-input也只綁定在了工作線程。作為正常網關轉發(fā)是可以工作的，因為進來的ipsec報文都會走到dpdk-crypto-input節(jié)點處理，在工作線程處理，包含dpdk的解加密資源。
但是如果是類似于上面的操作，通過網關去直接ping對端的話，是無法通的，有如下報錯

dpdk-esp4-encrypt           Cipher/Auth not supported

查看/var/log/messages顯示以下的log

dpdk_esp_encrypt_inline:247: unsupported SA by thread index 0

猜測可能是ping操作類似于協(xié)議和控制報文處理，在vpp_main主線程處理，vpp_main是沒有分配dpdk的加解密資源的，所以ESP報文處理時，會報錯主線程查找不到SA。猜測這是官方的IPSEC的一個BUG，沒有考慮到此類場景。單線程工作模式不受影響。如果想要解決該問題，可以嘗試修改代碼，dpdk_ipsec_process該函數(shù)中加密算法注冊時，不進行skip_master操作，但是我沒有嘗試過。

vpp# show dpdk crypto placement 
Thread 1 (vpp_wk_0):
  crypto_aesni_mb      dev-id  0 queue-pair  0

Thread 2 (vpp_wk_1):
  crypto_aesni_mb      dev-id  0 queue-pair  1

Thread 3 (vpp_wk_2):
  crypto_aesni_mb      dev-id  0 queue-pair  2

Thread 4 (vpp_wk_3):
  crypto_aesni_mb      dev-id  0 queue-pair  3

Thread 5 (vpp_wk_4):
  crypto_aesni_mb      dev-id  0 queue-pair  4

Thread 6 (vpp_wk_5):
  crypto_aesni_mb      dev-id  0 queue-pair  5

Thread 7 (vpp_wk_6):
  crypto_aesni_mb      dev-id  0 queue-pair  6

查看ipsec加密后端是否變成dpdk backend

vpp# show ipsec backends 
IPsec AH backends available:
           Name                     Index             Active  
  crypto engine backend               0                 yes   
IPsec ESP backends available:
           Name                     Index             Active  
  crypto engine backend               0                 no    
       dpdk backend                   1                 yes