概述

?對于一個ceph開發(fā)人員來說編譯源碼以及打rpm是其必備技能。無論是fix bug還是向社區(qū)提交pull request都離不開編譯源碼。而rpm包又是linux系統(tǒng)發(fā)行產(chǎn)品時常用的格式。在官網(wǎng)和其他blog可以很多這方面的介紹，我在這里把自己編譯的步奏，以及遇到的坑及解決辦法記錄一下，僅供新人參考。

編譯源碼

環(huán)境介紹：

• ceph version: Luminous 12.2.10
• 硬件環(huán)境：Centos7虛擬機/Centos7 docker container

準備工作

• 安裝yum源
  這些源用于安裝常用工具，執(zhí)行install-reps.sh安裝以來包使用，列表如下：

-rw-r--r--. 1 root root 2523 Jan  8 07:10 CentOS-Base.repo
-rw-r--r--. 1 root root 1309 Jan  8 08:22 CentOS-CR.repo
-rw-r--r--. 1 root root  649 Nov 23 13:16 CentOS-Debuginfo.repo
-rw-r--r--. 1 root root  314 Nov 23 13:16 CentOS-fasttrack.repo
-rw-r--r--. 1 root root  630 Nov 23 13:16 CentOS-Media.repo
-rw-r--r--. 1 root root  916 Jan  8 07:10 CentOS-SCLo-scl.repo
-rw-r--r--. 1 root root  898 Jan  8 07:10 CentOS-SCLo-scl-rh.repo
-rw-r--r--. 1 root root 1331 Nov 23 13:16 CentOS-Sources.repo
-rw-r--r--. 1 root root 5701 Nov 23 13:16 CentOS-Vault.repo
-rw-r--r--. 1 root root  664 Jan  8 07:10 epel.repo
-rw-r--r--. 1 root root  951 Oct  2  2017 epel.repo.rpmnew
-rw-r--r--. 1 root root 1050 Oct  2  2017 epel-testing.repo
-rw-r--r--. 1 root root  155 Jan  8 07:10 mirrors.aliyun.com_epel_7_x86_64_.repo
-rw-r--r--. 1 root root  153 Jan  8 07:10 mirrors.aliyun.com_epel_x86_64_.repo

• 安裝git

yum install git

You have enabled checking of packages via GPG keys. This is a good thing.
However, you do not have any GPG public keys installed. You need to download
the keys for packages you wish to install and install them.
You can do that by running the command:
    rpm --import public.gpg.key

從別的機器拷貝/etc/ceph/rpm-gpg 過來,然后執(zhí)行
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

• 下載12.2.10源碼
  git clone -b v12.2.10 --single-branch git://github.com/ceph/ceph.git master

• 更新子模塊代碼
  git submodule update —init —recursive

下面兩步如果環(huán)境已經(jīng)有了切版本合適就可以跳過

• 安裝gcc

git clone [https://github.com/gcc-mirror/gcc.git](https://github.com/gcc-mirror/gcc.git)

b.mkdir build ; cd build
../configure --prefix=/usr --disable-multilib 
c.yum install -y gmp-devel libmpc-devel mpfr-devel flex
d.make -j16 > log 2>&1 && make install >log 2>&1 &

• 升級cmake

wget https://cmake.org/files/v3.9/cmake-3.9.2.tar.gz
//解壓后自行安裝

編譯

• 執(zhí)行install-deps.sh來安裝編譯時需要的各種依賴包
  sh install-deps.sh

Note:
1.這一步一般要耗費較長時間，所以我們可以執(zhí)行這一步的時候打開yum的緩存模式，把依賴包緩存在本地，隨后把這些rpm包建立一個自己的yum源，這樣下次安裝的時候就從本地的yum源安裝，會大大縮減時間!

[root@xt-53 ceph-12.2.10]# cat /etc/yum.conf
[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=1  //置為1，打開緩存模式
debuglevel=2
...

緩存的包會保留在/var/cache/yum/x86_64/7/ 下面
2.如果是容器內(nèi)編譯也可以把安裝完依賴包的容器提交到新的鏡像，之后用這個鏡像啟動的容器就不用再安裝依賴包了。
3.install-deps.sh帶的默認的yum源比較慢，可以更改更快的yum源
install-deps.sh中

$SUDO yum-config-manager --add-repo https://dl.fedoraproject.org/pub/epel/$VERSION_ID/x86_64/
更改為
$SUDO yum-config-manager --add-repo https://mirrors.aliyun.com/epel/$VERSION_ID/x86_64/

• 執(zhí)行do_cmake.sh
  sh do_cmake.sh

• 開始make
  make -j16 > log 2>&1 & //線程數(shù)等于cpu core的2倍，可以提高編譯的速度
  當然我們也可以只編譯某一個模塊,比如只編譯client就可以
  make client > log 2>&1 &

最小測試用例

?當我們更改了代碼準備提交到公司內(nèi)部repo或者社區(qū)repo都需要先執(zhí)行一下最小測試集，看看自己修改的代碼有沒有影響到別的模塊(社區(qū)也會進行同樣的測試)。

• ctest
  可以單獨跑一個測試用例，若不加參數(shù)就跑所有的測試，和make一樣，可以用-j選項來多線程同時跑。
  例如：

# ctest -R cephfs
Test project /home/ivan/ws/Luminous/code/ceph-12.2.10/build
    Start 42: unittest_libcephfs_config
1/1 Test #42: unittest_libcephfs_config ........   Passed    0.02 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.06 sec

?也可以用-V選項來使運行過程打印到前臺

# ctest -V -R client
UpdateCTestConfiguration  from :/home/ivan/ws/Luminous/code/ceph-12.2.10/build/DartConfiguration.tcl
UpdateCTestConfiguration  from :/home/ivan/ws/Luminous/code/ceph-12.2.10/build/DartConfiguration.tcl
Test project /home/ivan/ws/Luminous/code/ceph-12.2.10/build
Constructing a list of tests
Done constructing a list of tests
Checking test dependency graph...
Checking test dependency graph end
test 138
    Start 138: unittest_mclock_client_queue

138: Test command: /home/ivan/ws/Luminous/code/ceph-12.2.10/build/bin/unittest_mclock_client_queue
138: Test timeout computed to be: 3600
138: [==========] Running 4 tests from 1 test case.
138: [----------] Global test environment set-up.
138: [----------] 4 tests from MClockClientQueueTest
138: [ RUN      ] MClockClientQueueTest.TestSize
138: [       OK ] MClockClientQueueTest.TestSize (0 ms)
138: [ RUN      ] MClockClientQueueTest.TestEnqueue
138: [       OK ] MClockClientQueueTest.TestEnqueue (0 ms)
138: [ RUN      ] MClockClientQueueTest.TestEnqueueStrict
138: [       OK ] MClockClientQueueTest.TestEnqueueStrict (0 ms)
138: [ RUN      ] MClockClientQueueTest.TestRemoveByClass
138: [       OK ] MClockClientQueueTest.TestRemoveByClass (0 ms)
138: [----------] 4 tests from MClockClientQueueTest (1 ms total)
138:
138: [----------] Global test environment tear-down
138: [==========] 4 tests from 1 test case ran. (1 ms total)
138: [  PASSED  ] 4 tests.
1/1 Test #138: unittest_mclock_client_queue .....   Passed    0.04 sec

The following tests passed:
    unittest_mclock_client_queue

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.08 sec

• make check
  編譯以及順便跑所有的測試用例，可以加-j選項來啟動多線程。

構(gòu)建RPM包

?當我們需要發(fā)布產(chǎn)品時就需要把各個模塊打成rpm包。我把官網(wǎng)上面的環(huán)節(jié)再加上自己的特殊需求寫了個腳本來實現(xiàn)一鍵打包功能。
?我們首先需要更改make-disk，該腳本主要用于生成ceph.spec和ceph-*.tar.bz2文件以供后面打rpm包使用。我們主要改變了rpm_version,rpm_release ，是否下載boost庫等.

# cat make-dist
#!/bin/sh -e

DOWNLOAD_BOOST=$1

if [ ! -d .git ]; then
    echo "no .git present.  run this from the base dir of the git checkout."
    exit 1
fi

##############################################################
#
# Version is very import, we add a Version.txt to record it 
# There should use own Version.txt instead git tag
#
##############################################################
version=`cat Version.txt | grep VERSION |awk -F '=' '{print $2}'`

[ -z "$version" ] && version=`git describe --match 'v*' | sed 's/^v//'`
outfile="ceph-$version"

echo "version $version"

# update submodules
echo "updating submodules..."
force=$(if git submodule usage 2>&1 | grep --quiet 'update.*--force'; then echo --force ; fi)
if ! git submodule sync || ! git submodule update $force --init --recursive; then
    echo "Error: could not initialize submodule projects"
    echo "  Network connectivity might be required."
    exit 1
fi

download_boost() {
    boost_version=$1
    shift
    boost_md5=$1
    shift
    boost_version_underscore=$(echo $boost_version | sed 's/\./_/g')
    boost_fname=boost_${boost_version_underscore}.tar.bz2
    set +e
    if [ "${DOWNLOAD_BOOST}" == "yes" ];then
        while true; do
            url_base=$1
            shift
            if [ -z $url_base ]; then
                echo "Error: failed to download boost."
                exit
            fi
            url=$url_base/$boost_fname

            wget -c --no-verbose -O $boost_fname $url
            if [ $? != 0 -o ! -e $boost_fname ]; then
                echo "Download of $url failed"
            elif [ $(md5sum $boost_fname | awk '{print $1}') != $boost_md5 ]; then
                echo "Error: failed to download boost: MD5 mismatch."
            else
                break
            fi
        done
    fi
    set -e
    tar xjf $boost_fname -C src \
        --exclude="$boost_version_underscore/libs/*/doc" \
        --exclude="$boost_version_underscore/libs/*/example" \
        --exclude="$boost_version_underscore/libs/*/examples" \
        --exclude="$boost_version_underscore/libs/*/meta" \
        --exclude="$boost_version_underscore/libs/*/test" \
        --exclude="$boost_version_underscore/tools/boostbook" \
        --exclude="$boost_version_underscore/tools/quickbook" \
        --exclude="$boost_version_underscore/tools/auto_index" \
        --exclude='doc' --exclude='more' --exclude='status'
    mv src/boost_${boost_version_underscore} src/boost
    tar cf ${outfile}.boost.tar ${outfile}/src/boost
    rm -rf src/boost
}

# clean out old cruft...
echo "cleanup..."
rm -f $outfile*

# build new tarball
echo "building tarball..."
bin/git-archive-all.sh --prefix ceph-$version/ \
               --verbose \
               --ignore corpus \
               $outfile.tar

# populate files with version strings
echo "including src/.git_version, ceph.spec"

# The command finds the most recent tag that is reachable from a commit. If the tag points to the commit, 
# then only the tag is shown. Otherwise, it suffixes the tag name with the number of additional commits on
# top of the tagged object and the abbreviated object name of the most recent commit. The result is a 
# human-readable" object name which can also be used to identify the commit to other git commands.
(git rev-parse HEAD ; git describe) 2> /dev/null > src/.git_version

# if the version has '-' in it, it has a 'release' part,
# like vX.Y.Z-N-g<shortsha1>.  If it doesn't, it's just
# vX.Y.Z.  Handle both, and translate - to . for rpm
# naming rules (the - separates version and release).

if expr index $version '-' > /dev/null; then
    rpm_version=`echo $version | cut -d - -f 1-1`
    rpm_release=`echo $version | cut -d - -f 2- | sed 's/-/./'`
else
    rpm_version=$version
        # rpm style: ceph-"module"-"version"-"commit_id"
        # example: ceph-mds-12.2.10.mh.0-1.ga200519.el7.x86_64.rpm
    rpm_release=`tail -n 1 src/.git_version | cut -d - -f 2- | sed 's/-/./'`
fi

for spec in ceph.spec.in alpine/APKBUILD.in; do
    cat $spec |
        sed "s/@VERSION@/$rpm_version/g" |
        sed "s/@RPM_RELEASE@/$rpm_release/g" |
        sed "s/@TARBALL_BASENAME@/ceph-$version/g" > `echo $spec | sed 's/.in$//'`
done
ln -s . $outfile
tar cvf $outfile.version.tar $outfile/src/.git_version $outfile/ceph.spec $outfile/alpine/APKBUILD
# NOTE: If you change this version number make sure the package is available
# at the three URLs referenced below (may involve uploading to download.ceph.com)
boost_version=1.66.0
    download_boost $boost_version b2dfbd6c717be4a7bb2d88018eaccf75 \
    https://dl.bintray.com/boostorg/release/$boost_version/source \
    https://downloads.sourceforge.net/project/boost/boost/$boost_version \
    https://download.ceph.com/qa
tar --concatenate -f $outfile.all.tar $outfile.version.tar
tar --concatenate -f $outfile.all.tar $outfile.boost.tar
tar --concatenate -f $outfile.all.tar $outfile.tar
mv $outfile.all.tar $outfile.tar
rm $outfile
rm -f $outfile.version.tar
rm -f $outfile.boost.tar

echo "compressing..."
bzip2 -9 $outfile.tar

echo "done."

腳本make-rpm.sh如下：

# cat make-rpm.sh
#!/bin/bash

#echo $@

#######################################################
#
# build Ceph rpm packages
#
# ./make-rpm.sh
#
########################################################

CLEANBUILD="yes"
INSTALL_DEPS="false"
COMMAND=$1
DOWNLOAD_BOOST="false"

# current dir of this script
BASEDIR=`cd $(dirname $0); pwd -P`

function usage()
{
cat <<EOF

Usage: ./$0 <build|clean> [Flags]

Build ceph and generate rpm packages.

Flags:
    -i,--install-deps      //don't excute install-deps.sh
    -n,--no-cleanbuild     //don't clean build
    --download_boost       // download boost library

    -h,--help              //list help info
EOF
}

function build_rpms()
{
    # we don't need install deps every time
    if [  "$INSTALL_DEPS" == "yes" ];then
        echo "sh ./install-deps.sh"
        #sh ./install-deps.sh
    fi

    if [ "$MODULE" != "" ];then
        echo "compile $MODULE"
    else
        echo "compile all modules"
    fi
    # we only need download boot at first time
    if [ "$DOWNLOAD_BOOST" == "yes" ];then
        echo "make-dist yes"
        ./make-dist yes
    else
        echo "make-dist false"
        ./make-dist false
    fi
    ###########################################
    #
    # Version is very import, we add a Version.txt to record it 
    #
    ###########################################
    version=`cat Version.txt | grep VERSION |awk -F '=' '{print $2}'`

    rm -rf ${BASEDIR}/rpmbuild
    mkdir ${BASEDIR}/rpmbuild && cd ${BASEDIR}/rpmbuild && mkdir BUILD  BUILDROOT  RPMS  SOURCES  SPECS  SRPMS
    cp ${BASEDIR}/ceph-${version}.tar.bz2 ${BASEDIR}/rpmbuild/SOURCES
    tar --strip-components=1 -C ${BASEDIR}/rpmbuild/SPECS/ --no-anchored -xvjf ${BASEDIR}/rpmbuild/SOURCES/ceph-${version}.tar.bz2 "ceph.spec"

    if [ "$CLEANBUILD" == "yes" ];then
        #rpmbuild --define="_topdir "${BASEDIR}"" -ba ${BASEDIR}/rpmbuild/SPECS/ceph.spec --with clean_build
        rpmbuild --define="_topdir "${BASEDIR}"/rpmbuild" -ba ./SPECS/ceph.spec --with clean_build
    else
        #rpmbuild --define '_topdir ./rpmbuild' -ba ${BASEDIR}/rpmbuild/SPECS/ceph.spec
        rpmbuild --define="_topdir "${BASEDIR}"/rpmbuild" -ba ./SPECS/ceph.spec
    fi

    mv ${BASEDIR}/rpmbuild/RPMS/*/* ${BASEDIR}/rpmbuild/
    mv ${BASEDIR}rpmbuild/SRPMS/* ${BASEDIR}/rpmbuild/
}

function make_clean()
{
    echo "clean last build result ..."
    rm -rf rpmbuild
}

function do_work()
{
   if [ "$COMMAND" == "build" ];then
       echo "start building ..."
       build_rpms
   else
       echo "start cleaning"
       make_clean

   fi

}
ARGS=`getopt -o inmc --long install-deps,no-cleanbuild,module:,download_boost,clean -n 'example.sh' -- "$@"`
eval set -- "${ARGS}"

while true
do
    case "$1" in
        -i|--install-deps)
            INSTALL_DEPS="yes"
            echo INSTALL_DEPS $INSTALL_DEPS
            shift
            ;;
        -n|--no-cleanbuild)
            CLEANBUILD="false";
            echo CLEANBULD: $CLEANBUILD;
            shift
            ;;
        --download_boost)
            DOWNLOAD_BOOST="yes";
            echo DOWNLOAD_BOOST $DOWNLOAD_BOOST;
            shift
            ;;
        -c|--clean)
            COMMAND="clean";
            echo COMMAND: $COMMAND;
            shift
            ;;
        --)
            shift;
            break;;
        *) echo "Internal error!;"; exit 1 ;;
    esac
done

if [ $# -lt 1 ];then
    usage
    exit -1
fi

do_work

• 腳本使用方法

# sh make-rpm.sh -h
Usage: ./make-rpm.sh <build|clean> [Flags]

Build ceph and generate rpm packages.

Flags:
    -i,--install-deps      //don't excute install-deps.sh
    -n,--no-cleanbuild     //don't clean build
    --download_boost       // download boost library

    -h,--help              //list help info

End.

• build rpm 包

// 開始打包,不安裝依賴包，且不下載boost庫
sh make-rpm.sh build

• 清除掉上次編譯的殘留文件

sh make-rpm.sh clean
或者
sh make-rpm.sh -c