無(wú)標(biāo)題文章

# OmniAdapter編譯指導(dǎo)

#### 介紹

本文檔將介紹如何編譯OmniAdapter。

#### 編譯要求

1. 編譯工具要求

| 編譯工具? ? | 推薦版本? ? ? |

|----------|-----------|

| GCC? ? ? | 7.3.0+? ? |

| JDK? ? ? | jkd8u242+ |

| CMake? ? | 3.20+? ? |

| MAVEN? ? | 3.5.4+? ? |

| autoconf | 2.69? ? ? |

2. 開源依賴組件要求

| 開源組件? ? | 推薦版本? ? ? | 鏈接? ? ? |

|----------|-----------|-----------|

| Arrow? | 11.0.0? ? | https://github.com/apache/arrow/archive/refs/tags/apache-arrow-11.0.0.zip? ? |

| boost? | 1.7.3? ? | https://boostorg.jfrog.io/artifactory/main/release/1.73.0/source/boost_1_73_0.tar.gz? ? |

| cyrus-sasl | 2.1.28 | https://github.com/cyrusimap/cyrus-sasl.git? ? |

| googletest | 1.10.0 | https://github.com/google/googletest/archive/refs/tags/release-1.10.0.zip? |

| Hadoop? | 3.2.0? ? | https://github.com/apache/hadoop/archive/refs/tags/rel/release-3.2.0.zip? ? |

| jemalloc| 5.3.0? ? | https://github.com/jemalloc/jemalloc/archive/refs/tags/5.3.0.tar.gz? ? |

| json? ? | V3.10.1? | https://github.com/nlohmann/json/archive/refs/tags/v3.10.1.zip? ? |

| jsoncpp | 1.9.3? ? | https://github.com/open-source-parsers/jsoncpp/archive/refs/tags/1.9.3.zip? ? |

| LZ4? ? | v1.9.3? ? | https://github.com/lz4/lz4.git? ? |

| OpenSSL | 1.1.1n? ? | https://github.com/openssl/openssl/archive/refs/tags/OpenSSL_1_1_1n.tar.gz? |

| ORC? ? | V1.7.4? ? | https://github.com/apache/orc/archive/refs/tags/v1.7.4.zip? ? |

| protobuf | 3.13.0? | https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.13.0.zip? |

| Snappy? | 1.1.7? ? | https://github.com/google/snappy.git? ? |

| ZLIB? ? | 1.2.11? ? | https://github.com/madler/zlib.git? ? |

| Zstd? ? | v1.5.2? ? | https://github.com/facebook/zstd/archive/refs/tags/v1.5.2.zip? ? |

#### 編譯前準(zhǔn)備

1.? 設(shè)置環(huán)境變量


```

? ? BUILD_OMNIOP為編譯目錄，可自定義

? ? mkdir -p /home/build_omniop_spark_extension/

? ? export BUILD_OMNIOP=/home/build_omniop_spark_extension

? ? mkdir -p /opt/omni-operator/lib/

? ? export OMNI_HOME=/opt/omni-operator

```

2.? 下載Omnioperator內(nèi)核軟件包以及及依賴包

? ? 內(nèi)核軟件包鏈接：https://kunpeng-repo.obs.cn-north-4.myhuaweicloud.com/Kunpeng%20BoostKit/Kunpeng%20BoostKit%2023.0.0/BoostKit-omniruntime_1.1.0.zip

? ? 獲?。築oostKit-omniruntime_1.1.0.zip

? ? 依賴包鏈接：https://gitee.com/kunpengcompute/boostkit-bigdata/releases/tag/20231230-OmniOperator

? ? 獲?。? LLVM.zip ORC.zip Protobuf.zip Arrow.zip Parquet.zip Hdfs.zip



```

? ? mkdir -p ${BUILD_OMNIOP}/omni/

? ? cd ${BUILD_OMNIOP}

? ? wget --no-check-certificate https://kunpeng-repo.obs.cn-north-4.myhuaweicloud.com/Kunpeng%20BoostKit/Kunpeng%20BoostKit%2023.0.0/BoostKit-? ? ? ? omniruntime_1.1.0.zip

? ? wget --no-check-certificate https://gitee.com/kunpengcompute/boostkit-bigdata/releases/download/20231230-OmniOperator/LLVM.zip

? ? wget --no-check-certificate https://gitee.com/kunpengcompute/boostkit-bigdata/releases/download/20231230-OmniOperator/ORC.zip

? ? wget --no-check-certificate https://gitee.com/kunpengcompute/boostkit-bigdata/releases/download/20231230-OmniOperator/Protobuf.zip

? ? wget --no-check-certificate https://gitee.com/kunpengcompute/boostkit-bigdata/releases/download/20231230-OmniOperator/Arrow.zip

? ? wget --no-check-certificate https://gitee.com/kunpengcompute/boostkit-bigdata/releases/download/20231230-OmniOperator/Parquet.zip

? ? wget --no-check-certificate https://gitee.com/kunpengcompute/boostkit-bigdata/releases/download/20231230-OmniOperator/Hdfs.zip

? ? 解壓

? ? unzip "*.zip"

? ? cp BoostKit-omniop_1.3.0.zip *.so* ${OMNI_HOME}/lib/

? ? cd ${OMNI_HOME}/lib/

? ? tar -zxvf boostkit-omniop-operator-1.3.0-aarch64.tar.gz --strip-components=1

? ? chmod -R 550 ${OMNI_HOME}/lib/*

? ? mkdir -p ${OMNI_HOME}/lib/include/huawei_secure_c/include/

? ? cp ${OMNI_HOME}/lib/include/securec.h ${OMNI_HOME}/lib/include/huawei_secure_c/include/


```

3.? 拉取依賴開源組件以及OmniAdapter源碼


```

? ? cd ${BUILD_OMNIOP}

? ? wget --no-check-certificate https://github.com/protocolbuffers/protobuf/archive/refs/tags/v3.13.0.zip

? ? wget --no-check-certificate https://github.com/google/googletest/archive/refs/tags/release-1.10.0.zip

? ? wget --no-check-certificate https://github.com/openssl/openssl/archive/refs/tags/OpenSSL_1_1_1n.tar.gz

? ? wget --no-check-certificate https://github.com/facebook/zstd/archive/refs/tags/v1.5.2.zip

? ? wget --no-check-certificate https://github.com/apache/arrow/archive/refs/tags/apache-arrow-11.0.0.zip

? ? wget --no-check-certificate https://github.com/nlohmann/json/archive/refs/tags/v3.10.1.zip

? ? wget --no-check-certificate https://github.com/apache/orc/archive/refs/tags/v1.7.4.zip

? ? wget --no-check-certificate https://github.com/open-source-parsers/jsoncpp/archive/refs/tags/1.9.3.zip

? ? wget --no-check-certificate https://github.com/apache/hadoop/archive/refs/tags/rel/release-3.2.0.zip

? ? wget --no-check-certificate https://boostorg.jfrog.io/artifactory/main/release/1.73.0/source/boost_1_73_0.tar.gz

? ? wget --no-check-certificate https://github.com/jemalloc/jemalloc/archive/refs/tags/5.3.0.tar.gz

? ? git clone -b v1.2.11 https://github.com/madler/zlib.git

? ? git clone -b v1.9.3 https://github.com/lz4/lz4.git

? ? git clone -b 1.1.7 https://github.com/google/snappy.git

? ? git clone -b cyrus-sasl-2.1.28 https://github.com/cyrusimap/cyrus-sasl.git

? ? git clone https://gitee.com/kunpeng_compute/omni-adapter.git

? ? 解壓

? ? unzip "*.zip"

? ? find . -maxdepth 1 -name "*.tar.gz" -exec tar -zxvf {} \;

? ? 設(shè)置環(huán)境變量

? ? export ORC_HOME=${BUILD_OMNIOP}/orc-1.7.4

? ? export BOOST_HOME=${BUILD_OMNIOP}/boost_1_73_0

? ? export ARROW_HOME=${BUILD_OMNIOP}/arrow-apache-arrow-11.0.0

? ? export JSON_HOME=${BUILD_OMNIOP}/json-3.10.1

? ? export JSONCPP_HOME=${BUILD_OMNIOP}/jsoncpp-1.9.3

```

#### 編譯開源組件

```

? ? 指定編譯時(shí)的線程數(shù)

? ? export num_cpus=16

```

1.? 編譯protobuf


```

? ? cd ${BUILD_OMNIOP}

? ? cd protobuf-3.13.0/

? ? mkdir build

? ? sh autogen.sh

? ? ./configure --enable-static=no

? ? make -j ${num_cpus}

? ? make install

```


2.? 編譯boost

```

? ? cd ${BUILD_OMNIOP}

? ? cd boost_1_73_0/

? ? ./bootstrap.sh

? ? ./b2

```

3.? 編譯googletest


```

? ? cd ${BUILD_OMNIOP}

? ? cd googletest-release-1.10.0/

? ? mkdir build

? ? cd build

? ? cmake -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX:PATH=/usr ..

? ? make -j ${num_cpus}

? ? make install

```

4.? 編譯jemalloc

```

? ? cd ${BUILD_OMNIOP}

? ? cd jemalloc-5.3.0

? ? ./autogen.sh --disable-initial-exec-tls --with-lg-page=16 --with-lg-hugepage=29

? ? make -j ${num_cpus}

? ? make install

? ? export JEMALLOC_HOME=${BUILD_OMNIOP}/jemalloc-5.3.0/lib

? ? cp ${JEMALLOC_HOME}/libjemalloc.so.2 ${OMNI_HOME}/lib/

```

5.? 編譯OpenSSL


```

? ? cd ${BUILD_OMNIOP}

? ? cd openssl-OpenSSL_1_1_1n/

? ? ./config

? ? make -j ${num_cpus}

? ? make install

```

6.? 編譯zlib


```

? ? cd ${BUILD_OMNIOP}

? ? cd zlib

? ? ./configure

? ? make -j ${num_cpus}

? ? make install

```

7.? 編譯lz4


```

? ? cd ${BUILD_OMNIOP}

? ? cd lz4

? ? make -j ${num_cpus}

? ? make install

```

8.? 編譯snappy


```

? ? cd ${BUILD_OMNIOP}

? ? cd snappy

? ? mkdir -p build

? ? cd build

? ? cmake .. -DCMAKE_BUILD_TYPE=RELEASE -DBUILD_SHARED_LIBS=O

? ? make -j ${num_cpus}

? ? make install

```

9.? 編譯zstd


```

? ? cd ${BUILD_OMNIOP}

? ? cd zstd-1.5.2/

? ? make -j ${num_cpus}

? ? make install? ?

```

10.? 編譯cyrus-sasl


```

? ? cd ${BUILD_OMNIOP}

? ? cd cyrus-sasl/

? ? ./autogen.sh

? ? make -j ${num_cpus}

? ? make install

? ? ln -sf /usr/local/lib/sasl2 /usr/lib/sasl2

```

11.? 編譯ARROW


```

? ? cd ${BUILD_OMNIOP}

? ? cd arrow-apache-arrow-11.0.0/cpp

? ? mkdir build

? ? cd build

? ? cmake \

? ? -DARROW_BUILD_INTEGRATION=ON \

? ? -DARROW_BUILD_STATIC=OFF \

? ? -DARROW_BUILD_TESTS=OFF \

? ? -DARROW_ENABLE_TIMING_TESTS=OFF \

? ? -DARROW_BUILD_INTEGRATION=OFF \

? ? -DARROW_COMPUTE=ON \

? ? -DARROW_DATASET=ON \

? ? -DARROW_EXTRA_ERROR_CONTEXT=ON \

? ? -DARROW_MIMALLOC=ON \

? ? -DARROW_SUBSTRAIT=ON \

? ? -DARROW_WITH_BROTLI=ON \

? ? -DARROW_WITH_BZ2=ON \

? ? -DARROW_WITH_LZ4=ON \

? ? -DARROW_WITH_RE2=ON \

? ? -DARROW_WITH_SNAPPY=ON \

? ? -DARROW_WITH_UTF8PROC=ON \

? ? -DARROW_WITH_ZLIB=ON \

? ? -DARROW_WITH_ZSTD=ON \

? ? -DARROW_HDFS=ON \

? ? -DCMAKE_BUILD_TYPE=Release \

? ? -S .. -B .

? ? make -j ${num_cpus}

? ? make install

```

12.? 編譯orc


```

? ? cd ${BUILD_OMNIOP}

? ? cd orc-1.7.4/

? ? mkdir build && cd build

? ? cmake ../ -DBUILD_JAVA=OFF -DANALYZE_JAVA=OFF -DBUILD_LIBHDFSPP=ON -DBUILD_CPP_TESTS=OFF -DBUILD_TOOLS=ON -DBUILD_POSITION_INDEPENDENT_LIB=ON

? ? make -j ${num_cpus}

? ? cd ${ORC_HOME}/c++/libs/libhdfspp

? ? tar -zxvf libhdfspp.tar.gz

```

#### 編譯OmniAdapter源碼

1.? 設(shè)置編譯環(huán)境變量


```

? ? export PATH=$PATH:${PROTOBUF_PATH}/bin

? ? export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib/:/usr/lib64/:${OMNI_HOME}/lib:${JAVA_HOME}/jre/lib/aarch64/server/

? ? export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/lib/:/usr/lib64/:${OMNI_HOME}/lib:${JAVA_HOME}/jre/lib/aarch64/server/

? ? export C_INCLUDE_PATH=$C_INCLUDE_PATH:${OMNI_HOME}/lib/include:${ORC_HOME}/c++/include/:${ORC_HOME}/build/c++/include:${JSONCPP_HOME}/include:${ORC_HOME}/c++/libs/libhdfspp/include:${BUILD_OMNIOP}/hadoop-rel-release-3.2.0/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include/hdfs/:${ARROW_HOME}/cpp/src/:${BOOST_HOME}

? ? export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:${OMNI_HOME}/lib/include:${JSON_HOME}/include:${ORC_HOME}/c++/include:${ORC_HOME}/build/c++/include:${JSONCPP_HOME}/include:${ORC_HOME}/c++/libs/libhdfspp/include:${BUILD_OMNIOP}/hadoop-rel-release-3.2.0/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/include/hdfs/:${ARROW_HOME}/cpp/src/:${BOOST_HOME}

```

2.? 編譯OmniAdapter

```

? ? cd ${BUILD_OMNIOP}

? ? cd omni-adapter/omniop-spark-extension

? ? sed -i '21s/3.13.0-h19/3.13.0/' pom.xml

? ? mvn clean package -Ddep.os.arch=-aarch64 -Dscoverage.skip=true -DskipTests

```

3.? 獲取編譯輸出件


```

? ? 在${BUILD_OMNIOP}/omni-adapter/omniop-spark-extension/java/target/目錄下可以獲取到boostkit-omniop-spark-3.3.1-1.3.0-aarch64.jar ${OMNI_HOME}/lib

? ? cp ${BUILD_OMNIOP}/omni-adapter/omniop-spark-extension/java/target/boostkit-omniop-spark-3.3.1-1.3.0-aarch64.jar ${OMNI_HOME}/lib

```



/**

* Licensed to the Apache Software Foundation (ASF) under one

* or more contributor license agreements.? See the NOTICE file

* distributed with this work for additional information

* regarding copyright ownership.? The ASF licenses this file

* to you under the Apache License, Version 2.0 (the

* "License"); you may not use this file except in compliance

* with the License.? You may obtain a copy of the License at

*

*? ? http://www.apache.org/licenses/LICENSE-2.0

*

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

*/

#ifndef LIBHDFSPP_OPTIONS_H_

#define LIBHDFSPP_OPTIONS_H_

#include "hdfspp/uri.h"

#include <algorithm>

#include <string>

#include <vector>

#include <map>

namespace hdfs {

class Token {

public:

? ? Token() {}

? ? Token (const Token &other) {

? ? ? ? setIdentifier(other.getIdentifier());

? ? ? ? setPassword(other.getPassword());

? ? ? ? setService(other.getService());

? ? ? ? setKind(other.getKind());

? ? }

? ? Token& operator= (const Token &other) {

? ? ? ? setIdentifier(other.getIdentifier());

? ? ? ? setPassword(other.getPassword());

? ? ? ? setService(other.getService());

? ? ? ? setKind(other.getKind());

? ? ? ? return *this;

? ? }

? ? const std::string & getIdentifier() const {

? ? ? ? return identifier;

? ? }

? ? void setIdentifier(const std::string & identifier_) {

? ? ? ? this->identifier = identifier_;

? ? }

? ? const std::string & getKind() const {

? ? ? ? return kind;

? ? }

? ? void setKind(const std::string & kind_) {

? ? ? ? this->kind = kind_;

? ? }

? ? const std::string & getPassword() const {

? ? ? ? return password;

? ? }

? ? void setPassword(const std::string & password_) {

? ? ? ? this->password = password_;

? ? }

? ? const std::string & getService() const {

? ? ? ? return service;

? ? }

? ? void setService(const std::string & service_) {

? ? ? ? this->service = service_;

? ? }

? ? bool hasValue() const {

? ? ? ? return !identifier.empty();

? ? }

private:

? ? std::string identifier;

? ? std::string password;

? ? std::string kind;

? ? std::string service;

};

struct NamenodeInfo {

? NamenodeInfo(const std::string &nameservice_, const std::string &nodename_, const URI &uri_) :

? ? ? ? ? ? ? ? nameservice(nameservice_), name(nodename_), uri(uri_) {}

? NamenodeInfo(){}

? //nameservice this belongs to

? std::string nameservice;

? //node name

? std::string name;

? //host:port

? URI uri;

? //get server hostname and port (aka service)

? std::string get_host() const;

? std::string get_port() const;

};

/**

* Options to control the behavior of the libhdfspp library.

**/

struct Options {

? /**

? * Time out of RPC requests in milliseconds.

? * Default: 30000

? **/

? int rpc_timeout;

? static const int kDefaultRpcTimeout = 30000;

? /**

? * Time to wait for an RPC connection before failing

? * Default: 30000

? **/

? int rpc_connect_timeout;

? static const int kDefaultRpcConnectTimeout = 30000;

? /**

? * Maximum number of retries for RPC operations

? **/

? int max_rpc_retries;

? static const int kNoRetry = 0;

? static const int kDefaultMaxRpcRetries = kNoRetry;

? /**

? * Number of ms to wait between retry of RPC operations

? **/

? int rpc_retry_delay_ms;

? static const int kDefaultRpcRetryDelayMs = 10000;

? /**

? * Exclusion time for failed datanodes in milliseconds.

? * Default: 60000

? **/

? unsigned int host_exclusion_duration;

? static const unsigned int kDefaultHostExclusionDuration = 600000;

? /**

? * URI to connect to if no host:port are specified in connect

? */

? URI defaultFS;

? /**

? * Namenodes used to provide HA for this cluster if applicable

? **/

? std::map<std::string, std::vector<NamenodeInfo>> services;

? /**

? * Client failover attempts before failover gives up

? **/

? int failover_max_retries;

? static const unsigned int kDefaultFailoverMaxRetries = 4;

? /**

? * Client failover attempts before failover gives up if server

? * connection is timing out.

? **/

? int failover_connection_max_retries;

? static const unsigned int kDefaultFailoverConnectionMaxRetries = 0;

? /*

? * Which form of authentication to use with the server

? * Default: simple

? */

? enum Authentication {

? ? ? kSimple,

? ? ? kKerberos

? };

? Authentication authentication;

? static const Authentication kDefaultAuthentication = kSimple;

? /**

? * Block size in bytes.

? * Default: 128 * 1024 * 1024 = 134217728

? **/

? long block_size;

? static const long kDefaultBlockSize = 128*1024*1024;

? /**

? * Asio worker thread count

? * default: -1, indicates number of hardware threads

? **/

? int io_threads_;

? static const int kDefaultIoThreads = -1;

? std::map<std::pair<std::string, std::string>, Token> tokens_;

? void addTokens(const Options &other) {

? ? ? const std::map<std::pair<std::string, std::string>, Token> tokens = other.getTokens();

? ? ? for (auto token : tokens)

? ? ? {

? ? ? ? ? this->addToken(token.second);

? ? ? }

? }

? void addToken(const Token &token) {

? ? ? std::string kind;

? ? ? std::string service;

? ? ? kind.resize(token.getKind().size());

? ? ? service.resize(token.getService().size());

? ? ? std::transform(token.getKind().begin(), token.getKind().end(), kind.begin(), tolower);

? ? ? std::transform(token.getService().begin(), token.getService().end(), service.begin(), tolower);

? ? ? if (kind != "hdfs_delegation_token") {

? ? ? ? ? return;

? ? ? }

? ? ? tokens_[std::make_pair(service, kind)] = token;

? }

? const std::map<std::pair<std::string, std::string>, Token> & getTokens() const {

? ? ? return tokens_;

? }

? Options();

};

}

#endif

最后編輯于 ：
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請(qǐng)聯(lián)系作者
【社區(qū)內(nèi)容提示】社區(qū)部分內(nèi)容疑似由AI輔助生成，瀏覽時(shí)請(qǐng)結(jié)合常識(shí)與多方信息審慎甄別。
平臺(tái)聲明：文章內(nèi)容（如有圖片或視頻亦包括在內(nèi)）由作者上傳并發(fā)布，文章內(nèi)容僅代表作者本人觀點(diǎn)，簡(jiǎn)書系信息發(fā)布平臺(tái)，僅提供信息存儲(chǔ)服務(wù)。

相關(guān)閱讀更多精彩內(nèi)容

友情鏈接更多精彩內(nèi)容