下载或copy以下开发环境到目标开发机
# ll total 20188 drwxr-xr-x 8 root root 255 May 12 04:32 jdk drwxr-xr-x 6 root root 99 May 12 04:32 maven drwxr-xr-x 5 root root 51 May 12 04:32 sbt drwxrwxr-x 6 root root 79 May 12 04:32 scala drwxrwxr-x 6 root root 79 May 12 04:32 scala-2.12.10 -rw-r--r-- 1 root root 20669259 May 12 04:32 scala-2.12.10.tgz
然后设置环境变量即可
export JAVA_HOME=/opt/dev/jdk export PATH=/opt/dev/jdk/bin:/opt/dev/jdk/sbin:$PATH export MAVEN_HOME=/opt/dev/maven export PATH=/opt/dev/maven/bin:/opt/dev/maven/sbin:$PATH export SBT_HOME=/opt/dev/sbt export PATH=/opt/dev/sbt/bin:/opt/dev/sbt/sbin:$PATH export SCALA_HOME=/opt/dev/scala export PATH=/opt/dev/scala/bin:$PATH export LD_LIBRARY_PATH=/usr/lib64/:/usr/local/lib64/:$LD_LIBRARY_PATH设置LD_LIBRARY_PATH
export LD_LIBRARY_PATH=/usr/lib64/:/usr/local/lib64/:$LD_LIBRARY_PATH
CentOS 安装基础环境yum -y install gcc gcc-c++ python-devel libcurl-devel python3 git yum -y install automake autoconf libtool yum -y install gmp-devel mpfr-devel libmpc-devel安装GCC 9.3
注意GCC 编译安装耗时比较长，半小时左右
wget https://bigsearcher.com/mirrors/gcc/releases/gcc-9.3.0/gcc-9.3.0.tar.xz --no-check-certificate tar -xvf gcc-9.3.0.tar.xz cd gcc-9.3.0 ./configure --prefix=/usr --disable-multilib make -j$(nproc) make install -j$(nproc) cd ..安装 openssl-1.1.1n
openssl version wget https://www.openssl.org/source/openssl-1.1.1n.tar.gz --no-check-certificate tar zxvf openssl-1.1.1n.tar.gz cd openssl-1.1.1n #安装在/usr/，可以覆盖旧的openssl ./config --prefix=/usr/ make -j$(nproc) make install -j$(nproc) #如果 ./config --prefix=/usr/local/openssl echo "xxxx" >> /etc/ld.so.conf ldconfig openssl version cd ..安装cmake-3.19
wget https://cmake.org/files/v3.19/cmake-3.19.4.tar.gz tar -zxvf cmake-3.19.4.tar.gz cd cmake-3.19.4 ./configure --prefix=/usr/ make -j$(nproc) make install -j$(nproc) cmake --version cd ..安装LLVM-11
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-11.0.0/llvm-11.0.0.src.tar.xz tar xf llvm-11.0.0.src.tar.xz cd llvm-11.0.0.src mkdir build cd build cmake .. -DCMAKE_BUILD_TYPE=Release cmake --build . -j cmake --build . --target install llvm-config --version cd ../../安装clang-11
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-11.0.0/clang-11.0.0.src.tar.xz tar xf clang-11.0.0.src.tar.xz cd clang-11.0.0.src mkdir build cd build cmake .. -DCMAKE_BUILD_TYPE=Release make -j$(nproc) make install -j$(nproc) cd ../../安装boost-1.7.3
wget https://boostorg.jfrog.io/artifactory/main/release/1.73.0/source/boost_1_73_0.tar.gz tar zxvf boost_1_73_0.tar.gz cd boost_1_73_0 ./bootstrap.sh --prefix=/usr/ ./b2 -j$(nproc) ./b2 install -j$(nproc) ldconfig cd ../安装protobuf
wget https://github.com/protocolbuffers/protobuf/releases/download/v3.13.0/protobuf-all-3.13.0.tar.gz tar -zxvf protobuf-all-3.13.0.tar.gz cd protobuf-3.13.0/ ./autogen.sh ./configure make -j$(nproc) make install -j$(nproc) ldconfig # refresh shared library cache.安装 googletest
git clone https://github.com/google/googletest.git cd googletest git checkout release-1.10.0 mkdir build cd build cmake -DBUILD_GTEST=ON -DBUILD_GMOCK=ON -DINSTALL_GTEST=ON -DINSTALL_GMOCK=ON -DBUILD_SHARED_LIBS=ON .. make -j$(nproc) make install -j$(nproc) cd ../../安装 google benchmark
wget https://github.com/google/benchmark/archive/refs/tags/v1.6.0.tar.gz tar -zxvf v1.6.0.tar.gz cd benchmark-1.6.0 mkdir build cd build cmake -DBENCHMARK_DOWNLOAD_DEPENDENCIES=OFF -DBENCHMARK_ENABLE_GTEST_TESTS=OFF -DBENCHMARK_ENABLE_TESTING=OFF -DCMAKE_BUILD_TYPE=Release ../ make -j$(nproc) make install -j$(nproc) cd ../../Ubuntu apt install
apt install gcc apt install g++ apt install cmake apt install openssl apt install libssl-dev apt install libcurl4-openssl-dev apt install libboost-all-devinstall from source install llvm 11
https://github.com/llvm/llvm-project/releases/download/llvmorg-11.0.0/llvm-11.0.0.src.tar.xz
tar xf llvm-11.0.0.src.tar.xz cd llvm-11.0.0.src mkdir build cd build cmake .. -DCMAKE_BUILD_TYPE=Release cmake --build . -j cmake --build . --target install llvm-config --versionboost
如果不想使用 apt install libboost-all-dev，也可以下载source 编译安装
wget https://boostorg.jfrog.io/artifactory/main/release/1.73.0/source/boost_1_73_0.tar.gz
https://github.com/protocolbuffers/protobuf/releases/download/v3.13.0/protobuf-all-3.13.0.tar.gz
follow https://github.com/protocolbuffers/protobuf/blob/main/src/README.md
tar -zxvf protobuf-all-3.13.0.tar.gz cd protobuf-3.13.0/ ./autogen.sh ./configure make -j #make check make install ldconfig # refresh shared library cache.
make check 的目的是看看有没有error。可省略，输出如下
PASS: protobuf-test PASS: protobuf-lazy-descriptor-test PASS: protobuf-lite-test PASS: google/protobuf/compiler/zip_output_unittest.sh PASS: google/protobuf/io/gzip_stream_unittest.sh PASS: protobuf-lite-arena-test PASS: no-warning-test ============================================================================ Testsuite summary for Protocol Buffers 3.13.0 ============================================================================ # TOTAL: 7 # PASS: 7 # SKIP: 0 # XFAIL: 0 # FAIL: 0 # XPASS: 0 # ERROR: 0 ============================================================================ make[3]: Leaving directory '/home/shen/software/protobuf-3.13.0/src' make[2]: Leaving directory '/home/shen/software/protobuf-3.13.0/src' make[1]: Leaving directory '/home/shen/software/protobuf-3.13.0/src'FQA case
/bin/sh: Python3_EXECUTABLE-NOTFOUND: command not found
安装 python3， 然后重新cmake
caseAn exception or error caused a run to abort: /tmp/spark_columnar_plugin_2660379366566816190/libspark_columnar_jni.so: libprotobuf.so.26: cannot open shared object file: No such file or directory java.lang.UnsatisfiedLinkError: /tmp/spark_columnar_plugin_2660379366566816190/libspark_columnar_jni.so: libprotobuf.so.26: cannot open shared object file: No such file or directory at java.lang.ClassLoader$NativeLibrary.load(Native Method)
不能直接从其他机器copy已经编译好的成勋运行。可以直接copy source code 在当前机器编译
