使用 Ubuntu 机器做主服务
Master 192.168.50.15 (Ubuntu) Worker 1 192.168.50.15 (Ubuntu) Worker 2 192.168.50.16 (Windows 10)
先安装 JDK 和下载 Apache Spark
sudo apt-get install openjdk-11-jdk wget https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz tar xvf spark-3.1.2-bin-hadoop3.2.tgz sudo mv spark-3.1.2-bin-hadoop3.2 /opt/spark
设置 SPARK_HOME
vi ~/.profile export SPARK_HOME=/opt/spark source ~/.profile
设置 服务器 IP
cd /opt/spark/conf cp spark-env.sh.template spark-env.sh SPARK_LOCAL_IP=192.168.50.15 SPARK_MASTER_HOST=192.168.50.15
起动 Master 和 Worker 1
sudo /opt/spark/sbin/start-master.sh sudo /opt/spark/sbin/start-worker.sh spark://192.168.50.15:7077
Windows 下载 Apache Spark 和 Hadoop winutils.exe 放到 e.g. D:app
https://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.2/bin
设置环境变量 SPARK_HOME, HADOOP_HOME 和 PATH
SPARK_HOME D:appspark PATH add %SPARK_HOME%bin HADOOP_HOME D:apphadoop PATH add %HADOOP_HOME%bin
打开命令字元
spark-class org.apache.spark.deploy.worker.Worker spark://192.168.50.15:7077
查看 Spark Master UI
使用 Ctrl-C 来停止 Windows Worker 2
停止 Worker 1 和 Master
sudo /opt/spark/sbin/stop-worker.sh sudo /opt/spark/sbin/stop-master.sh
