Docker 搭建 Spark 环境

sudo apt install docker-compose

检查 docker-compose 是否安装成功

docker-compose --version

docker pull singularities/spark

检查 spark 的 docker 镜像是否安装成功：

docker image ls

mkdir singularitiesCR
cd singularitiesCR
touch docker-compose.yml
vim docker-compose.yml

编辑 docker-compose.yml 的内容如下：

version: "2" 
services:
  master:
    image: singularities/spark
    command: start-spark master
    hostname: master
    ports:
      - "6066:6066"
      - "7070:7070"
      - "8080:8080"
      - "50070:50070"
  worker:
    image: singularities/spark
    command: start-spark worker master
    environment:
      SPARK_WORKER_CORES: 1
      SPARK_WORKER_MEMORY: 2g
    links:
      - master

在 docker-compose.yml 同级目录下执行如下命令

docker-compose up -d

查看容器

docker-compose ps

          Name                      Command            State                   Ports                 
-----------------------------------------------------------------------------------------------------
singularitiescr_master_1   start-spark master          Up      10020/tcp, 13562/tcp, 14000/tcp,      
                                                               19888/tcp, 50010/tcp, 50020/tcp,      
                                                               0.0.0.0:50070->50070/tcp, 50075/tcp,  
                                                               50090/tcp, 50470/tcp, 50475/tcp,      
                                                               0.0.0.0:6066->6066/tcp,               
                                                               0.0.0.0:7070->7070/tcp, 7077/tcp,     
                                                               8020/tcp, 0.0.0.0:8080->8080/tcp,     
                                                               8081/tcp, 9000/tcp                    
singularitiescr_worker_1   start-spark worker master   Up      10020/tcp, 13562/tcp, 14000/tcp,      
                                                               19888/tcp, 50010/tcp, 50020/tcp,      
                                                               50070/tcp, 50075/tcp, 50090/tcp,      
                                                               50470/tcp, 50475/tcp, 6066/tcp,       
                                                               7077/tcp, 8020/tcp, 8080/tcp,         
                                                               8081/tcp, 9000/tcp              

查看本机 IP

ifconfig

ens33: flags=4163  mtu 1500
        inet 192.168.31.27  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::6841:cba4:a8ab:6691  prefixlen 64  scopeid 0x20
        ether 00:0c:29:53:c7:3e  txqueuelen 1000  (Ethernet)
        RX packets 1027788  bytes 1507409925 (1.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 459275  bytes 36319898 (34.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

在浏览器中分别输入对应 URL 查看

192.168.31.27:50070

192.168.31.27:8080

docker-compose stop

docker-compose rm