Docker实操6——配置好的强化学习Dockerfile

• 环境概述
• 一、文件内容
• • 1.1 sh文件
  • 1.2 换源的txt文件
  • 1.3 测试环境的py文件
• 终：完整的Dockerfile
• 小坑

可在dockerfile中修改对应的版本，但修改前须了解它们彼此的依赖关系，可阅读setup.py

• 执行时的状态：

操作： sh build_image.sh ; sh start_container.sh

可能大家会问一个问题：cuda和cudatoolkit有什么关系？

1. 这里的cuda:10.0是系统层面的关于CUDA的完整工具包，包括Nvidia驱动、编译程序、工具包、IDE、调试器等一系列程序
2. cudatoolkit是为了使用pytorch框架而“事先被编译好的”动态链接库
3. 因此如果只为了使用pytorch，只需关心cudatoolkit就好了（如果要写pytorch以外的cuda程序，就需要关心cuda完整工具包的版本）

# build_image.sh
## 使用当前路径.下的Dockerfile，创建名为stable_rl:launch的镜像
docker build -t stable_rl:launch .

# start_container.sh
## mount_dir:挂载主机上的mount_dir路径到容器内的/home/pamirl/rl_dev
## 利用image_name创建容器
jupyter_port=8888
ssh_port=8889
container_nick_name="stable_rl"
image_name="stable_rl:launch"
mount_dir="/home/pami/rl_dev"

nvidia-docker run -itd -p ${jupyter_port}:8888 -p ${ssh_port}:22 -u pamirl -v ${mount_dir}:/home/pamirl/rl_dev --name ${container_nick_name} ${image_name} bash

自行修改端口、名字、挂载路径、容器内的用户名等

# change_apt.txt
deb https://mirrors.ustc.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb-src https://mirrors.ustc.edu.cn/ubuntu/ bionic main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb-src https://mirrors.ustc.edu.cn/ubuntu/ bionic-updates main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb-src https://mirrors.ustc.edu.cn/ubuntu/ bionic-backports main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
deb-src https://mirrors.ustc.edu.cn/ubuntu/ bionic-security main restricted universe multiverse
deb https://mirrors.ustc.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src https://mirrors.ustc.edu.cn/ubuntu/ bionic-proposed main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu/ xenial main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial main
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates main
deb http://mirrors.aliyun.com/ubuntu/ xenial universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-updates universe
deb http://mirrors.aliyun.com/ubuntu/ xenial-security main
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security main
deb http://mirrors.aliyun.com/ubuntu/ xenial-security universe
deb-src http://mirrors.aliyun.com/ubuntu/ xenial-security universe

#change_conda.txt
channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
  - defaults
show_channel_urls: true
custom_channels:
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch-lts: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud

• pip的源直接在dockerfile中指定了pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
• 从github下载的均使用了它的镜像网站github.cnpmjs.org
• 所以如果不熟悉docker挂https_proxy，或者懒的话，国内就根据自己需要换掉apt , pip , conda, github的源

# test_cuda.py
import torch
print(torch.__version__)
print(torch.cuda.is_available())  # cuda是否可用
print(torch.cuda.device_count())  # 返回GPU的数量
print(torch.cuda.get_device_name(0))  # 返回gpu名字，设备索引默认从0开始

# test_mujoco.py
import mujoco_py
import os
mj_path = mujoco_py.utils.discover_mujoco()
xml_path = os.path.join(mj_path, 'model', 'humanoid.xml')
model = mujoco_py.load_model_from_path(xml_path)
sim = mujoco_py.MjSim(model)

print(sim.data.qpos)
sim.step()
print(sim.data.qpos)

# test_pybullet.py
import os
import gym
import pybullet_envs

from stable_baselines3.common.vec_env import DummyVecEnv, VecNormalize
from stable_baselines3 import PPO

env = DummyVecEnv([lambda: gym.make("HalfCheetahBulletEnv-v0")])
# Automatically normalize the input features and reward
env = VecNormalize(env, norm_obs=True, norm_reward=True,
                   clip_obs=10.)

model = PPO('MlpPolicy', env)
model.learn(total_timesteps=2000)
print("finish learning")
# Don't forget to save the VecNormalize statistics when saving the agent
log_dir = "/tmp/"
model.save(log_dir + "ppo_halfcheetah")
print("saving model at {0}".format(log_dir))
stats_path = os.path.join(log_dir, "vec_normalize.pkl")
env.save(stats_path)
print("saving env at {}".format(stats_path))
# To demonstrate loading
del model, env

# Load the saved statistics
env = DummyVecEnv([lambda: gym.make("HalfCheetahBulletEnv-v0")])
print("loading env and model")
env = VecNormalize.load(stats_path, env)
#  do not update them at test time
env.training = False
# reward normalization is not needed at test time
env.norm_reward = False

# Load the agent
model = PPO.load(log_dir + "ppo_halfcheetah", env=env)

# test_stable_baselines.py
import gym
from stable_baselines3 import PPO

env = gym.make("Ant-v3")
model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=1000)

obs = env.reset()
for i in range(1000):
    action, _states = model.predict(obs, deterministic=True)
    obs, reward, done, info = env.step(action)
    #env.render()
    if done:
      obs = env.reset()

env.close()

这些基本是从各官网copy下来的例子，进行测试。

FROM nvidia/cuda:10.0-base

COPY ./change_apt.txt /
RUN cp /etc/apt/sources.list /etc/apt/sources_init.list 
	&& cat /change_apt.txt > /etc/apt/sources.list 
	&& rm /change_apt.txt 
	&& apt update

RUN apt update && DEBIAN_FRONTEND=noninteractive apt install -y --allow-unauthenticated --no-install-recommends 
    build-essential apt-utils cmake git curl vim ca-certificates sudo tmux jq unrar
    libjpeg-dev libpng-dev 
    libgtk3.0 libsm6 cmake ffmpeg pkg-config 
    qtbase5-dev libqt5opengl5-dev libassimp-dev 
    libboost-python-dev libtinyxml-dev bash 
    wget unzip libosmesa6-dev software-properties-common 
    libopenmpi-dev libglew-dev openssh-server 
    libosmesa6-dev libgl1-mesa-glx libgl1-mesa-dev patchelf libglfw3 
    && rm -rf /var/lib/apt/lists/*

RUN useradd --create-home --shell /bin/bash pamirl 
	&& adduser pamirl sudo 
	&& echo 'pamirl:qwerty' | chpasswd 
	&& su pamirl

USER pamirl
WORKDIR /home/pamirl
# miniconda
RUN wget -q https://mirrors.tuna.tsinghua.edu.cn/anaconda/miniconda/Miniconda3-latest-Linux-x86_64.sh && 
    bash Miniconda3-latest-Linux-x86_64.sh -b -p miniconda3 && 
    rm Miniconda3-latest-Linux-x86_64.sh
ENV PATH /home/pamirl/miniconda3/bin:$PATH
# mujoco
RUN wget https://github.com.cnpmjs.org/deepmind/mujoco/releases/download/2.1.0/mujoco210-linux-x86_64.tar.gz 
		&& mkdir -p .mujoco 
		&& whoami 
		&& pwd 
		&& tar -zxf mujoco210-linux-x86_64.tar.gz -C "$HOME/.mujoco" 
		&& rm mujoco210-linux-x86_64.tar.gz
ENV LD_LIBRARY_PATH /home/pamirl/.mujoco/mujoco210/bin:${LD_LIBRARY_PATH}

# conda env
COPY ./change_conda.txt ./
RUN conda config --describe 
	&& cat ./change_conda.txt > .condarc 
	&& rm change_conda.txt 
	&& conda create -y -n deep_rl python=3.8

SHELL ["/bin/bash", "--login", "-c"]

# basis pytorch
RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple 
	&& source activate deep_rl 
	&& cat .condarc 
	&& pip install -q dm_control gym 
	&& pip install -U 'mujoco-py<2.2,>=2.1' 
	&& conda install -y pytorch=1.10.1 torchvision torchaudio cudatoolkit=10.2 -c https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch 
	&& pip install glfw Cython imageio lockfile

# stable-baselines3
RUN pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple 
	&& source activate deep_rl 
	&& pip install -U stable-baselines3[extra] pyglet pybullet

# atari
#RUN source activate deep_rl 
#	&& wget -q http://www.atarimania.com/roms/Roms.rar 
#	&& mkdir ~/atari_roms
#	&& unrar e Roms.rar ~/atari_roms 
#	&& python -m atari_py.import_roms ~/atari_roms 
#	&& rm Roms.rar 
#	&& rm -rf ~/atari_roms

# install jupyerlab
RUN source activate deep_rl 
	&& conda install jupyterlab ipykernel 
	&& python -m ipykernel install --user --name deep_rl --display-name deep_rl:stable-baselines3

# configure jupyter lab & vscode
RUN source activate deep_rl 
	&& jupyter lab --generate-config 
	&& cd .jupyter 
	#&& echo c.ServerApp.password=$(cat jupyter_server_config.json | jq .ServerApp.password) >> jupyter_lab_config.py 
	&& echo "c.ServerApp.password = "$(python -c "from jupyter_server.auth import passwd;print(passwd('qwerty'))")"" >> jupyter_lab_config.py 
	&& echo "c.ServerApp.ip = '*'" >> jupyter_lab_config.py 
	&& echo "c.ServerApp.allow_remote_access=True" >> jupyter_lab_config.py

# Add test scripts
ADD test*.py ./
# Add conda dependency of your code
RUN source activate deep_rl 
	&& conda install -y scikit-learn tensorboardX 
	&& conda init

系统用户、密码、jupyterlab的配置、端口都可根据自行需要修改

1. GPU是RTX 3070或3090，就需要指定cudatoolkit=11.3，否则会报错误：GeForce RTX 3090 with CUDA capability sm_86 is not compatible with the current PyTorch installation
2. 如果使用cuda:11.0-base基础镜像，会出现apt无法安装的问题，因为11.0的操作系统是Ubuntu 20.04 LTS，需要对apt的源进行额外更换