准备条件spark on hadoop环境已经安装好
版本：
spark-3.2.1-bin-hadoop3.2
hadoop-3.2.2
操作系统：
ubuntu 18.04
安装步骤：
1、执行安装pip3
apt-get update
apt-get upgrade
apt-get update
apt install python3-pip
2、下载并安装notebook
pip3 install --user jupyterlab notebook
pip3 install --user findspark
///
root@zw001:/opt# pip3 install --user jupyterlab notebook -i https://pypi.org/simple
Collecting jupyterlab
Using cached https://files.pythonhosted.org/packages/4b/0d/03deff4501e9ffafe755e561e375ffa9f5822fec93a09ce1c7c5147bdcb3/jupyterlab-3.2.9-py3-none-any.whl
Collecting notebook
Downloading https://files.pythonhosted.org/packages/27/b7/7e602dc8b868bba8a542269205237b400be3427d8489b5851de5f7587996/notebook-6.4.10-py3-none-any.whl (9.9MB)
51% |████████████████▍ | 5.1MB 120kB/s eta 0:00:41
root@zw001:/usr/local# pip3 install --user findspark
WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip.
Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue.
To avoid this problem you can invoke Python with '-m pip' instead of running pip directly.
Collecting findspark
Downloading findspark-2.0.1-py2.py3-none-any.whl (4.4 kB)
Installing collected packages: findspark
Successfully installed findspark-2.0.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
root@zw001:/usr/local#
3、修改/root/.local/share/jupyter/kernels/python3的kernel.json文件，添加配置如下
{
"display_name": "PySpark",
"language": "python",
"argv": [
"/usr/bin/python3",
"-m",
"ipykernel",
"-f",
"{connection_file}"
],
"env": {
"SPARK_HOME": "/opt/spark-3.2.1-bin-hadoop3.2",
"PYTHONPATH": "/opt/spark-3.2.1-bin-hadoop3.2/python:/opt/spark-3.2.1-bin-hadoop3.2/python/lib/py4j-0.10.9.3-src.zip",
"PYTHONSTARTUP": "/opt/spark-3.2.1-bin-hadoop3.2/python/pyspark/shell.py",
"PYSPARK_SUBMIT_ARGS": "--master yarn --deploy-mode client --num-executors 2 --executor-memory 2g --executor-cores 2 pyspark-shell"
}
}
4、生成jupyter的初始配置文件和密码，并配置配置文件
安装目录在：
/root/.local/
/root/.local/share/jupyter/kernels/python3
root@zw001:/opt/spark-3.2.1-bin-hadoop3.2/bin# jupyter notebook --generate-config
Writing default config to: /root/.jupyter/jupyter_notebook_config.py
root@zw001:/opt/spark-3.2.1-bin-hadoop3.2/bin# ipython
Python 3.6.9 (default, Jun 29 2022, 11:45:57)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.3 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from notebook.auth import passwd
In [2]: passwd()
Enter password: root
Verify password: root
Out[2]: 'argon2:$argon2id$v=19$m=10240,t=10,p=8$23gK0IXt5P9kufjOH5s2CQ$hn5kY4lpHK9ucLmo/TqRkKVIURb9WlhifLtxXyLRP4c'
修改配置并且运行远程访问和root运行：
vi /root/.jupyter/jupyter_notebook_config.py
# 去掉每个条目前的注释 '#'
c.NotebookApp.ip = '*' # 设置Jupyter监听的ip地址，修改为*表示监听所有ip地址,最好绑定一个特定的自己的ip，不然所有位置都可以访问
c.NotebookApp.password = u'argon2:$argon2id$v=19$m=10240,t=10,p=8$23gK0IXt5P9kufjOH5s2CQ$hn5kY4lpHK9ucLmo/TqRkKVIURb9WlhifLtxXyLRP4c' # 将该内容替换为上一步设置密码时生成的sha1值
c.NotebookApp.open_browser = False # 禁止启动时自动打开浏览器(本来在桌面平台上安装使用时可以开启，在服务器上不需要此设置，因此设置为False)
c.NotebookApp.port = 1024 # 指定访问的端口，按照自己喜好设定，默认是8888，注意不要和已用端口冲突
c.NotebookApp.notebook_dir = '/Your/Directory' # 设置运行时的目录，因为以root身份运行时默认会在/root目录下，因此最好修改成自己喜欢的目录，例如'/home/jupyter'
c.NotebookApp.allow_remote_access = True
## Whether to allow the user to run the notebook as root.
c.NotebookApp.allow_root = True
5、修改spark-env.sh 增加运行环境参数配置：
export PYSPARK_PYTHON='python3'
export PYSPARK_DRIVER_PYTHON='python3'
export PYSPARK_DRIVER_PYTHON_OPTS=' -m jupyter notebook'
6、执行启动在spark的bin目录下执行
./pyspark
7、参照apache spark 官网进行执行pyspark
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
错误和问题解决：
1、错误1：
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-r_j4mv6z/argon2-cffi-bindings/
执行：
pip3 install --upgrade pip
root@zw001:/usr/local# pip3 install --upgrade pip
Cache entry deserialization failed, entry ignored
Collecting pip
Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out. (read timeout=15)",)': /packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl
Downloading https://files.pythonhosted.org/packages/a4/6d/6463d49a933f547439d6b5b98b46af8742cc03ae83543e4d7688c2420f8b/pip-21.3.1-py3-none-any.whl (1.7MB)
100% |████████████████████████████████| 1.7MB 62kB/s
Installing collected packages: pip
Found existing installation: pip 9.0.1
Not uninstalling pip at /usr/lib/python3/dist-packages, outside environment /usr
Successfully installed pip-21.3.1
2、错误处理2：
WARNING: The scripts jupyter, jupyter-migrate and jupyter-troubleshoot are installed in '/root/.local/bin' which is not on PATH.
将 /root/.local/bin 添加到环境变量,编辑/etc/profile,添加一下配置，执行 source /etc/profile
export PATH=$PATH:/root/.local/bin
