问题关键:
Exception: Python in worker has different version 3.7 than that in driver 3.8, PySpark cannot run with different minor versions. Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.
参考 Python运行spark时出现版本不同的错误
pyspark提交集群报错Exception: Python in worker has different version 2.7 than that in driver 3.7, PySpark
原因在于我使用python3执行关于pyspark的py文件时,这个python3软链指向的python的版本是3.8,而我配置的conda和spark,pyspark的版本都是3.7.6,所以应该使用3.7版本的python去执行py文件,我设置的python使用的是conda的python,用python xx.py则不会报以上错误



