栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 软件开发 > 后端开发 > Python

python下的pyspark报错集锦

Python 更新时间: 发布时间: IT归档 最新发布 模块sitemap 名妆网 法律咨询 聚返吧 英语巴士网 伯小乐 网商动力

python下的pyspark报错集锦

1.WARN Utils: Service ‘SparkUI’ could not bind on port 4040. Attempting port 4041.

出现这种错误是是在spark启动从节点时出现的。 
解决的方法是,在spark-env.sh中加入一条 
SPARK_LOCAL_IP=127.0.0.1 
然后就完美解决报错了! 
D:sparkspark-2.2.0-bin-hadoop2.7bin找到load-spark-env.sh,之后notepad打开,增加如下,完成 

2.WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/D:/spark/spark-2.2.0-bin-hadoop2.7/jars/hadoop-auth-2.7.3.jar) to method sun.security.krb5.Config.getInstance() 
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil 
WARNING: Use –illegal-access=warn to enable warnings of further illegal reflective access operations 
WARNING: All illegal access operations will be denied in a future release 
18/05/11 17:07:07 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable

出现以上错误,由于JDK版本为9,太高了,但是不影响运行。可以无事

3.ython in worker has different version 3.6 than that in driver 3.5, PySpark cannot run with different minor versions.Please check environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHON are correctly set.

问题解决:

import os
os.environ["PYSPARK_PYTHON"]="D:office3python\anaconda3.5\3.5envspython35\python"12

指定运行的python环境位置。


转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/221081.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号