- Container exited with a non-zero exit code 1
- 本地浏览器访问虚拟机域名配置
- 配置方法
- 登录验证
- 查看日志
- 环境问题
- ModuleNotFoundError: No module named 'py4j'
- pyspark重装
- spark-submit再次验证
- 报错:Incomplete HDFS URI, no host: hdfs:///test/softresult
- 报错:path hdfs://192.168.56.10:8020/test/softresult already exists.
- SUCCEEDED
昨天在dolphinscheduler涉及HDFS功能测试（三）spark task中遇到的额问题，一直没有解决，今天再来瞅一眼，就一眼
每次查看页面还要把虚拟机域名host1替换为具体IP,浏览器才能正常访问，太麻烦了
配置方法- 将本机的C:WindowsSystem32driversetc'hosts配置成和虚拟机/etc/hosts地址一致
查看输出日志
-
stderr（没发现有用信息）
-
stdout（总算摸到门路了）
具体日志：
Tools /home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/context.py:77: FutureWarning: Deprecated in 3.0.0. Use SparkSession.builder.getOrCreate() instead. /home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/dataframe.py:138: FutureWarning: Deprecated in 2.0, use createOrReplaceTempView instead. Traceback (most recent call last): File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/sparktasktest.py", line 42, in 环境问题df_result.coalesce(1).write.json(sys.argv[2]) File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/readwriter.py", line 846, in json File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/py4j-0.10.9.3-src.zip/py4j/java_gateway.py", line 1321, in __call__ File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/pyspark.zip/pyspark/sql/utils.py", line 111, in deco File "/home/dolphinscheduler/app/hadoop-2.7.3/data/tmp/nm-local-dir/usercache/dolphinscheduler/appcache/application_1655121288928_0003/container_1655121288928_0003_01_000001/py4j-0.10.9.3-src.zip/py4j/protocol.py", line 326, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o56.json. : java.io.IOException: Incomplete HDFS URI, no host: hdfs:///test/softresult at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:143) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.spark.sql.execution.datasources.DataSource.planForWritingFileFormat(DataSource.scala:461) at org.apache.spark.sql.execution.datasources.DataSource.planForWriting(DataSource.scala:556) at org.apache.spark.sql.DataFrameWriter.saveToV1Source(DataFrameWriter.scala:382) at org.apache.spark.sql.DataFrameWriter.saveInternal(DataFrameWriter.scala:355) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:239) at org.apache.spark.sql.DataFrameWriter.json(DataFrameWriter.scala:763) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:282) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:182) at py4j.ClientServerConnection.run(ClientServerConnection.java:106) at java.lang.Thread.run(Thread.java:748)
ModuleNotFoundError: No module named ‘py4j’
昨晚空跑脚本是可以成功的，现在看来应该还是脚本执行报错了，单独执行确实报模块不存在
在线安装，昨晚直接离线安装的，可能下载的包有问题
sudo /usr/local/python3/bin/pip3 install pyspark
- 再次执行脚本不再报模块缺失错
- 手动安装pyspark步骤（解压安装包执行sudo python3 setup.py install）
错误依旧：
网上说可能是没引入hadoop配置文件，结果一检查，发现配置的还真有问题（配的hadoop_home的地址）
- 索性HADOOP_HOME也配置上吧（conf/spark-env.sh）
export JAVA_HOME=/usr/local/java/jdk1.8.0_151 export HADOOP_HOME=/home/dolphinscheduler/app/hadoop-2.7.3 export HADOOP_CONF_DIR=/home/dolphinscheduler/app/hadoop-2.7.3/etc/hadoop export SPARK_PYTHON=/usr/local/bin/python3报错:path hdfs://192.168.56.10:8020/test/softresult already exists.
问题到这应该已经解决了，之前 no host: hdfs:///test/softresult，怀疑是目录不存在导致的，手动建立了，所以报目录已存在，指定新目录再次查看
- 终于成功了
- 验证结果，还是有点不完美，数据没写进去，这肯定是程序的问题了（接下来要研究python嘛？越研究越无知呀）
