20211231
2021-06-25-Spark-39(ERROR TransportResponseHandler: Still have 1 requests outstanding when conne... https://www.jianshu.com/p/00dbcacb2e07
20211230
spark的输入和输出都基于hadoop的hdfs上面hdfs不别 "."的当前目录 直接从根目录开始
/data
http://blog.chinaunix.net/uid-29454152-id-5645182.html
关闭和修改日志
关闭屏幕输出
https://mp.weixin.qq.com/s/mo2hYHT13SSMp8iSrsG5xg
https://www.cnblogs.com/lenmom/p/12022038.html
https://blog.csdn.net/kyle1314608/article/details/122234853
上面两篇参考起来一起看
Spark-submit 参数调优完整攻略
SPARK_WORKER_WEBUI_PORT
工作者Web UI的端口（默认值：8081）。
配置参数是对应的？
https://blog.csdn.net/weixin_42712704/article/details/101556912
spark WARN scheduler.TaskSetManager: Lost task报错
https://blog.csdn.net/u011564172/article/details/69703662
Spark 任务调度之启动CoarseGrainedExecutorBackend
https://mp.weixin.qq.com/s/sC7_cvzfEbS3Gm2q1-2ECQ
Spark常见的脚本及参数详解和端口
https://blog.csdn.net/LastYHM/article/details/108715939
sparksubmit进程无法杀死
https://blog.csdn.net/weixin_30667301/article/details/99494592
spark进程关系
20211229
http://192.168.1.122:8080/#completed-app
spark控制台端口号
https://blog.csdn.net/chenyuangege/article/details/51513569
先启动hadoop再启动spark 只需要在master上启动就够了
启动关闭hadoop集群和spark集群 重点
bin/spark-submit --master spark://hadoop01:7077 --executor-memory 1G --total-executor-cores 2 /tmp/test.py pyspark集群提交任务 spark-submit为sparkbin里面的spark-submit 可以建立软连接 hadoop01是主机名称
https://www.cnblogs.com/luckyboylch/p/12567710.html
linux安装pyspark
https://blog.csdn.net/qq_15349687/article/details/82748074
https://www.cnblogs.com/zhangyongli2011/p/10572152.html
https://blog.csdn.net/a532672728/article/details/72358422
spark集群搭建
https://www.cnblogs.com/startnow/p/13697739.html
pyspark提交集群任务
https://blog.csdn.net/ruth13156402807/article/details/118962105
pyspark更好python版本
pyspark helloword test
from pyspark import SparkConf
from pyspark.sql import SparkSession
print("==========app start=========")
spark = SparkSession.builder.enableHiveSupport().getOrCreate()
df = spark.sql('show databases')
df.show()
print("==========app stop=========")
spark.stop()
