- 错误信息：
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 22/05/24 09:41:52 INFO SparkContext: Running Spark version 3.0.0 22/05/24 09:41:52 INFO ResourceUtils: ============================================================== 22/05/24 09:41:52 INFO ResourceUtils: Resources for spark.driver: 22/05/24 09:41:52 INFO ResourceUtils: ============================================================== 22/05/24 09:41:52 INFO SparkContext: Submitted application: DataDispose$ 22/05/24 09:41:52 INFO SecurityManager: Changing view acls to: yikuai 22/05/24 09:41:52 INFO SecurityManager: Changing modify acls to: yikuai 22/05/24 09:41:52 INFO SecurityManager: Changing view acls groups to: 22/05/24 09:41:52 INFO SecurityManager: Changing modify acls groups to: 22/05/24 09:41:52 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yikuai); groups with view permissions: Set(); users with modify permissions: Set(yikuai); groups with modify permissions: Set() 22/05/24 09:41:54 INFO Utils: Successfully started service 'sparkDriver' on port 50862. 22/05/24 09:41:54 INFO SparkEnv: Registering MapOutputTracker 22/05/24 09:41:54 INFO SparkEnv: Registering BlockManagerMaster 22/05/24 09:41:54 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 22/05/24 09:41:54 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 22/05/24 09:41:54 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 22/05/24 09:41:54 INFO DiskBlockManager: Created local directory at C:UsersyikuaiAppDataLocalTempblockmgr-90db67ff-bb82-4670-bcc8-db1e2d3e3d42 22/05/24 09:41:54 INFO MemoryStore: MemoryStore started with capacity 1983.3 MiB 22/05/24 09:41:54 INFO SparkEnv: Registering OutputCommitCoordinator 22/05/24 09:41:54 INFO Utils: Successfully started service 'SparkUI' on port 4040. 22/05/24 09:41:54 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://DESKTOP-T2AEBTN:4040 22/05/24 09:41:54 INFO Executor: Starting executor ID driver on host DESKTOP-T2AEBTN
- 分析：
需要修改spark的显示日志等级
- 解决：
在resource文件夹下，创建一个log4j.properties文件，将以下内容复制到该文件即可
log4j.rootCategory=ERROR, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
# Set the default spark-shell log level to ERROR. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=ERROR
# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=ERROR
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=ERROR
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=ERROR
log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR
# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR
参考文章：https://blog.csdn.net/OWBY_Phantomhive/article/details/123086181
