Spark需要的Maven仓库及修改Log4j日志输出级别

整新IDEA环境的时候下了很长时间的依赖,就把下载过的Maven仓库打包了,方便下次使用

其中scala有2.1.1和2.4.8两个版本,但名字是2.1.1

    4.0.0

    spark
    spark
    1.0-SNAPSHOT


    
        
            org.apache.spark
            spark-core_2.11
            2.4.8
        

        
            mysql
            mysql-connector-java
            8.0.25
        

    

    
        World
        
            
                net.alchim31.maven
                scala-maven-plugin
                3.3.2
                
                    
                        
                            compile
                            testCompile
                        
                    
                
            

            
                org.apache.maven.plugins
                maven-assembly-plugin
                3.0.0
            
        
    

链接：https://pan.baidu.com/s/188sxt4ihtfF4DUOolORqNg 
提取码：ihww 

log4j 修改日志输出级别

在项目中创建文件夹:resources

在resources中创建文本修改名字为:log4j.properties添加以下内容:

log4j.rootCategory=ERROR, console
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

# Set the default spark-shell log level to WARN. When running the spark-shell, the
# log level for this class is used to overwrite the root logger's log level, so that
# the user can have different defaults for the shell and regular Spark apps.
log4j.logger.org.apache.spark.repl.Main=WARN

# Settings to quiet third party logs that are too verbose
log4j.logger.org.spark_project.jetty=WARN
log4j.logger.org.spark_project.jetty.util.component.AbstractLifeCycle=ERROR
log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO

# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent UDFs in SparkSQL with Hive support
log4j.logger.org.apache.hadoop.hive.metastore.RetryingHMSHandler=FATAL
log4j.logger.org.apache.hadoop.hive.ql.exec.FunctionRegistry=ERROR