springboot打jar包在spark yarn 集群上部署运行

1.8
2.12.15
2.12
3.1.2
3.0.8
3.3.0
2.5.2
2.10.5

/home/software/spark-3.1.2/bin/spark-submit --class cnki.bdms.servicespark.BdcServiceSparkApplication 
	--conf spark.yarn.jars="hdfs://0.0.0.0:8020/spark/apps/jars/*,hdfs://0.0.0.0:8020/lib/*" 
	--driver-java-options "-Dorg.springframework.boot.logging.LoggingSystem=none -Dspring.profiles.active=test -Dspark.yarn.dist.files=/home/software/hadoop-3.3.0/etc/hadoop/yarn-site.xml" 
    --master yarn 
    --deploy-mode cluster 
    --driver-memory 4g 
    --executor-memory 2g 
    --executor-cores 1 
    --queue default 
    /data/jar/bdcServiceSpark-1.0.0.jar

@SpringBootApplication(exclude= {DataSourceAutoConfiguration.class,QuartzAutoConfiguration.class, MongoAutoConfiguration.class, ElasticsearchDataAutoConfiguration.class, SecurityAutoConfiguration.class, RedisAutoConfiguration.class})
public class BdcServiceSparkApplication implements CommandLineRunner {
    public static void main(String[] args) {
        if(args==null||args.length==0){
            args=new String[]{"1495"};
//            System.out.println("缺少资源id,程序退出");
//            return;
        }
        SpringApplication.run(BdcServiceSparkApplication.class, args);
    }

    @Override
    public void run(String... args) throws Exception {
        System.out.println("==================程序开始====================");
        //应用程序的逻辑，这是是利用反射执行对应的服务类的方法
        WorkFlowDirectorServiceImpl flowDirectorService = SpringUtils.getBean(WorkFlowDirectorServiceImpl.class);
        int dataAnalyseId=Integer.parseInt(args[0]);
        flowDirectorService.direct(dataAnalyseId);
        SpringApplication.exit(SpringUtils.getApplicationContext());
        System.out.println("==================程序结束====================");
    }
}

spark不支持使用spring-boot-maven-plugin打包的springboot项目结构，要改成

maven-shade-plugin 打包

 
        
            
                org.apache.maven.plugins
                maven-shade-plugin
                
                    
                        org.springframework.boot
                        spring-boot-maven-plugin
                        2.4.11
                    
                
                
                    
                        package
                        
                            shade
                        
                    
                
                
                    false
                    false
                    
                        
                            *:*
                            
                                meta-INF/*.SF
                                meta-INF/*.DSA
                                meta-INF/*.RSA
                            
                        
                    
                    
                        
                            meta-INF/spring.handlers
                        
                        
                            meta-INF/spring.factories
                        
                        
                            meta-INF/spring.schemas
                        
                        
                        
                            cnki.bdms.servicespark.BdcServiceSparkApplication
                        
                    
                
            
        
    

 这一块写成自己的启动类

1、 snakeyaml.jar冲突，保留springboot的版本，直接把spark.yarn.jars这个路径上的删除掉，保留最新版本

2、gson-2.4.jar包冲突，改成和springboot的版本一致的，比如我的是gson-2.7.jar，直接把spark.yarn.jars这个路径上的删除掉，换成gson-2.7.jar

3. guava-14.0.1.jar冲突 提示java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V

升级到guava-20.0.jar，将guava-20.0.jar替换到spark.yarn.jars路径

4、jackson-databind版本太高，提示Scala moudle 2.10.0 require Jackson Databind version >=2.10.0 and <2.11.0 ,我这边环境里spark.yarn.jars里的是2.10.5，springboot里用的是2.11.4，采取方案是直接降级springboot里的jackson-databind

先排除所有的

org.apache.hadoop 下的jackson,如

    org.apache.hadoop
    hadoop-client
    ${hadoop.version}
    
        
            com.fasterxml.jackson.module
            *
        
        
            com.fasterxml.jackson.core
            *
        
    

一共5个，

hadoop-common hadoop-hdfs hadoop-mapreduce-client-core hadoop-yarn-common hadoop-client

排除所有org.apache.spark下的jackson

    org.apache.spark
    spark-sql_${scala.binary.version}
    ${spark.version}
    
        
            com.fasterxml.jackson.module
            *
        
        
            com.fasterxml.jackson.core
            *
        
    

一共3个，spark-core,spark-sql,spark-yarn

具体的看自己项目的依赖，降级处理，然后添加依赖

    
        
            
                com.fasterxml.jackson.dataformat
                jackson-dataformat-yaml
                2.10.5
            
            
                com.fasterxml.jackson.core
                jackson-databind
                2.10.5
            
        
    

 也可以直接使用 dependencyManagement 控制版本这样就不用上面的挨个排除了

Maven 可以通过 dependencyManagement 元素对依赖进行管理，它具有以下 2 大特性：

在该元素下声明的依赖不会实际引入到模块中，只有在 dependencies 元素下同样声明了该依赖，才会引入到模块中。该元素能够约束 dependencies 下依赖的使用，即 dependencies 声明的依赖若未指定版本，则使用 dependencyManagement 中指定的版本，否则将覆盖 dependencyManagement 中的版本。

    
        
            
                com.fasterxml.jackson.core
                jackson-core
                ${fasterxml.jackson.version}
            
            
                com.fasterxml.jackson.core
                jackson-databind
                ${fasterxml.jackson.version}
            
            
                com.fasterxml.jackson.core
                jackson-annotations
                ${fasterxml.jackson.version}
            
            
            
                com.fasterxml.jackson.module
                jackson-module-scala_${scala.binary.version}
                ${fasterxml.jackson.version}
            

            
                com.fasterxml.jackson.jaxrs
                jackson-jaxrs-base
                ${fasterxml.jackson.version}
            
            
                com.fasterxml.jackson.jaxrs
                jackson-jaxrs-json-provider
                ${fasterxml.jackson.version}
            
            
                com.fasterxml.jackson.module
                jackson-module-jaxb-annotations
                ${fasterxml.jackson.version}
            
            
                com.fasterxml.jackson.module
                jackson-module-paranamer
                ${fasterxml.jackson.version}
            
        
    

https://blog.csdn.net/hzs33/article/details/83183217

https://blog.csdn.net/laksdbaksjfgba/article/details/86023892

https://blog.csdn.net/suwei825/article/details/119236798