[ Spark ] java.lang.NoSuchMethodError: com.google.common.hash.Funnels.stringFunnel

• • 问题
  • 解决
  • • 两种方式
    • • 一. 初始化 spark 时，设置优先使用 jar 包中的依赖
      • 二. 使用 shade 插件
  • 参考链接

• 相关环境参数

spark.version：2.1.0
scala.version：2.11.8
guava：22.0

• 代码中使用到 guava 的布隆过滤器，产生如下报错

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.hash.Funnels.stringFunnel(Ljava/nio/charset/Charset;)Lcom/google/common/hash/Funnel;
	at BloomFilterCase$.main(BloomFilterCase.scala:11)
	at BloomFilterCase.main(BloomFilterCase.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:853)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:928)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:937)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

• spark-core_2.11 中也有使用到 guava，先排除掉

    
        com.google.guava
        guava
    

val spark = SparkSession.builder
      .config("spark.executor.userClassPathFirst", value = true)
      .config("spark.driver.userClassPathFirst", value = true)
      .getOrCreate()

• 这种方式可以解决 guava 的报错，但是会带来新的问题
• 如下 序列化的问题

Caused by: java.io.InvalidClassException: org.apache.spark.scheduler.Task;
 local class incompatible: stream classdesc serialVersionUID = 
 -3444186963810984713, local class serialVersionUID = -8136233304550334863

• 需要去掉 assembly 插件

maven-assembly-plugin

• 引入 maven-shade-plugin 插件

    org.apache.maven.plugins
    maven-shade-plugin
    3.1.0
    
        
            package
            
                shade
            
            
                
                    
                        
                        MXHistoryBloomFilter
                    
                
                
                    
                        com.google.common
                        my_guava.common
                    
                
                
                    
                        *:*
                        
                            meta-INF/maven/**
                            meta-INF/*.SF
                            meta-INF/*.DSA
                            meta-INF/*.RSA
                        
                    
                
            
        
    

• https://zhuanlan.zhihu.com/p/44956574
• https://blog.csdn.net/adorechen/article/details/90722933