【Spark】Spark开发报错总结

【报错一】
Scala文件编译报错：

【报错原因】
Scala自2.11版本开始，移除了弃用的-make选项（https://groups.google.com/forum/#!msg/scala-tools/9xLQO263sjg/ThXw-uqKCG4J），而idea的scala插件没有更新，因此，Scala升级到2.11后，就出现了上述错误。
【解决方案】
编辑H:IDEAWorkspaceGuPaoStudySparkCoreDemo.ideascala_compiler.xml文件，注释掉即可。

## scala_compiler.xml



  
    
      
        
        
        
      
    
  

【报错二】

【报错原因】Scala项目的pom.xml配置文件导包不正确。
【解决方案】pom.xml配置文件如下：

  4.0.0
  org.example
  GuPaoStudySparkCoreDemo
  1.0-SNAPSHOT
  ${project.artifactId}
  My wonderfull scala app
  2010

  
    1.8
    1.8
    UTF-8
    2.11.8
  

  
    
      nexus-aliyun
      http://maven.aliyun.com/nexus/content/groups/public
    
  

  
    
      org.apache.spark
      spark-core_2.11
      2.4.0
    
    
      org.apache.spark
      spark-streaming_2.11
      2.4.0
    
    
      org.apache.spark
      spark-yarn_2.11
      2.4.0
    
    
      org.apache.spark
      spark-mllib-local_2.11
      2.4.0
    
    
      org.apache.spark
      spark-mllib_2.11
      2.4.0
    
    
      org.apache.spark
      spark-sql_2.11
      2.4.0
    
    
      org.antlr
      antlr4-runtime
      4.9.3
    
    
      org.slf4j
      slf4j-api
      1.7.5
    
    
      org.scala-lang
      scala-library
      ${scala.version}
    
    
      org.eclipse.jetty
      jetty-util
      9.4.33.v20201020
    
    
      junit
      junit
      4.8.1
      test
    
    
      org.scalatest
      scalatest_2.11
      3.0.5
      test
    
    
      org.springframework
      spring-context
      4.3.11.RELEASE
    
    
      org.apache.spark
      spark-hive-thriftserver_2.11
      2.4.0
      provided
    
    
      org.mybatis
      mybatis
      3.5.1
    
    
      com.redislabs
      spark-redis
      2.4.0
    
    
      com.alibaba
      druid
      1.1.20
    
  

  
    src/main/scala
    src/test/scala
    
      
        net.alchim31.maven
        scala-maven-plugin
        3.3.1
        
          
            
              compile
              testCompile
            
            
              
                -feature
                -deprecation
                -dependencyfile
                ${project.build.directory}/.scala_dependencies
              
            
          
        
      
      
        org.apache.maven.plugins
        maven-surefire-plugin
        2.18.1
        
          false
          true
          
          
          
            ***Suite.*
          
        
      
      
        org.codehaus.mojo
        exec-maven-plugin
        1.5.0
        
          
            run-local
            
              exec
            
            
              spark-submit
              
                --master
                local
                ${project.build.directory}/${project.artifactId}-${project.version}-uber.jar
              
            
          
          
            run-yarn
            
              exec
            
            
              
                
                  ${basedir}/spark-remote/conf
                
              
              spark-submit
              
                --master
                yarn
                ${project.build.directory}/${project.artifactId}-${project.version}-uber.jar
              
            
          
        
      
      
        org.apache.maven.plugins
        maven-shade-plugin
        3.1.0
        
          
            package
            
              shade
            
            
              
                
                  *:*
                  
                    meta-INF/*.SF
                    meta-INF/*.DSA
                    meta-INF/*.RSA
                  
                
              
              
                
                  
                    net.martinprobson.spark.spark_example.SparkTest
                  
                
              
              
                
                  javax.servlet:*
                  org.apache.hadoop:*
                  org.apache.maven.plugins:*
                  org.apache.spark:*
                  org.apache.avro:*
                  org.apache.parquet:*
                  org.scala-lang:*
                
              
              ${project.artifactId}-${project.version}-uber
            
          
        
      
    
  

删除src/test目录下的所有内容，重新编译Scala文件，报错即可解除。