Dolphinscheduler集成Flink任务踩坑记录

flink任务编写完成，在本地运行调试正常后，我打包提交到Dolphinscheduler平台进行测试。运行后没多久就报错：

 [taskAppId=TASK-10-108-214]:[138] -  -> java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer
		at com.bigdata.flink.FlinkKafka.main(FlinkKafka.java:30)
Caused by: java.lang.ClassNotFoundException: org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer

看日志详情，我们可以看到任务已经开始准备提交了，但是在环境里找不到FlinkKafkaConsumer。

flink run -m yarn-cluster -ys 1 -yjm 1G -ytm 2G -yqu default -p 1 -sae -c com.bigdata.flink.FlinkKafka flink-job/bigdata-flink.jar

我的打包方式是通过IDEA 菜单的Build Artfacts…打出的jar文件很小，而且依赖包也都没有被打进去。
于是换了一种方式：

• 配置pom文件

        
        
            
                org.codehaus.mojo
                build-helper-maven-plugin
                1.8
                
                    
                        add-source
                        generate-sources
                        
                            add-source
                        
                        
                            
                                src/main/scala
                                src/test/scala
                            
                        
                    
                    
                        add-test-source
                        generate-sources
                        
                            add-test-source
                        
                        
                            
                                src/test/scala
                            
                        
                    
                
            
            
                net.alchim31.maven
                scala-maven-plugin
                3.1.5
                
                    
                        
                            compile
                            testCompile
                        
                    
                
                
                    2.11.8
                
            
            
                org.apache.maven.plugins
                maven-compiler-plugin
                
                    1.8
                    1.8
                    utf-8
                
                
                    
                        compile
                        
                            compile
                        
                    
                
            

            
                maven-assembly-plugin
                
                    false
                    
                        jar-with-dependencies
                    
                    
                        
                            
                            com.bigdata.flink.FlinkKafka
                        
                    
                

                
                    
                        make-assembly
                        package
                        
                            assembly
                        
                    
                
            
        
        
    

• 执行maven下的package命令；可以看到这种打包方式执行后的jar包里包含了所需的依赖包。

环境依赖的问题解决后，在执行中又出现了HADOOP_CLASSPATH环境变量的报错。

taskAppId=TASK-10-112-218]:[138] -  -> 
	------------------------------------------------------------
	 The program finished with the following exception:
	
	org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpath. For more information refer to the "Deployment" section of the official Apache Flink documentation.
		at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
		at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
		at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114)
		at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812)
		at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246)
		at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
		at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
		at org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
		at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
	Caused by: java.lang.IllegalStateException: No Executor found. Please make sure to export the HADOOP_CLASSPATH environment variable or have hadoop in your classpath. For more information refer to the "Deployment" section of the official Apache Flink documentation.

这次需要修改worker所在节点的flink文件。

• 找到 flink/bin/flink文件
• 在第一行加入

export HADOOP_CLASSPATH=`hadoop classpath`

• 无需重启集群机器，直接重跑flink任务即可。