目录
1. 基于docker部署cdh
2. 遇到的异常
2.1 flink下缺少hadoop相关依赖
2.2 jdk7造成的错误
3. 启动flink on yarn模式
3.1 调整yarn集群可用内存大小
3.2指定flink submit端口库，固定下来
4. 启动flink on yarn的wordcount例子
1. 基于docker部署cdh
基于Docker搭建CDH单机单节点集群_litlit023的专栏-CSDN博客
2. 遇到的异常
2.1 flink下缺少hadoop相关依赖
基于docker 部署cdn后，启动flink on yarn模式，报了以下错误
./bin/yarn-session.sh -n 2 -jm 1024 -tm 1024
报了以下错误
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:650)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:632)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 7 more
解决方法：
提示flink启动缺少Hadoop yarn的相关jar包，从官网下载Flink依赖Hadoop的相关jar包；
Apache Flink: Downloads
进入官网，下载对应的相关依赖jar包，放到flink lib目录下即可
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException - RICH-ATONE - 博客园Apache Flink: Downloads
根据官网部署Flink on YARN过程中遇到的问题及解决方法 - 简书
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException - RICH-ATONE - 博客园2
2.2 jdk7造成的错误
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1643103905134_0001
at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1219)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:607)
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:419)
... 7 more
2022-01-25 17:49:17,827 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cancelling deployment from Deployment Failure Hook
2022-01-25 17:49:17,840 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
2022-01-25 17:49:17,845 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Killing YARN application
2022-01-25 17:49:17,931 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Killed application application_1643103905134_0001
2022-01-25 17:49:18,033 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deleting files in hdfs://quickstart.cloudera:8020/user/root/.flink/application_1643103905134_0001.
[root@vm01 flink-1.14.3]# yarn logs -applicationId application_1643103905134_0001
22/01/25 17:49:32 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
Container: container_1643103905134_0001_01_000001 on quickstart.cloudera_8041
===============================================================================
LogType:jobmanager.err
Log Upload Time:星期二 一月 25 17:49:19 +0800 2022
LogLength:160
Log Contents:
Unrecognized VM option 'MaxmetaspaceSize=268435456'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
LogType:jobmanager.out
Log Upload Time:星期二 一月 25 17:49:19 +0800 2022
LogLength:0
Log Contents:
Unrecognized VM option 'MaxmetaspaceSize=268435456'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
需要将cdh组件所用的jdk改为jdk8以上，我的环境时基于docker来部署的cdh 5.x，因此进入到容器内，修改容器内的jdk为jdk8即可
修改jdk为自己的jdk8
进入到容器docker exec -it mycdh3 bash docker cp /work/jdk/jdk1.8.0_271 02155f0f3679:/root/ cd /usr/java/jdk1.7.0_67-cloudera rm -rf ./* cp -r /root/jdk1.8.0_271/* /usr/java/jdk1.7.0_67-cloudera/
修改jdk版本：
Flink on yarn JDK 版本支持问题
java - Unrecognized option: -MaxmetaspaceSize=256m - Stack Overflow
How to configure java home on flink job on yarn - JiKe DevOps Community
java - Unrecognized option: -MaxmetaspaceSize=256m - Stack Overflow
Exposing a port on a live Docker container - Stack Overflow
flink on yarn 集群模式启动报错及解决方案汇总_大数据技术博客-CSDN博客_flink on yarn 报错
flink on yarn遇到的yarn资源不足问题_无名子七叶莲的博客-程序员宝宝 - 程序员宝宝
3. 启动flink on yarn模式
3.1 调整yarn集群可用内存大小
容器内存
yarn.nodemanager.resource.memory-mb
默认是3G，我们改为5G
3.2指定flink submit端口库，固定下来
启动yarn session模式前，修改flink中conf文件下的flink-conf.yml文件，固定flink作业提交的端口
，否则由于容器没有对外暴露随机端口，会无法提交flink 作业。
在启动docker容器时，也需要指定暴露的端口
docker run -id --hostname=quickstart.cloudera --privileged=true -t -i -d -p 8020:8020 -p 7180:7180 -p 21050:21050 -p 50070:50070 -p 50075:50075 -p 50010:50010 -p 50020:50020 -p 8890:8890 -p 60010:60010 -p 10002:10002 -p 25010:25010 -p 25020:25020 -p 18088:18088 -p 8088:8088 -p 19888:19888 -p 7187:7187 -p 11000:11000 -t -p 8888:8888 -p 8032:8032 -p 8042:8042 -p 35399:35399 --name=mycdh3 cloudera/quickstart /usr/bin/docker-quickstart 进入容器，启动并停止 docker exec -it mycdh3 bash 启动cmf /home/cloudera/cloudera-manager --enterprise --force
#rest.bind-port: 8080-8090
rest.bind-port: 35399
vi flink-conf.yaml #============================================================================== # Rest & web frontend #============================================================================== # The port to which the REST client connects to. If rest.bind-port has # not been specified, then the server will bind to this port as well. # #rest.port: 8081 # The address to which the REST client will connect to # #rest.address: 0.0.0.0 # Port range for the REST and web server to bind to. # #rest.bind-port: 8080-8090 rest.bind-port: 35399
[root@vm01 flink-1.14.3]# ./bin/yarn-session.sh -jm 1024 -tm 1024
2022-01-25 20:38:55,885 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost
2022-01-25 20:38:55,894 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2022-01-25 20:38:55,896 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2022-01-25 20:38:55,897 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
2022-01-25 20:38:55,897 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2022-01-25 20:38:55,898 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2022-01-25 20:38:55,898 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2022-01-25 20:38:57,062 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop user set to root (auth:SIMPLE)
2022-01-25 20:38:57,095 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-3655855445473519286.conf.
2022-01-25 20:38:57,390 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/work/flink/streamx_dir/flink-1.14.3/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2022-01-25 20:38:57,951 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
2022-01-25 20:38:58,732 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-01-25 20:38:58,736 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-01-25 20:38:58,737 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction network memory (57.600mb (60397978 bytes)) is less than its min value 64.000mb (67108864 bytes), min value will be used instead
2022-01-25 20:38:59,287 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, slotsPerTaskManager=1}
2022-01-25 20:39:00,148 WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-01-25 20:39:05,590 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-01-25 20:39:05,671 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting application master application_1643114118293_0002
2022-01-25 20:39:05,740 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted application application_1643114118293_0002
2022-01-25 20:39:05,740 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for the cluster to be allocated
2022-01-25 20:39:05,745 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deploying cluster, current state ACCEPTED
2022-01-25 20:39:24,071 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - YARN application has been deployed successfully.
2022-01-25 20:39:24,072 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface quickstart.cloudera:36283 of application 'application_1643114118293_0002'.
JobManager Web Interface: http://quickstart.cloudera:36283
4. 启动flink on yarn的wordcount例子
cd /work/flink/streamx_dir/flink-1.14.3/bin
./flink run /work/flink/streamx_dir/flink-1.14.3/examples/batch/WordCount.jar
2022-01-25 21:22:47,893 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-root.
2022-01-25 21:22:47,893 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Found Yarn properties file under /tmp/.yarn-properties-root.
Executing WordCount example with default input data set.
Use --input to specify file input.
Printing result to stdout. Use --output to specify output path.
2022-01-25 21:22:48,877 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/work/flink/streamx_dir/flink-1.14.3/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2022-01-25 21:22:49,077 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at quickstart.cloudera/172.17.0.3:8032
2022-01-25 21:22:49,275 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - No path for the flink jar passed. Using the location of class org.apache.flink.yarn.YarnClusterDescriptor to locate the jar
2022-01-25 21:22:49,408 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface quickstart.cloudera:36283 of application 'application_1643114118293_0002'.
Job has been submitted with JobID 0daa3e5ea6c73fc6cb8cb1afbe995c01
Program execution finished
Job with JobID 0daa3e5ea6c73fc6cb8cb1afbe995c01 has finished.
Job Runtime: 14678 ms
Accumulator Results:
- 21336f04879a793df8159e7bdcbd1213 (java.util.ArrayList) [170 elements]
(a,5)
(action,1)
(after,1)
(against,1)
(all,2)
(and,12)
(arms,1)
(arrows,1)
(awry,1)
(ay,1)
(bare,1)
(be,4)
(bear,3)
(bodkin,1)
(bourn,1)
(but,1)
(by,2)
(calamity,1)
Flink on yarn执行Wordcount_想拥有一片海的博客-CSDN博客_flink on yarn运行wordcount
