官网下载对应的版本
flink版本-1.13.6
hadoop版本-3.2.1
步骤：
首先安装jdk
1、配置hosts
vi /etc/hosts
例如：
192.168.0.1 flink1 192.168.0.2 flink2 192.168.0.3 flink3
2、防火墙关闭
firewall-cmd --state # 查看是否running systemctl stop firewalld #停止firewall systemctl disable firewalld #禁止firewall开机启动
3、ssh免密
datanode节点对namenode节点做免密
命令：
ssh-keygen -t rsa
生成id_rsa.pub文件
cat /root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
各个节点的id_rsa.pub文件添加到authorized_keys
4、配置环境变量
vi /etc/profile
HADOOP_HOME=/opt/hadoop/hadoop-3.2.1
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_PID_DIR=${HADOOP_HOME}/pids
export HADOOP_CONF_DIR=/opt/hadoop/hadoop-3.2.1/etc/hadoop
5、hadoop配置文件按功能对应修改
/etc/hadoop core-site.xml hadoop-env.sh hdfs-site.xml mapred-env.sh mapred-site.xml yarn-env.sh yarn-site.xml workers
格式化namenode
hdfs namenode -format
启动hadoop集群
start-all.sh 包含start-dfs.sh、start-yarn.sh
查看标签页返回是否成功或者直接浏览器调用
curl flink1:8088/cluster
flink修改/bin/flink-conf.yaml配置文件对应参数
scp发送到对应的namenode节点
配置flink环境变量 别忘了source一下
FLINK_HOME=/opt/flink/flink-1.13.6 export PATH=$PATH:$FLINK_HOME/bin
命令运行yarn-session.sh -jm 1024 -tm 4096 -s 4 -nm ExamDIMJob -d -st
发现报错如下：
Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2701) at java.lang.Class.privateGetMethodRecursive(Class.java:3048) at java.lang.Class.getMethod0(Class.java:3018) at java.lang.Class.getMethod(Class.java:1784) at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544) at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 7 more
1、添加HADOOP_CLASSPATH环境变量
2、由于缺少flink hadoop下载jar包：flink-shaded-hadoop-3-uber-3.1.1.7.2.9.0-173-9.0.jar
添加到flink/lib目录下
再运行又报错：
2022-04-21 14:00:02,794 ERROR org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - Error while running the Flink session. java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder; at org.apache.flink.yarn.cli.FlinkYarnSessionCli.(FlinkYarnSessionCli.java:230) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.yarn.cli.FlinkYarnSessionCli. (FlinkYarnSessionCli.java:156) ~[flink-dist_2.11-1.13.6.jar:1.13.6] at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:851) [flink-dist_2.11-1.13.6.jar:1.13.6] ------------------------------------------------------------ The program finished with the following exception: java.lang.NoSuchMethodError: org.apache.commons.cli.Option.builder(Ljava/lang/String;)Lorg/apache/commons/cli/Option$Builder; at org.apache.flink.yarn.cli.FlinkYarnSessionCli. (FlinkYarnSessionCli.java:230) at org.apache.flink.yarn.cli.FlinkYarnSessionCli. (FlinkYarnSessionCli.java:156) at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:851)
下载jar包：commons-cli-1.5.0.jar 添加到flink/lib目录下
再运行出现如下日志表示已成功
2022-04-21 14:07:16,009 INFO org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorFrameworkImpl [] - Default schema
2022-04-21 14:07:16,011 ERROR org.apache.flink.shaded.curator4.org.apache.curator.ConnectionState [] - Authentication failed
2022-04-21 14:07:16,016 INFO org.apache.flink.shaded.curator4.org.apache.curator.framework.state.ConnectionStateManager [] - State change: CONNECTED
2022-04-21 14:07:16,184 INFO org.apache.flink.runtime.leaderretrieval.DefaultLeaderRetrievalService [] - Starting DefaultLeaderRetrievalService with ZookeeperLeaderRetrievalDriver{retrievalPath='/leader/rest_server_lock'}.
JobManager Web Interface: http://VM-2-237-centos:8081
2022-04-21 14:07:16,201 INFO org.apache.flink.yarn.cli.FlinkYarnSessionCli [] - The Flink YARN session cluster has been started in detached mode. In order to stop Flink gracefully, use the following command:
$ echo "stop" | ./bin/yarn-session.sh -id application_1650509744418_0001
If this should not be possible, then you can also kill Flink via YARN's web interface or via:
$ yarn application -kill application_1650509744418_0001
Note that killing Flink might not clean up all job artifacts and temporary files.
登入hadoop的dashboard页面查看
点击跳转flink页面
完工。
