官方配置：Configuration | Apache Flink
1、TM进程过一段时间就停止报错信息：org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Task did not exit gracefully within 180 + seconds.
org.apache.flink.util.FlinkRuntimeException: Task did not exit gracefully within 180 + seconds.
at org.apache.flink.runtime.taskmanager.Task$TaskCancelerWatchDog.run(Task.java:1791) [flink-dist_2.11-1.14.4.jar:1.14.4]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291]
原因：任务取消超时
解决：TM配置文件${FLINK_HOME}/conf/flink-conf.yml
#取消任务取消watchdog
task.cancellation.timeout: 0
参数说明：Timeout in milliseconds after which a task cancellation times out and leads to a fatal TaskManager error. A value of 0 deactivates the watch dog. Notice that a task cancellation is different from both a task failure and a clean shutdown. Task cancellation timeout only applies to task cancellation and does not apply to task closing/clean-up caused by a task failure or a clean shutdown.
2、web端上传的jar包，在独立集群重启后全部丢失原因：文件默认保存在/tmp目录，会被清除
解决：JM配置文件${FLINK_HOME}/conf/flink-conf.yml
web.upload.dir: /usr/local/flink/upload
web.tmpdir: /usr/local/flink/tmpdir
原因：pid文件默认保存在/tmp目录，会被清除导致脚本找不到pid结束进程
解决：JM配置文件${FLINK_HOME}/conf/flink-conf.yml
env.pid.dir: /usr/local/flink/piddir
4、zookeeper存储value太长，zookeeper集群down掉导致TM全部down掉，zookeeper报错信息：Unexpected exception causing shutdown while sock still open
java.io.IOException: Unreasonable length = 1970218037
at org.apache.jute.BinaryInputArchive.readBuffer(BinaryInputArchive.java:95)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:85)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:249)
Zookeeper server went down in HA cluster. Please replay if there is any work around.
You can attempt to increase your jute.maxbuffer Java System Property on the ZK servers to a value higher than 2-3 GB (in bytes) to overcome this. It appears a very large record was somehow placed into your ZK by an application, which appears to have then caused this issue.
解决方法：配置zookeeper的jute.maxbuffer参数到合适的长度
5、java.lang.OutOfMemoryError: Metaspace. 详细报错信息：java.lang.OutOfMemoryError: Metaspace. The metaspace out-of-memory error has occurred. This can mean two things: either the job requires a larger size of JVM metaspace to load classes or there is a class loading leak. In the first case 'taskmanager.memory.jvm-metaspace.size' configuration option should be increased. If the error persists (usually in cluster after several job (re-)submissions) then there is probably a class loading leak in user code or some of its dependencies which has to be investigated and fixed. The task executor has to be shutdown...
at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0_291]
at java.lang.ClassLoader.defineClass(ClassLoader.java:756) ~[?:1.8.0_291]
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[?:1.8.0_291]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[?:1.8.0_291]
at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[?:1.8.0_291]
at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[?:1.8.0_291]
at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[?:1.8.0_291]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_291]
at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[?:1.8.0_291]
原因：没有找到具体原因，持续观察，网上搜索有两种说法：代码阻塞、背压
短期解决方案：TM配置文件${FLINK_HOME}/conf/flink-conf.yml
修改配置(默认256m)taskmanager.memory.jvm-metaspace.size: 512m
