pyflink执行任务问题总结

pyflink执行任务问题总结

• 问题：
• • 一、Timeout expired while fetching topic metadata
  • 二、Deployment took more than 60 seconds
  • 三、SyntaxError: invalid syntax
  • 四、Failed to deserialize JobGraph
  • 五、The cluster does not have the requested resources for the JobManager available!
  • 六、Permission denied: user=root, access=WRITE, inode="/user":hadoop:supergroup:drwxr-xr-x
  • 七、Timeout expired while fetching topic metadata

问题： 一、Timeout expired while fetching topic metadata

yarn模式下报错：
Caused by: org.apache.flink.kafka.shaded.org.apache.kafka.common.errors.TimeoutException: Timeout expired while fetching topic metadata
原因：内存不足
默认不设置的情况下最大为8G，最小为1G，
修改hadoop/etc/hadoop/yarn-site.xml中的value值

    
        yarn.scheduler.minimum-allocation-mb
        1024
        一个container可申请的最小内存
    
    
        yarn.scheduler.maximum-allocation-mb
        8192
        一个container可以申请的最大内存
    
    
        yarn.nodemanager.resource.memory-mb
        8192
        nodemanager可使用内存资源大小
    

二、Deployment took more than 60 seconds

flink-yarn启动yarn-session时报错：
Deployment took more than 60 seconds. Please check if the requested resources are available in the YARN cluster

1、由于flink配置了高可用，所以在启动时需要启动zookeeper，如果不启动zookeeper，高可用则不能使用，会报如上错误。
启动zookeeper再启动flink-yarn
2、有可能是内存不足的问题
有时使用yarn application -kill app
杀死Flink可能不会清除所有作业工件和临时文件
使用：
yarn cache clear
如果cache没有命令就重启yarn

三、SyntaxError: invalid syntax

File "/tmp/pyflink/flink/test.py", line 34
    """
      ^
SyntaxError: invalid syntax

python环境问题
1、修改python环境
2、修改flink/bin/pyflink_shell.sh

PYFlink_PYTHON="${PYFlink_PYTHON:-“python”}"

改成：

PYFlink_PYTHON="${PYFlink_PYTHON:-“python3”}"

四、Failed to deserialize JobGraph

报错：
Failed to deserialize JobGraph
原因：
版本问题可能有多个版本的flink
统一flink版本和flink包

五、The cluster does not have the requested resources for the JobManager available!

Caused by: org.apache.flink.yarn.YarnClusterDescriptor$YarnDeploymentException: The cluster does not have the requested resources for the JobManager available!
Maximum Memory: 2048MB Requested: 3072MB. Please check the 'yarn.scheduler.maximum-allocation-mb' and the 'yarn.nodemanager.resource.memory-mb' configuration values

原因：
flink内存配置大于yarn配置内存
修改flink设置中内存或者修改yarn配置内存

六、Permission denied: user=root, access=WRITE, inode="/user":hadoop:supergroup:drwxr-xr-x

报错：
Permission denied: user=root, access=WRITE, inode="/user":hadoop:supergroup:drwxr-xr-x
原因：
往HDFS写文件是不容许的。权限问题
在hadoop/etc/hadoop/hdfs-site.xml中加入：

    dfs.permissions
    false

七、Timeout expired while fetching topic metadata

kafka链接不成功
查看kafka启动是否正常

持续更新，未完待续。。。