yarn logs -show_application_log_info --applicationId application_1639366579669_896581
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS. Application State: Completed. Container: container_e50_1639366579669_896581_01_000005 on 192-168-16-13.yyipa.com_8041 Container: container_e50_1639366579669_896581_01_000006 on 192-168-16-13.yyipa.com_8041 Container: container_e50_1639366579669_896581_01_000007 on 192-168-16-13.yyipa.com_8041 Container: container_e50_1639366579669_896581_01_000002 on 192-168-16-13.yyipa.com_8041 Container: container_e50_1639366579669_896581_01_000003 on 192-168-16-13.yyipa.com_8041 Container: container_e50_1639366579669_896581_01_000004 on 192-168-16-13.yyipa.com_8041 Container: container_e50_1639366579669_896581_01_000001 on 192-168-16-43.yyipa.com_8041获取每个container上的日志信息
yarn logs -containerId container_e50_1639366579669_896581_01_000005 --applicationId application_1639366579669_896581
WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS.
End of LogType:prelaunch.err
******************************************************************************
Container: container_e50_1639366579669_896581_01_000007 on 192-168-16-13.yyipa.com_8041
LogAggregationType: AGGREGATED
=======================================================================================
LogType:prelaunch.out
LogLastModifiedTime:Sun Mar 20 03:08:56 +0800 2022
LogLength:70
LogContents:
Setting up env variables
Setting up job resources
Launching container
End of LogType:prelaunch.out
******************************************************************************
Container: container_e50_1639366579669_896581_01_000007 on 192-168-16-13.yyipa.com_8041
LogAggregationType: AGGREGATED
=======================================================================================
LogType:taskmanager.err
LogLastModifiedTime:Sun Mar 20 03:08:56 +0800 2022
LogLength:557
LogContents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/yarn/nm/usercache/bduser/appcache/application_1639366579669_896581/filecache/11/log4j-slf4j-impl-2.12.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
End of LogType:taskmanager.err
********************************************************************************
Container: container_e50_1639366579669_896581_01_000007 on 192-168-16-13.yyipa.com_8041
LogAggregationType: AGGREGATED
=======================================================================================
LogType:taskmanager.out
LogLastModifiedTime:Sun Mar 20 03:08:56 +0800 2022
LogLength:12237
LogContents:
12:58:24.502 [main-EventThread] ERROR org.apache.flink.shaded.curator4.org.apache.curator.ConnectionState - Authentication failed
03:08:39.648 [Legacy Source Thread - Source: Custom Source -> Flat Map -> Timestamps/Watermarks -> Process -> Sink: Unnamed (4/6)#0] ERROR com.youyuan.flink.app.YouyueDataAllocation - Occur Exception: java.net.SocketTimeoutException: Read timed out data: {"user_id":1060491087,"event_ts":1647716917585,"heart_time":1647716914132,"sex":1}
03:08:55.508 [Curator-framework-0] ERROR org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl - Background exception was not retry-able or retry gave up
org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = Nonode for /flink/application_1639366579669_896581/leader
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:114) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:308) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CreateBuilderImpl$9.performBackgroundOperation(CreateBuilderImpl.java:801) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.OperationAndData.callPerformBackgroundOperation(OperationAndData.java:84) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.performBackgroundOperation(CuratorframeworkImpl.java:965) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.backgroundOperationsLoop(CuratorframeworkImpl.java:943) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.access$300(CuratorframeworkImpl.java:66) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl$4.call(CuratorframeworkImpl.java:346) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
03:08:55.627 [Curator-framework-0] ERROR org.apache.flink.runtime.taskexecutor.TaskExecutor - Fatal error occurred in TaskExecutor akka.tcp://flink@192-168-16-13.yyipa.com:40673/user/rpc/taskmanager_0.
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Unhandled error in ZooKeeperLeaderRetrievalDriver:Background exception was not retry-able or retry gave up
at org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalDriver.unhandledError(ZooKeeperLeaderRetrievalDriver.java:184) [youyue-data-allocation-1.0.jar:?]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl$6.apply(CuratorframeworkImpl.java:713) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl$6.apply(CuratorframeworkImpl.java:709) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.logError(CuratorframeworkImpl.java:708) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.handleBackgroundOperationException(CuratorframeworkImpl.java:924) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.performBackgroundOperation(CuratorframeworkImpl.java:1001) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.backgroundOperationsLoop(CuratorframeworkImpl.java:943) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.access$300(CuratorframeworkImpl.java:66) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl$4.call(CuratorframeworkImpl.java:346) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = Nonode for /flink/application_1639366579669_896581/leader
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:114) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:308) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CreateBuilderImpl$9.performBackgroundOperation(CreateBuilderImpl.java:801) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.OperationAndData.callPerformBackgroundOperation(OperationAndData.java:84) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.performBackgroundOperation(CuratorframeworkImpl.java:965) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
... 9 more
03:08:55.629 [Curator-framework-0] ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Fatal error occurred while executing the TaskManager. Shutting it down...
org.apache.flink.runtime.leaderretrieval.LeaderRetrievalException: Unhandled error in ZooKeeperLeaderRetrievalDriver:Background exception was not retry-able or retry gave up
at org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalDriver.unhandledError(ZooKeeperLeaderRetrievalDriver.java:184) [youyue-data-allocation-1.0.jar:?]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl$6.apply(CuratorframeworkImpl.java:713) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl$6.apply(CuratorframeworkImpl.java:709) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer$1.run(ListenerContainer.java:100) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.shaded.com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.listen.ListenerContainer.forEach(ListenerContainer.java:92) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.logError(CuratorframeworkImpl.java:708) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.handleBackgroundOperationException(CuratorframeworkImpl.java:924) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.performBackgroundOperation(CuratorframeworkImpl.java:1001) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.backgroundOperationsLoop(CuratorframeworkImpl.java:943) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.access$300(CuratorframeworkImpl.java:66) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl$4.call(CuratorframeworkImpl.java:346) [flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = Nonode for /flink/application_1639366579669_896581/leader
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:114) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:792) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.utils.ZKPaths.mkdirs(ZKPaths.java:308) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CreateBuilderImpl$9.performBackgroundOperation(CreateBuilderImpl.java:801) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.OperationAndData.callPerformBackgroundOperation(OperationAndData.java:84) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
at org.apache.flink.shaded.curator4.org.apache.curator.framework.imps.CuratorframeworkImpl.performBackgroundOperation(CuratorframeworkImpl.java:965) ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-12.0]
... 9 more
End of LogType:taskmanager.out
********************************************************************************
