在添加tez引擎之后，2.6 GB of 2.1 GB virtual memory used. Killing containe
Application application_1642949186322_0005 failed 2 times due to AM Container for appattempt_1642949186322_0005_000002 exited with exitCode: -103 Failing this attempt.Diagnostics: [2022-01-23 22:54:55.170]Container [pid=17888,containerID=container_1642949186322_0005_02_000001] is running 570644992B beyond the 'VIRTUAL' memory limit. Current usage: 157.9 MB of 1 GB physical memory used; 2.6 GB of 2.1 GB virtual memory used. Killing container. Dump of the process-tree for container_1642949186322_0005_02_000001 : |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE |- 17888 17887 17888 17888 (bash) 0 0 11964416 348 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/yingzi/appcache/application_1642949186322_0005/container_1642949186322_0005_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx1024m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1642949186322_0005/container_1642949186322_0005_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster --session 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1642949186322_0005/container_1642949186322_0005_02_000001/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1642949186322_0005/container_1642949186322_0005_02_000001/stderr |- 17900 17888 17888 17888 (java) 493 35 2813538304 40062 /opt/module/jdk1.8.0_212/bin/java -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/yingzi/appcache/application_1642949186322_0005/container_1642949186322_0005_02_000001/tmp -server -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx1024m -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator -Dlog4j.configuration=tez-container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1642949186322_0005/container_1642949186322_0005_02_000001 -Dtez.root.logger=INFO,CLA -Dsun.nio.ch.bugLevel= org.apache.tez.dag.app.DAGAppMaster --session [2022-01-23 22:54:55.268]Container killed on request. Exit code is 143 [2022-01-23 22:54:55.277]Container exited with a non-zero exit code 143. For more detailed output, check the application tracking page: http://hadoop103:8088/cluster/app/application_1642949186322_0005 Then click on links to logs of each attempt. . Failing the application.
我试着修改map,reduce端参数,如下:
set mapreduce.map.memory.mb=4096; set mapreduce.map.java.opts=-Xmx3276M; set mapreduce.reduce.memory.mb=4096; set mapreduce.reduce.java.opts=-Xmx3276M;
可还是运行失败。另寻它解
解决方法如下：
关闭虚拟内存检查：即在yarn-site.xml文件添加如下配置
yarn.nodemanager.vmem-check-enabled false
重启集群，终于运行出来了！！！！
参考资料：
https://zhuanlan.zhihu.com/p/220197000
https://blog.csdn.net/weixin_46584887/article/details/121900677
