栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

flink on yarn ——调节堆内存

flink on yarn ——调节堆内存

报错:OutOfMemoryError: GC overhead limit exceeded

java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.Arrays.copyOfRange(Arrays.java:3664)
	at java.lang.String.(String.java:207)
	at java.lang.StringBuilder.toString(StringBuilder.java:407)
	at org.apache.hadoop.conf.Configuration$Parser.handleEndElement(Configuration.java:3246)
	at org.apache.hadoop.conf.Configuration$Parser.parseNext(Configuration.java:3331)
	at org.apache.hadoop.conf.Configuration$Parser.parse(Configuration.java:3114)
	at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:3007)
	at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2968)
	at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2848)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:1200)
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:211)
	at com.gtcom.governance.until.HdfsUtils.getFileSystem(HdfsUtils.java:65)
	at com.gtcom.governance.source.zip.ScanZipExtractData.invoke(ScanZipExtractData.java:224)
	at com.gtcom.governance.source.zip.ScanZipExtractData$1.lambda$visitFile$1(ScanZipExtractData.java:159)
	at com.gtcom.governance.source.zip.ScanZipExtractData$1$$Lambda$637/1084422546.accept(Unknown Source)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
	at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
	at com.gtcom.governance.source.zip.ScanZipExtractData$1.visitFile(ScanZipExtractData.java:157)
	at com.gtcom.governance.source.zip.ScanZipExtractData$1.visitFile(ScanZipExtractData.java:88)
	at java.nio.file.Files.walkFileTree(Files.java:2670)
	at java.nio.file.Files.walkFileTree(Files.java:2742)
	at com.gtcom.governance.source.zip.ScanZipExtractData.scan(ScanZipExtractData.java:88)
	at com.gtcom.governance.source.zip.ScanZipExtractData.run(ScanZipExtractData.java:78)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
	at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
2022-03-21 16:41:38.961 [Source: Custom Source (1/1)] INFO  com.gtcom.governance.source.zip.ScanZipExtractData - cancelling.......

增加-ytm :

$Flink_HOME/bin/flink run -d -m yarn-cluster -yjm 2048m -ytm 15360m -ynm yarn_monitorZip_news_second -yD env.java.opts="-Duser.timezone=GMT+8"  -ys 1 $Flink_HOME/jobs/gtcom-governance-service-jk-2.3.0.jar --active pro --driver MonitorZip --topicName MinitorZip_wait_second --sinkPara 3 --sourcePara 1  --opertatorPara 3 --processorType MonitorZip  --scanPath /home/data/packageAll/secondPackageZiptmp/   --targetPath /home/data/packageAll/secondTargetPath/ --passWd 12345 --backupPath /home/data/packageAll/secondBackupPath/

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/780925.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号