栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

iceberg读写常见报错

iceberg读写常见报错

flink写iceberg报错

场景:flink sink iceberg时,使用equalityFieldColumns指定字段时,报错如下:

java.lang.IllegalArgumentException: Cannot write delete files in a v1 table
	at org.apache.iceberg.ManifestFiles.writeDeleteManifest(ManifestFiles.java:154)
	at org.apache.iceberg.SnapshotProducer.newDeleteManifestWriter(SnapshotProducer.java:365)
	at org.apache.iceberg.MergingSnapshotProducer.newDeleteFilesAsManifest(MergingSnapshotProducer.java:480)
	at org.apache.iceberg.MergingSnapshotProducer.prepareDeleteManifests(MergingSnapshotProducer.java:469)
	at org.apache.iceberg.MergingSnapshotProducer.apply(MergingSnapshotProducer.java:358)
	at org.apache.iceberg.SnapshotProducer.apply(SnapshotProducer.java:163)
	at org.apache.iceberg.SnapshotProducer.lambda$commit$2(SnapshotProducer.java:276)
	at org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
	at org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:213)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:197)
	at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:189)
	at org.apache.iceberg.SnapshotProducer.commit(SnapshotProducer.java:275)
	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitOperation(IcebergFilesCommitter.java:298)
	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitDeltaTxn(IcebergFilesCommitter.java:285)
	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.commitUpToCheckpoint(IcebergFilesCommitter.java:210)
	at org.apache.iceberg.flink.sink.IcebergFilesCommitter.initializeState(IcebergFilesCommitter.java:147)
	at org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.initializeOperatorState(StreamOperatorStateHandler.java:106)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:258)
	at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:290)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
	at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
	at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
	at java.lang.Thread.run(Thread.java:748)

解决办法:
社区版本现在没有把v1打开,是因为v2还有一些功能不是特别完善。如果用户想用v2展开测试的话,需要通过java API把v1升级到v2,调用方式如下:

Table table = …
TableOperations ops = ((baseTable) table).operations();
Tablemetadata meta = ops.current();
ops.commit(meta, meta.upgradeToFormatVersion(2));
转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/674040.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号