hive bug1:
Exception in thread "org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor@34a2d6e0" java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.ArrayList.
at sun.management.ManagementFactoryHelper.getGarbageCollectorMXBeans(ManagementFactoryHelper.java:131)
at java.lang.management.ManagementFactory.getGarbageCollectorMXBeans(ManagementFactory.java:416)
at org.apache.hadoop.hive.common.JvmPauseMonitor.getGcTimes(JvmPauseMonitor.java:141)
at org.apache.hadoop.hive.common.JvmPauseMonitor.access$400(JvmPauseMonitor.java:45)
at org.apache.hadoop.hive.common.JvmPauseMonitor$Monitor.run(JvmPauseMonitor.java:186)
at java.lang.Thread.run(Thread.java:748)
Exception in thread "HiveServer2-Handler-Pool: Thread-213" java.lang.OutOfMemoryError: GC overhead limit exceeded
修改hive-env.sh调整heap大小
hive bug2:
WARN org.apache.flink.runtime.taskmanager.Task
[] - PartitionCommitter -> Sink: end (1/1)#205
(349f5d83bf18f490002b6817c89f0d5a) switched from INITIALIZING to FAILED
with failure cause:
org.apache.flink.streaming.runtime.tasks.StreamTaskException:
Cannot load user class: org.apache.flink.connectors.hive.HadoopFileSystemFactory
找到相应版本的jar包、connector,这个也是最长遇到的问题
hive bug3:
ERROR [a287cd2c-87cf-4572-abe4-cfaa15ec44b1 main] exec.Task (SessionState.java:printError(1250)) - Ended Job = job_1635418137267_0003 with errors
Error during job, obtaining debugging information...
2021-11-01 02:52:13,271 ERROR [Thread-53] exec.Task (SessionState.java:printError(1250)) - Error during job, obtaining debugging information...
2021-11-01 02:52:13,276 INFO [a287cd2c-87cf-4572-abe4-cfaa15ec44b1 main] impl.YarnClientImpl (YarnClientImpl.java:killApplication(497)) - Killed application application_1635418137267_0003
2021-11-01 02:52:13,277 INFO [a287cd2c-87cf-4572-abe4-cfaa15ec44b1 main] reexec.ReOptimizePlugin (ReOptimizePlugin.java:run(70)) - ReOptimization: retryPossible: false
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
2021-11-01 02:52:13,277 ERROR [a287cd2c-87cf-4572-abe4-cfaa15ec44b1 main] ql.Driver (SessionState.java:printError(1250)) - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
测试方法:修改etc/hadoop/mapred-site.xml配置后，重启hdfs生效。
etc/hadoop/hdfs-site.xml
调整hadoop配置文件yarn-site.xml中值：
hive bug4:
执行部分hive sql脚本后hive直接挂掉，报错栈溢出、内存错误等。
报错：解决第一个bug3后消失，无法复现
hive bug5:
flink跟hive版本不兼容，降级hive失败后升级flink； 注意使用最新版本的hive跟flink
比如下面错误：
Before 3.1.2 of hive version, getQueryCurrentTimestamp return Timestamp. But when hive version is 3.1.2，getCurrentTSMethod invoke return Instant. So the code `(Timestamp)getCurrentTSMethod.invoke(sessionState)` will result the ClassCastException. It should be compatibility with this situation.
when I use hive dialect to create hive table, it will tirgger this error. The error is below:
Exception in thread "main" java.lang.ClassCastException: java.time.Instant cannot be cast to java.sql.TimestampException in thread "main" java.lang.ClassCastException: java.time.Instant cannot be cast to java.sql.Timestamp at org.apache.flink.table.planner.delegation.hive.HiveParser.setCurrentTimestamp(HiveParser.java:365) at org.apache.flink.table.planner.delegation.hive.HiveParser.startSessionState(HiveParser.java:350) at org.apache.flink.table.planner.delegation.hive.HiveParser.parse(HiveParser.java:218) at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:722)
