Flink集成Iceberg时 用到HiveCatalog 依赖于Hive metaStore
Flink任务打包上传到集群运行时 报错类找不到
java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
具体报错信息如下:
org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: Failed to connect to Hive metastore at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:114) at org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:812) at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:246) at org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054) at org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836) at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132) Caused by: org.apache.iceberg.hive.RuntimemetaException: Failed to connect to Hive metastore at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:72) at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:35) at org.apache.iceberg.ClientPoolImpl.get(ClientPoolImpl.java:125) at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:56) at org.apache.iceberg.ClientPoolImpl.run(ClientPoolImpl.java:51) at org.apache.iceberg.hive.CachedClientPool.run(CachedClientPool.java:76) at org.apache.iceberg.hive.HiveCatalog.createNamespace(HiveCatalog.java:258) at org.apache.iceberg.flink.FlinkCatalog.createDatabase(FlinkCatalog.java:203) at org.apache.iceberg.flink.FlinkCatalog.open(FlinkCatalog.java:120) at org.apache.flink.table.catalog.CatalogManager.registerCatalog(CatalogManager.java:195) at org.apache.flink.table.api.internal.TableEnvironmentImpl.createCatalog(TableEnvironmentImpl.java:1297) at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1122) at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:742) at importData$.createIcebergCatalog(importData.scala:623) at importData$.main(importData.scala:34) at importData.main(importData.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ... 11 more Caused by: java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.hadoop.hive.metastore.utils.JavaUtils.getClass(JavaUtils.java:52) at org.apache.hadoop.hive.metastore.RetryingmetaStoreClient.getProxy(RetryingmetaStoreClient.java:146) at org.apache.hadoop.hive.metastore.RetryingmetaStoreClient.getProxy(RetryingmetaStoreClient.java:119) at org.apache.hadoop.hive.metastore.RetryingmetaStoreClient.getProxy(RetryingmetaStoreClient.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.iceberg.common.DynMethods$UnboundMethod.invokeChecked(DynMethods.java:65) at org.apache.iceberg.common.DynMethods$UnboundMethod.invoke(DynMethods.java:77) at org.apache.iceberg.common.DynMethods$StaticMethod.invoke(DynMethods.java:196) at org.apache.iceberg.hive.HiveClientPool.newClient(HiveClientPool.java:55) ... 31 more Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357)原因分析:
hive-exec的包有的版本不带libfb303.jar 需要手动下载添加到flink的lib目录下
下载地址:https://repo1.maven.org/maven2/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar



