[hive-3.1.3]Beeline 用zookeeper 发现模式连接 HiveServer2 连接 leader 异常

使用 beeline 连接 hivesever 时，抛出异常如下：

[root@vhost-120-28 hive]# beeline -u "jdbc:hive2://vhost-120-26:2181,vhost-120-27:2181,vhost-120-28:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" --verbose=true
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/wdp/1.0/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/wdp/1.0/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
!connect jdbc:hive2://vhost-120-26:2181,vhost-120-27:2181,vhost-120-28:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 '' [passwd stripped] 
Connecting to jdbc:hive2://vhost-120-26:2181,vhost-120-27:2181,vhost-120-28:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
Error: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper (state=,code=0)
java.sql.SQLException: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper
        at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:170)
        at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:208)
        at org.apache.hive.beeline.DatabaseConnection.connect(DatabaseConnection.java:145)
        at org.apache.hive.beeline.DatabaseConnection.getConnection(DatabaseConnection.java:209)
        at org.apache.hive.beeline.Commands.connect(Commands.java:1641)
        at org.apache.hive.beeline.Commands.connect(Commands.java:1536)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hive.beeline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:56)
        at org.apache.hive.beeline.BeeLine.execCommandWithPrefix(BeeLine.java:1384)
        at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:1423)
        at org.apache.hive.beeline.BeeLine.connectUsingArgs(BeeLine.java:900)
        at org.apache.hive.beeline.BeeLine.initArgs(BeeLine.java:795)
        at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:1048)
        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:538)
        at org.apache.hive.beeline.BeeLine.main(BeeLine.java:520)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper
        at org.apache.hive.jdbc.ZooKeeperHiveClientHelper.configureConnParams(ZooKeeperHiveClientHelper.java:147)
        at org.apache.hive.jdbc.Utils.configureConnParamsFromZooKeeper(Utils.java:511)
        at org.apache.hive.jdbc.Utils.parseURL(Utils.java:334)
        at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:168)
        ... 25 more

HiveServer2.java startPrivilegeSynchonizer 里注册的 znode 和 hiveserver2 的 znode 在同一个目录下。
如系统有 3 台 hiveserver2，那么会出现 4 个 znode， beeline 连接的时候，会从里面随机选择一台，25% 的概率选择 leader。

社区 jira:HIVE-19825 ，没有合入。

社区 jira:HIVE-25096 ，没有合入。由于 HiveServer 的 znode 以 “serverUri=” 开头，所以仅保留以 “serverUri=” 开头的 znode，再从中随机选择一个。

不注册PrivilegeSynchonizer，就不会生成 leader 的 znode。

public void startPrivilegeSynchonizer(HiveConf hiveConf) throws Exception {

    PolicyProviderContainer policyContainer = new PolicyProviderContainer();
    HiveAuthorizer authorizer = SessionState.get().getAuthorizerV2();
    if (authorizer.getHivePolicyProvider() != null) {
      policyContainer.addAuthorizer(authorizer);
    }
    if (hiveConf.get(MetastoreConf.ConfVars.PRE_EVENT_LISTENERS.getVarname()) != null &&
        hiveConf.get(MetastoreConf.ConfVars.PRE_EVENT_LISTENERS.getVarname()).contains(
        "org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener") &&
        hiveConf.get(MetastoreConf.ConfVars.HIVE_AUTHORIZATION_MANAGER.getVarname())!= null) {
      List providers = HiveUtils.getMetaStoreAuthorizeProviderManagers(
          hiveConf, HiveConf.ConfVars.HIVE_METASTORE_AUTHORIZATION_MANAGER, SessionState.get().getAuthenticator());
      for (HiveMetastoreAuthorizationProvider provider : providers) {
        if (provider.getHivePolicyProvider() != null) {
          policyContainer.addAuthorizationProvider(provider);
        }
      }
    }

    if (policyContainer.size() > 0) {
      zKClientForPrivSync = startZookeeperClient(hiveConf);
      String rootNamespace = hiveConf.getVar(HiveConf.ConfVars.HIVE_SERVER2_ZOOKEEPER_NAMESPACE);
      String path = ZooKeeperHiveHelper.ZOOKEEPER_PATH_SEPARATOR + rootNamespace
          + ZooKeeperHiveHelper.ZOOKEEPER_PATH_SEPARATOR + "leader";
      LeaderLatch privilegeSynchonizerLatch = new LeaderLatch(zKClientForPrivSync, path);
      privilegeSynchonizerLatch.start();
      Thread privilegeSynchonizerThread = new Thread(
          new PrivilegeSynchonizer(privilegeSynchonizerLatch, policyContainer, hiveConf), "PrivilegeSynchonizer");
      privilegeSynchonizerThread.start();
    } else {
      LOG.warn(
          "No policy provider found, skip creating PrivilegeSynchonizer");
    }
  }

不注册的条件是 policyContainer.size() == 0。policyContainer 的 size 方法如下：

public int size() {
    return authorizers.size() + authorizationProviders.size();
  }

代表需要authorizers 和 authorizationProviders 都没有。

添加 authorizers 的条件：

if (authorizer.getHivePolicyProvider() != null) {
    policyContainer.addAuthorizer(authorizer);
}

添加 AuthorizationProvider 的条件：
设置 PRE_EVENT_LISTENERS 包含 “org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListene”。MetastoreConf 中的对应参数如下：

PRE_EVENT_LISTENERS("metastore.pre.event.listeners", "hive.metastore.pre.event.listeners", "",
        "List of comma separated listeners for metastore events."),

并且 provider 满足一下条件：

if (provider.getHivePolicyProvider() != null) {
    policyContainer.addAuthorizationProvider(provider);
}

对于 provider ，当 HiveConf.ConfVars.HIVE_METASTORE_AUTHORIZATION_MANAGER 的值是 org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider ，就会创建 leader 节点。当 HiveConf.ConfVars.HIVE_METASTORE_AUTHORIZATION_MANAGER 的值是 org.apache.hadoop.hive.ql.security.authorization.DefaultHiveMetastoreAuthorizationProvider ，就不会创建 leader 节点。

web-site.xml 中配置如下，则会出现 leader。当采用 Ranger 或者类似组件时，必须配置 metastore.pre.event.listeners。

  metastore.pre.event.listeners
 org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener


  hive.security.metastore.authorization.manager
  org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider

没有配置 metastore.pre.event.listeners，仅配置 hive.security.metastore.authorization.manager 不会有 leader。

  hive.security.metastore.authorization.manager
  org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider

以下配置也没有 leader，因为 DefaultHiveMetastoreAuthorizationProvider 的 getHivePolicyProvider() 返回 null。

  metastore.pre.event.listeners
 org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener


  hive.security.metastore.authorization.manager
  org.apache.hadoop.hive.ql.security.authorization.DefaultHiveMetastoreAuthorizationProvider