- 环境信息
- 参考资料
- 配置步骤
- 测试
- 建议
Ambari 2.6.1+HDP 2.6.5(hive 单独升级到了2.3.2)+spark 2.0
参考资料https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark:+Getting+Started#HiveonSpark:GettingStarted-ConfiguringHive.
https://stackoverflow.com/questions/55422935/hive-on-spark-error-java-lang-nosuchfielderror-spark-rpc-server-address
https://www.cnblogs.com/lyy-blog/p/9598433.html
配置步骤- 将spark 依赖包 软链接到hive目录
ln -s /usr/hdp/2.6.5.0-292/spark2/jars/scala-library-2.11.8.jar /usr/hdp/2.6.5.0-292/hive/lib ln -s /usr/hdp/2.6.5.0-292/spark2/jars/spark-core_2.11-2.3.0.2.6.5.0-292.jar /usr/hdp/2.6.5.0-292/hive/lib ln -s /usr/hdp/2.6.5.0-292/spark2/jars/spark-network-common_2.11-2.3.0.2.6.5.0-292.jar /usr/hdp/2.6.5.0-292/hive/lib ln -s /usr/hdp/2.6.5.0-292/spark2/jars/spark-unsafe_2.11-2.3.0.2.6.5.0-292.jar /usr/hdp/2.6.5.0-292/hive/lib
- 在custom hive-site.xml配置里添加如下配置
spark.yarn.jars hdfs:///spark-jars/* spark.master yarn-cluster spark.sql.hive.metastore.version 2.3.2 spark.sql.hive.metastore.jars /usr/hdp/2.6.5.0-292/hive/lib/*:/usr/hdp/2.6.5.0-292/hadoop/client/*
- 将spark依赖的jar包上传到hdfs，避免加载本地jar包
hdfs dfs -mkdir /spark-jars hdfs dfs -put /usr/hdp/2.6.5.0-292/spark2/jars/*.jar /spark-jars/ hdfs dfs -rm /spark-jars/*hive*1.2.1*测试
beeline -u "jdbc:hive2://ip.ip.ip.ip:10000/" set hive.execution.engine=spark; select count(1) from hive_on_spark_test;建议
hive on spark 目前的兼容性十分有限，官方给出了以下兼容性表，后续演进也尚不明确，建议不要采用这种方案，能直接用sparksql 就用sparksql吧
