- 1、查询数据
1、Spark-sql查询正常
2、Hsql查询异常
Failed with exception java.io.IOException:java.lang.RuntimeException: ORC split generation failed with exception: java.lang.ArrayIndexOutOfBoundsException: 6
- spark-shell查询异常
3.1、查询报错
Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting
‘spark.debug.maxToStringFields’ in SparkEnv.conf.
3.2、无法获取到数据
二、解决异常
1、Hsql查询异常解决
参考链接：
hive2.1.1读取spark写入的orc:ORC split generation failed with exception:ArrayIndexOutOfBoundsException: 6_lixiaoksi的博客-CSDN博客https://blog.csdn.net/lixiaoksi/article/details/106855509
hive-exec-2.1.1-cdh6.1.1.jar_-Hive文档类资源-CSDN下载修改后的hive-exec-2.1.1-cdh6.1.1.jar包更多下载资源、学习资料请访问CSDN下载频道.https://download.csdn.net/download/lixiaoksi/16333490
替换CDH集群的jar包(每个节点都需要替换，注意备份原文件)，重启hive客户端：
/opt/cloudera/parcels/CDH/lib/hive/lib/hive-exec-2.1.1-cdh6.2.1.jar
/opt/cloudera/parcels/CDH/jars/hive-exec-2.1.1-cdh6.2.1.jar
/opt/cloudera/parcels/CDH/lib/spark/hive/hive-exec-2.1.1-cdh6.2.1.jar
- spark-shell异常解决
参考链接：
Spark: "Truncated the string representation of a plan since it was too large." Warning when using manually created aggregation expression - Stack Overflowhttps://stackoverflow.com/questions/43759896/spark-truncated-the-string-representation-of-a-plan-since-it-was-too-large-w
[SPARK-15705] Spark won't read ORC schema from metastore for partitioned tables - ASF JIRAhttps://issues.apache.org/jira/browse/SPARK-15705
spark-defaults.conf增加配置：
spark.debug.maxToStringFields=200
spark.sql.hive.convertmetastoreOrc=false
