栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to toAttribute on unresolve

org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to toAttribute on unresolve

执行形如下列的spark sql:

select xxx from TABLENAME WHERe x=1 having ConDITION = 1

报错:

org.apache.spark.sql.catalyst.analysis.UnresolvedException: Invalid call to toAttribute on unresolved object, tree: ArrayBuffer(a).*
	at org.apache.spark.sql.catalyst.analysis.Star.toAttribute(unresolved.scala:245)
	at org.apache.spark.sql.catalyst.plans.logical.Project$$anonfun$output$1.apply(basicLogicalOperators.scala:52)
	at org.apache.spark.sql.catalyst.plans.logical.Project$$anonfun$output$1.apply(basicLogicalOperators.scala:52)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.immutable.List.map(List.scala:296)
	at org.apache.spark.sql.catalyst.plans.logical.Project.output(basicLogicalOperators.scala:52)
	at org.apache.spark.sql.hive.HiveAnalysis$$anonfun$apply$3.applyOrElse(HiveStrategies.scala:160)
	at org.apache.spark.sql.hive.HiveAnalysis$$anonfun$apply$3.applyOrElse(HiveStrategies.scala:148)

原因:
https://spark.apache.org/docs/2.4.0/sql-migration-guide-upgrade.html#upgrading-from-spark-sql-23-to-24

spark sql在2.3和2.4的处理逻辑不一样,
在2.3中如果没有GROUP BY,则HAVINg会当成WHERe进行处理
但2.4中就会报上述错误。


可参考的解决方案:
1.改sql语句,加上group by
2. 启动的时候设置spark.sql.legacy.parser.havingWithoutGroupByAsWhere 为true ,即 ./bin/spark-sql --conf spark.sql.legacy.parser.havingWithoutGroupByAsWhere =true

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/433724.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号