栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

spark udf 出现java.lang.UnsupportedOperationException: Schema for type AnyVal is not supported

spark udf 出现java.lang.UnsupportedOperationException: Schema for type AnyVal is not supported

Exception in thread "main" java.lang.UnsupportedOperationException: Schema for type AnyVal is not supported
	at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:790)
	at org.apache.spark.sql.catalyst.ScalaReflection$$anonfun$schemaFor$1.apply(ScalaReflection.scala:724)
	at scala.reflect.internal.tpe.TypeConstraints$UndoLog.undo(TypeConstraints.scala:56)
	at org.apache.spark.sql.catalyst.ScalaReflection$class.cleanUpReflectionObjects(ScalaReflection.scala:907)
	at org.apache.spark.sql.catalyst.ScalaReflection$.cleanUpReflectionObjects(ScalaReflection.scala:45)
	at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:723)
	at org.apache.spark.sql.catalyst.ScalaReflection$.schemaFor(ScalaReflection.scala:720)
	at org.apache.spark.sql.UDFRegistration.register(UDFRegistration.scala:234)
	at myspark.warehouse.DataParsing$.main(DataParsing.scala:62)
	at myspark.warehouse.DataParsing.main(DataParsing.scala)

原因:

之前定义了一个函数,在spark sql中使用,该函数的返回值类型是AnyVal,导致Dataframe无法解析,然后抛出了这个错误

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/676380.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号