图数据源
spark2.4.8
介绍怎样在ML中使用数据源来加载数据. 除此之外一些常用的数据源像Parquet, CSV, JSON 和JDBC,提供了一些专用的数据源.
图数据源图数据源被用来加载目录中图文件,它可以加载被压缩的图片(jpeg,png,等)为原始图片,通过Java 类库中ImageIO. 加载的Dataframe有一个StructType列:“image”,包含图片数据存储图片模式. 图片列模式:
origin: StringType(代表图文件路径)height: IntegerType (图片高度)width: IntegerType (图片宽度)nChannels: IntegerType (number of image channels)mode: IntegerType (OpenCV-compatible type)data: BinaryType (Image bytes in OpenCV-compatible order: row-wise BGR in most cases)
为加载图数据ImageDataSource 实现Spark SQL数据源 API 来加载图片数据.
scala> val df = spark.read.format("image").option("dropInvalid", true).load("data/mllib/images/origin/kittens")
df: org.apache.spark.sql.Dataframe = [image: struct]
scala> df.select("image.origin", "image.width", "image.height").show(truncate=false)
+-----------------------------------------------------------------------+-----+------+
|origin |width|height|
+-----------------------------------------------------------------------+-----+------+
|file:///spark/data/mllib/images/origin/kittens/54893.jpg |300 |311 |
|file:///spark/data/mllib/images/origin/kittens/DP802813.jpg |199 |313 |
|file:///spark/data/mllib/images/origin/kittens/29.5.a_b_EGDP022204.jpg |300 |200 |
|file:///spark/data/mllib/images/origin/kittens/DP153539.jpg |300 |296 |
+-----------------------------------------------------------------------+-----+------+
https://spark.apache.org/docs/2.4.8/ml-datasource
