【RDD】Pair RDD Functions

Spark用几个函数定义了PairRDDFunctions类来操作Pair RDD或RDD key-value pair。当需要应用像hash partition, set operations, joins等transformations时，Pair RDD就派上了用场。

val rdd = spark.sparkContext.parallelize(
      List("Germany India USA","USA India Russia","India Brazil Canada China")
    )
 val wordsRdd = rdd.flatMap(_.split(" "))
 val pairRDD = wordsRdd.map(f=>(f,1))
 pairRDD.foreach(println)

这段代码通过在RDD中的每个元素上按空格分割来创建pair RDD，并将其拉平，在RDD中的每个元素上形成一个单词字符串，最后为每个单词分配一个整数“1”，运行结果如下：

(Germany,1)
(India,1)
(USA,1)
(USA,1)
(India,1)
(Russia,1)
(India,1)
(Brazil,1)
(Canada,1)
(China,1)

distinct – Returns distinct keys.

pairRDD.distinct().foreach(println)

//Prints below output
(Germany,1)
(India,1)
(Brazil,1)
(China,1)
(USA,1)
(Canada,1)
(Russia,1)

sortByKey – Transformation returns an RDD after sorting by key

println("Sort by Key ==>")
val sortRDD = pairRDD.sortByKey()
sortRDD.foreach(println)

//输出如下：
Sort by Key ==>
(Brazil,1)
(Canada,1)
(China,1)
(Germany,1)
(India,1)
(India,1)
(India,1)
(Russia,1)
(USA,1)
(USA,1)

reduceByKey – Transformation returns an RDD after adding value for each key.

结果RDD包含唯一key

println("Reduce by Key ==>")
val wordCount = pairRDD.reduceByKey((a,b)=>a+b)
wordCount.foreach(println)

以上通过为value求和聚合了keys，输出如下：

Reduce by Key ==>
(Brazil,1)
(Canada,1)
(China,1)
(USA,2)
(Germany,1)
(Russia,1)
(India,3)

aggregateByKey – Transformation same as reduceByKey

这部分没懂，下次补充。

参考文献

Spark Pair RDD Functions - Spark by {Examples}

Scala中下划线“_”的用法小结 - 简书

细数Scala下划线“_”的用法_菜刚的博客-CSDN博客

Scala中_(下划线)的常见用法 - 简书


What are all the uses of an underscore in Scala? - Stack Overflow