在 Spark 2.0 之前，Spark 的主要编程接口是弹性分布式数据集（RDD）。在 Spark 2.0 之后，RDD 被 Dataset 取代，它像 RDD 一样是强类型的，但在底层进行了更丰富的优化。仍然支持 RDD 接口，您可以在RDD 编程指南中获得更详细的参考。但是，我们强烈建议您改用 Dataset，它的性能比 RDD 更好。请参阅SQL 编程指南以获取有关数据集的更多信息。

也就是Spark1，对应RDD; Spark2, 对应 Dataset

Interactive Analysis with the Spark Shell

pyspark

>>> textFile = spark.read.text("README.md")
>>> textFile.count()  # Number of rows in this DataFrame
126

>>> textFile.first()  # First row in this DataFrame
Row(value=u'# Apache Spark')

Caching

>>> linesWithSpark.cache()

>>> linesWithSpark.count()
15

>>> linesWithSpark.count()
15

Self-Contained Applications 独立的应用程序

from pyspark.sql import SparkSession

logFile = "YOUR_SPARK_HOME/README.md"  # Should be some file on your system
spark = SparkSession.builder.appName("SimpleApp").getOrCreate()
logData = spark.read.text(logFile).cache()

numAs = logData.filter(logData.value.contains('a')).count()
numBs = logData.filter(logData.value.contains('b')).count()

print("Lines with a: %i, lines with b: %i" % (numAs, numBs))

spark.stop()

# Use spark-submit to run your application
$ YOUR_SPARK_HOME/bin/spark-submit 
  --master local[4] 
  SimpleApp.py
...
Lines with a: 46, Lines with b: 23

# Use the Python interpreter to run your application
$ python SimpleApp.py
...
Lines with a: 46, Lines with b: 23