Spark RDD : fault-tolerant collection of elements that can be operated on in parallel.
two types:
- parallelized collections : take an existing Scala collection and run functions on it in parallel
- Hadoop datasets : run functions on each record of a file in Hadoop distributed file system or any other storage system supported by Hadoop
scala> val data = Array(1,2,3,4,5)
data: Array[Int] = Array(1, 2, 3, 4, 5)
scala> val distData = sc.parallelize(data)
distData: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at <console>:14
One important parameter for parallel collections is the number of slices to cut the dataset into.
RDDs support two types of operations: transformations, which create a new dataset from an existing one, and actions, which return a value to the driver program after running a computation on the dataset.
