Resilient Distributed Datasets
Resilient Distributed Datasets (RDD) is a fundamental data structure of Spark. It is an immutable distributed collection of objects. Each dataset in RDD is divided into logical partitions, which may be computed on different nodes of the cluster. RDDs can contain any type of Python, Java, or Scala objects, including user-defined classes.
Formally, an RDD is a read-only, partitioned collection of records. RDDs can be created through deterministic operations on either data on stable storage or other RDDs. RDD is a fault-tolerant collection of elements that can be operated on in parallel.
There are two ways to create RDDs − parallelizing an existing collection in your driver program, or referencing a dataset in an external storage system, such as a shared file system, HDFS, HBase, or any data source offering a Hadoop Input Format.
Spark makes use of the concept of RDD to achieve faster and efficient MapReduce operations. Let us first discuss how MapReduce operations take place and why they are not so efficient.
Data Sharing using Spark RDD
Data sharing is slow in MapReduce due to replication, serialization, and disk IO. Most of the Hadoop applications, they spend more than 90% of the time doing HDFS read-write operations.
Recognizing this problem, researchers developed a specialized framework called Apache Spark. The key idea of spark is Resilient Distributed Datasets (RDD); it supports in-memory processing computation. This means, it stores the state of memory as an object across the jobs and the object is sharable between those jobs. Data sharing in memory is 10 to 100 times faster than network and Disk.
Let us now try to find out how iterative and interactive operations take place in Spark RDD.
Iterative Operations on Spark RDD
The illustration given below shows the iterative operations on Spark RDD. It will store intermediate results in a distributed memory instead of Stable storage (Disk) and make the system faster.
Note − If the Distributed memory (RAM) is not sufficient to store intermediate results (State of the JOB), then it will store those results on the disk.
Interactive Operations on Spark RDD
This illustration shows interactive operations on Spark RDD. If different queries are run on the same set of data repeatedly, this particular data can be kept in memory for better execution times.
By default, each transformed RDD may be recomputed each time you run an action on it. However, you may also persist an RDD in memory, in which case Spark will keep the elements around on the cluster for much faster access, the next time you query it. There is also support for persisting RDDs on disk, or replicated across multiple nodes.
5 Reasons For The Use Of RDDs
- You want low-level transformation and actions and control on your dataset;
- Your data is unstructured, such as media streams or streams of text;
- You want to manipulate your data with functional programming constructs than domain specific expressions;
- You don’t care about imposing a schema, such as columnar format while processing or accessing data attributes by name or column; and
- You can forgo some optimization and performance benefits available with DataFrames and Datasets for structured and semi-structured data.
How does RDD Make Working So Easy?
RDD lets you have all your input files like any other variable which is present. This is not possible by Using Map Reduce These RDDs get automatically distributed over the available network through partitions. Whenever an action is executed a task is launched per partition. This encourages parallelism, More the number of partitions more parallelism. The partitions are automatically determined by Spark. Once this is done two operations can be performed by RDDs. This includes actions and transformations.
What Can You do with RDD?
As mentioned in the previous point, it can be used for two operations. This includes actions and transformations. In the case of transformation, a new data set is created from an existing data set. Each data set is passed through a function. As a return value, it sends a new RDD as a result.
Actions on the other hand return value to the program. It performs the computations on the required data set. Here when the action is performed a new data set is not created. Hence they can be said as RDD operations that return non-RDD values. These values are stored either on external systems or to the drivers.
Advantages Of RDDs
The following are the major properties or advantages:
1. Immutable and Partitioned: All records are partitioned and hence RDD is the basic unit of parallelism. Each partition is logically divided and is immutable. This helps in achieving the consistency of data.
2. Coarse-Grained Operations: These are the operations that are applied to all elements which are present in a data set. To elaborate, if a data set has a map, a filter and a group by an operation then these will be performed on all elements which are present in that partition.
3. Transformation and Actions: After creating actions data can be read from only stable storage. This includes HDFS or by making transformations to existing RDDs. Actions can also be performed and saved separately.
4. Fault Tolerance: This is the major advantage of using it. Since a set of transformations are created all changes are logged and rather the actual data is not preferred to be changed.
5. Persistence: It can be reused which makes them persistent.
Drop your comment
