Explore topic-wise InterviewSolutions in .

• YARN is one of the key FEATURES PROVIDED by SPARK that PROVIDES a central resource management PLATFORM for delivering scalable operations throughout the cluster.
• YARN is a cluster management technology and a Spark is a tool for data processing.

The process of redistribution of data across different partitions which might or might not CAUSE data movement across the JVM processes or the EXECUTORS on the separate MACHINES is known as shuffling/repartitioning. Partition is NOTHING but a smaller logical division of data.

It is to be noted that SPARK has no control over what partition the data gets distributed across.

Spark supports both the raw files and the STRUCTURED file FORMATS for efficient reading and processing. File formats like paraquet, JSON, XML, CSV, RC, Avro, TSV, etc are supported by Spark.

Receivers are those entities that consume DATA from different data SOURCES and then MOVE them to Spark for processing. They are created by using streaming contexts in the form of long-running tasks that are scheduled for operating in a round-robin fashion. Each receiver is configured to use up only a single core. The receivers are made to run on various executors to accomplish the task of data streaming. There are two types of receivers depending on how the data is sent to Spark:

• RELIABLE receivers: Here, the receiver sends an acknowledegment to the data sources post successful reception of data and its replication on the Spark STORAGE space.
• Unreliable receiver: Here, there is no acknowledgement sent to the data sources.

There are 2 deploy modes in Spark. They are:

• Client Mode: The deploy mode is said to be in client mode when the spark driver component runs on the machine node from where the spark job is submitted.
  • The main disadvantage of this mode is if the machine node fails, then the entire job fails.
  • This mode supports both interactive shells or the job submission commands.
  • The performance of this mode is worst and is not preferred in production environments.
• CLUSTER Mode: If the spark job driver component does not run on the machine from which the spark job has been submitted, then the deploy mode is said to be in cluster mode.
  • The spark job launches the driver component within the cluster as a part of the sub-process of ApplicationMaster.
  • This mode supports deployment only using the spark-submit command (interactive shell mode is not SUPPORTED).
  • Here, since the driver programs are run in ApplicationMaster, in case the program fails, the driver program is re-instantiated.
  • In this mode, there is a dedicated cluster manager (such as stand-alone, YARN, Apache Mesos, Kubernetes, etc) for allocating the resources required for the job to run as shown in the below architecture.

Apart from the above two modes, if we have to run the application on our local machines for unit TESTING and development, the deployment mode is called “Local Mode”. Here, the JOBS run on a single JVM in a single machine which makes it highly inefficient as at some point or the other there would be a shortage of resources which results in the failure of jobs. It is also not possible to scale up resources in this mode DUE to the restricted memory and space.

DAG stands for Directed Acyclic GRAPH with no directed cycles. There WOULD be finite VERTICES and edges. Each edge from one vertex is directed to ANOTHER vertex in a sequential manner. The vertices refer to the RDDs of Spark and the edges represent the operations to be performed on those RDDs.

RDD stands for Resilient Distribution DATASETS. It is a fault-tolerant collection of parallel running operational elements. The partitioned data of RDD is distributed and IMMUTABLE. There are two types of datasets:

• Parallelized COLLECTIONS: Meant for running parallelly.
• Hadoop datasets: These PERFORM operations on FILE record systems on HDFS or other storage systems.

• High Processing Speed: Apache Spark helps in the achievement of a very high processing speed of data by reducing read-write operations to disk. The speed is almost 100x faster while performing in-memory computation and 10x faster while performing disk computation.
• Dynamic Nature: Spark provides 80 high-level operators which help in the easy development of parallel applications.
• In-Memory Computation: The in-memory computation feature of Spark due to its DAG execution engine increases the speed of data processing. This also SUPPORTS data caching and reduces the time required to fetch data from the disk.
• Reusability: Spark codes can be reused for batch-processing, data streaming, running ad-hoc queries, etc.
• Fault Tolerance: Spark supports fault tolerance using RDD. Spark RDDs are the abstractions designed to handle failures of worker nodes which ensures zero data loss.
• Stream Processing: Spark supports stream processing in real-time. The problem in the earlier MapReduce framework was that it could process only already existing data.
• Lazy Evaluation: Spark transformations done using Spark RDDs are lazy. Meaning, they do not generate results right away, but they CREATE new RDDs from existing RDD. This lazy evaluation increases the system efficiency.
• Support Multiple Languages: Spark supports multiple languages like R, Scala, Python, Java which provides dynamicity and helps in overcoming the Hadoop limitation of application development only using Java.
• Hadoop Integration: Spark also supports the Hadoop YARN cluster manager thereby making it flexible.
• Supports Spark GraphX for graph parallel execution, Spark SQL, libraries for MACHINE learning, etc.
• Cost Efficiency: Apache Spark is considered a better cost-efficient solution when compared to Hadoop as Hadoop required large storage and data CENTERS while data processing and replication.
• Active Developer’s Community: Apache Spark has a large developers BASE involved in continuous development. It is considered to be the most important project undertaken by the Apache community.

Apache Spark is an open-source framework engine that is known for its speed, easy-to-use nature in the field of big data PROCESSING and analysis. It also has built-in MODULES for GRAPH processing, machine LEARNING, streaming, SQL, etc. The spark execution engine supports in-memory computation and cyclic data flow and it can run either on cluster mode or STANDALONE mode and can access diverse data sources like HBase, HDFS, Cassandra, etc.