1.

What is Spark Driver and what are the roles and responsibility perform after the job submission?

Answer»

Spark driver prepares the context and declares the operations on the DATA using RDD transformations and actions. Driver submits the serialized RDD graph to the master, where master creates tasks out of it and submits them to the workers for execution. Executor is a distributed agent responsible for the execution of tasks.

Below is the key point for the reference:

  • Spark driver plays the vital role which is kickoff from execute the main () function.
  • control the node in the cluster and PERFORMING below three operation:
  •  maintaining information about the Spark Application
  •  responding to a user’s program or input
  •  analyzing, distributing, and scheduling work ACROSS the executors
  • Spark driver application has its own executor process.
  • Executor PERFORMS all the data processing.
  • Reads from and Writes data to EXTERNAL sources.
  • Stores the computation results data in-memory.
  • Interacts with the storage systems.

Spark driver coordinates the different job stages, where the tasks are actually executed. They should have the resources and network connectivity required to execute the operations requested on the RDDs.



Discussion

No Comment Found