1.

What are the different running modes of Spark Application?

Answer»

Spark application can be RUN in the following three modes:

  1. LOCAL mode
  2. Cluster mode
  3. Client mode

Local mode :

This mode runs the entire Spark application on a single machine. It achieves parallelism through threads on that single machine. This is a common way to learn Spark, to test your applications, or experiment iteratively with local development. However, it is not recommended USING local mode for running production applications.

Cluster mode :

Cluster mode is the most common way of running Spark Applications in the computer cluster. In cluster mode, the user will SUBMIT a pre-compiled JAR, Python script, or R script to a cluster manager. The cluster manager then launches the driver process on one of the worker nodes inside the cluster, in addition to the executor processes which means that the cluster manager is responsible for maintaining all Spark Application– RELATED processes.

Client mode :

Client mode is nearly the same as cluster mode except that the Spark driver remains on the client machine i.e. the machine where that submitted the application. This means that the client machine is responsible for maintaining the Spark driver process, and the cluster manager maintains the executor processes.



Discussion

No Comment Found