What are the steps to calculate the executor memory?

1.	What are the steps to calculate the executor memory?
Answer» Consider you have the below details regarding the cluster: Number of nodes = 10Number of cores in each node = 15 coresRAM of each node = 61GB To IDENTIFY the number of cores, we follow the approach: Number of Cores = number of CONCURRENT tasks that can be run parallelly by the EXECUTOR. The optimal value as part of a general rule of thumb is 5. Hence to calculate the number of executors, we follow the below approach: Number of executors = Number of cores/Concurrent Task = 15/5 = 3Number of executors = Number of nodes * Number of executor in each node = 10 * 3 = 30 executors per SPARK job

Answer»

Consider you have the below details regarding the cluster:

Number of nodes = 10Number of cores in each node = 15 coresRAM of each node = 61GB

To IDENTIFY the number of cores, we follow the approach:

Number of Cores = number of CONCURRENT tasks that can be run parallelly by the EXECUTOR. The optimal value as part of a general rule of thumb is 5.

Hence to calculate the number of executors, we follow the below approach:

Number of executors = Number of cores/Concurrent Task = 15/5 = 3Number of executors = Number of nodes * Number of executor in each node = 10 * 3 = 30 executors per SPARK job

Discussion

No Comment Found

Related InterviewSolutions

How can you achieve machine learning in Spark?
What API is used for Graph Implementation in Spark?
Define Piping in Spark.
How is Caching relevant in Spark Streaming?
How are automatic clean-ups triggered in Spark for handling the accumulated metadata?
What are Sparse Vectors? How are they different from dense vectors?
Can Apache Spark be used along with Hadoop? If yes, then how?
Differentiate between Spark Datasets, Dataframes and RDDs.
Why do we need broadcast variables in Spark?
What are the steps to calculate the executor memory?