How can the data transfers be minimized while working with Spark?

1.	How can the data transfers be minimized while working with Spark?
Answer» Data transfers correspond to the process of shuffling. Minimizing these transfers results in faster and RELIABLE RUNNING Spark applications. There are various ways in which these can be MINIMIZED. They are: Usage of Broadcast Variables: Broadcast variables increases the efficiency of the join between LARGE and small RDDs. Usage of Accumulators: These help to update the variable values parallelly during execution. Another common WAY is to avoid the operations which trigger these reshuffles.

Answer»

Data transfers correspond to the process of shuffling. Minimizing these transfers results in faster and RELIABLE RUNNING Spark applications. There are various ways in which these can be MINIMIZED. They are:

Usage of Broadcast Variables: Broadcast variables increases the efficiency of the join between LARGE and small RDDs.
Usage of Accumulators: These help to update the variable values parallelly during execution.
Another common WAY is to avoid the operations which trigger these reshuffles.

Discussion

No Comment Found

Related InterviewSolutions

How can you achieve machine learning in Spark?
What API is used for Graph Implementation in Spark?
Define Piping in Spark.
How is Caching relevant in Spark Streaming?
How are automatic clean-ups triggered in Spark for handling the accumulated metadata?
What are Sparse Vectors? How are they different from dense vectors?
Can Apache Spark be used along with Hadoop? If yes, then how?
Differentiate between Spark Datasets, Dataframes and RDDs.
Why do we need broadcast variables in Spark?
What are the steps to calculate the executor memory?

How can the data transfers be minimized while working with Spark?

Discussion

No Comment Found

Related InterviewSolutions

Reply to Comment