How Can You Minimize Data Transfers When Working With Spark?

1.	How Can You Minimize Data Transfers When Working With Spark?
Answer» MINIMIZING data TRANSFERS and avoiding shuffling helps WRITE spark programs that run in a fast and reliable manner. The VARIOUS ways in which data transfers can be minimized when working with Apache Spark are: Using Broadcast Variable- Broadcast variable enhances the efficiency of joins between small and LARGE RDDs. Using Accumulators – Accumulators help update the values of variables in parallel while executing. The most common way is to avoid operations ByKey, repartition or any other operations which trigger shuffles. Minimizing data transfers and avoiding shuffling helps write spark programs that run in a fast and reliable manner. The various ways in which data transfers can be minimized when working with Apache Spark are:

Answer»

MINIMIZING data TRANSFERS and avoiding shuffling helps WRITE spark programs that run in a fast and reliable manner.

The VARIOUS ways in which data transfers can be minimized when working with Apache Spark are:

Using Broadcast Variable- Broadcast variable enhances the efficiency of joins between small and LARGE RDDs.
Using Accumulators – Accumulators help update the values of variables in parallel while executing.
The most common way is to avoid operations ByKey, repartition or any other operations which trigger shuffles.

Minimizing data transfers and avoiding shuffling helps write spark programs that run in a fast and reliable manner.

The various ways in which data transfers can be minimized when working with Apache Spark are:

Discussion