Saved Bookmarks
| 1. |
Does Apache Spark Provide Check Pointing? |
|
Answer» Lineage graphs are always useful to recover RDDs from a failure but this is generally time consuming if the RDDs have long lineage CHAINS. SPARK has an API for check pointing i.e. a REPLICATE flag to PERSIST. However, the DECISION on which data to checkpoint - is decided by the user. Checkpoints are useful when the lineage graphs are long and have wide dependencies. Lineage graphs are always useful to recover RDDs from a failure but this is generally time consuming if the RDDs have long lineage chains. Spark has an API for check pointing i.e. a REPLICATE flag to persist. However, the decision on which data to checkpoint - is decided by the user. Checkpoints are useful when the lineage graphs are long and have wide dependencies. |
|