Saved Bookmarks
| 1. |
How Can You Trigger Automatic Clean-ups In Spark To Handle Accumulated Metadata? |
|
Answer» You can trigger the clean-ups by SETTING the parameter ‘spark.cleaner.ttl’ or by dividing the long running jobs into DIFFERENT batches and WRITING the intermediary results to the DISK. You can trigger the clean-ups by setting the parameter ‘spark.cleaner.ttl’ or by dividing the long running jobs into different batches and writing the intermediary results to the disk. |
|