You have a setup of YARN cluster where the total application memory av

1.	You have a setup of YARN cluster where the total application memory available is 30GB there are two company queues such as Wipro and TCS. Wipro queue has 15gb allocated and TCS queue has 5gb allocated. Each map task requires 25GB allocation. How does the fair scheduler assign the available memory resources under the Dominant Resource Fairness(DRF) scheduler?
Answer» Resource allocation within the queues is controlled separately. Within a queue: FairScheduler can apply any of FIFO policy, FairPolicy or DominantResouceFairnessPolicy. CapacityScheduler can apply either FifoPolicy and fair policy. Fair Scheduler can use different scheduling policies. The default scheduling policy is fair sharing, using memory as a resource. There’s also a FIFO policy first in first out which is not much use. It’s quite common to use the third type of scheduling policy, DRF, which allocates both memory and CPU resources to applications,DRF is similar to fair-scheduling, but it is important to keep in mind that it applies primarily to the allocation of resources AMONG queues, an activity which is already dominated by queue weights. Thus, the most important thing about DRF is that considers MULTIPLE resources, rather than that it attempts to provide equal resource allocation. Initially, TCS and Wipro each have some resources allocated to jobs in their respective queues and only 10 GB remains in the cluster. Each queue is requesting to run a map task requiring 20 GB, so memory is available 30 GB and the rest of the REQUIRED resource will take from CPU. WIPRO currently holds 15 GB resources. Another 10 GB is required for mapper task so the fair scheduler will award a CONTAINER the requested 10 GB of memory to WIPRO. Now the available memory is 5 GB for TCS and it will require another 20 GB to run the mapper task. In this case, there is no memory available for TCS and DRF will try to use 5GB from memory and rest 20 GB can be used from the CPU.

You have a setup of YARN cluster where the total application memory available is 30GB there are two company queues such as Wipro and TCS. Wipro queue has 15gb allocated and TCS queue has 5gb allocated. Each map task requires 25GB allocation. How does the fair scheduler assign the available memory resources under the Dominant Resource Fairness(DRF) scheduler?

Answer»

Resource allocation within the queues is controlled separately. Within a queue:

FairScheduler can apply any of FIFO policy, FairPolicy or DominantResouceFairnessPolicy.

CapacityScheduler can apply either FifoPolicy and fair policy.

Fair Scheduler can use different scheduling policies. The default scheduling policy is fair sharing, using memory as a resource. There’s also a FIFO policy first in first out which is not much use. It’s quite common to use the third type of scheduling policy, DRF, which allocates both memory and CPU resources to applications,DRF is similar to fair-scheduling, but it is important to keep in mind that it applies primarily to the allocation of resources AMONG queues, an activity which is already dominated by queue weights. Thus, the most important thing about DRF is that considers MULTIPLE resources, rather than that it attempts to provide equal resource allocation.

Initially, TCS and Wipro each have some resources allocated to jobs in their respective queues and only 10 GB remains in the cluster. Each queue is requesting to run a map task requiring 20 GB, so memory is available 30 GB and the rest of the REQUIRED resource will take from CPU. WIPRO currently holds 15 GB resources. Another 10 GB is required for mapper task so the fair scheduler will award a CONTAINER the requested 10 GB of memory to WIPRO. Now the available memory is 5 GB for TCS and it will require another 20 GB to run the mapper task. In this case, there is no memory available for TCS and DRF will try to use 5GB from memory and rest 20 GB can be used from the CPU.

Discussion

No Comment Found

Related InterviewSolutions

Reply to Comment