InterviewSolution
| 1. |
Can we use both Fair scheduler and Capacity Scheduler in the same Hadoop cluster, Brief about the same? |
|
Answer» Both the scheduler cannot be used in the same cluster. Both the scheduling algorithms have come up due to specific use-cases and cluster-wise you have to set up the configuration file for either Fair scheduler or Capacity Scheduler. you cannot set up both the scheduler for one cluster. you can choose the Fair Scheduler using below scheduler class in yarn-site.xml as mentioned below: <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value> </property>To use the Capacity Scheduler you have to configure the RESOURCE Manager in the conf/yarn-site.xml as mentioned below: yarn.resourcemanager.scheduler.class- org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler while setting up the queues in Capacity Scheduler you need to make some changes in etc/hadoop/capacity-scheduler.xml configuration file.The Capacity Scheduler has a predefined queue called root. whatever queues we will create in the system are children of the root queue.Setting up further queues- Configure property yarn.scheduler.capacity.root.queues with a list of comma-separated child queues.Setting up sub-queues within a queue- configure property yarn.scheduler.capacity.<queue-path>.queues queue-path can mention the full path of the queue’s hierarchy and it is starting at root with. (dot) as the delimiter. Queue capacity is provided in percentage (%). The sum of capacities for all queues, at each queue level, must be equal to 100. If there are free resources in the queue then APPLICATIONS in the queue may consume the required resources. Capacity scheduler queue configuration example: If there are two child queues starting from root XYZ and ABC. XYZ further divides the queue into technology and development. XYZ is given 60% of the cluster capacity and ABC is given 40% in this scenario please FIND the details as mentioned below to set up your yarn-site.xml. <property> <name>yarn.scheduler.capacity.root.queues</name> <value>XYZ, ABC</value> </property> <property> <name>yarn.scheduler.capacity.root.XYZ.queues</name> <value>technology,marketing</value> </property> <property> <name>yarn.scheduler.capacity.root.XYZ.capacity</name> <value>60</value> </property> <property> <name>yarn.scheduler.capacity.root.ABC.capacity</name> <value>40</value> </property> |
|