Hadoop 2.x includes YARN (Yet Another Resource Negotiator) which is a resource management layer on top of Hadoop ecosystem.
Prior to 2.x Job Tracker was responsible for managing the cluster resources and the lifecycle of MapReduce jobs (which includes scheduling, running application, monitoring progress, providing failover). This design hard binds MapReduce only workloads to be run atop HDFS data.
YARN was designed to split up the functionalities of Job Tracker into separate daemons, which decouples resource management and job scheduling from running, monitoring and providing failover execution flow. The separated daemons/processes are –
- ResourceManager (RM): for managing and allocating resources and scheduling applications
- ApplicationMaster (AM): for executing applications and providing failover
This decoupling of responsibilities enables better utilization of cluster resources and ability to run applications beyond MapReduce. Let’s see how.
There is one RM at cluster level, while there is one AM per application.
RM has two components –
- Scheduler – allocates resources to submitted applications (aka jobs) and schedules them for execution.
- ApplicationsManager (AsM) – accepts job-submissions, negotiatesresources for starting per application AM and also provides service to relaunch AM in case of failure.
RM has been implemented as Master with NodeManger (NM) being the workers/agents/slaves up and running on all the slave nodes of cluster.
Together RM and NM form a resource computation framework.
- NM tracks the total available resources local to the node it is running on and their utilization status and sends periodic reports to RM.
- NM primarily tracks – RAM (memory) and number of vCores (for simplicity let’s assume vCores = number of physical CPU cores).
- NMs are also responsible for launching and supervising the containersand communicating the status of resources consumed by them to the RM.
- A container in YARN is where a unit of work happens in the form of task. A job/application is split in tasks and each task gets executed in one container having a specific amount of allocated resources.
- A container can be understood as logical reservation of resources (memory and vCores) that will be utilized by task running in that container.
For example, your cluster has 4 nodes each having 4 GB RAM and 4 CPU Cores, making a total of 16 GB RAM and 16 CPU Cores, of these let’s say 12 GB x 14 cores are available for use by YARN.
Now, we submit a map only MapReduce job which would spawn 8 map tasks each requiring 1 GB memory and 1 core, than your NM will spawn 8 containers each having reserved resources = 1 GB x 1 core.
- There can be one or more containers on given slave node depending upon the size of requested resources. In our example cluster has sufficient resources to execute all 8 tasks in parallel. In case we had less resources available the task would get queued and get processed as and when resources becomes available.
Understanding lifecycle of a YARN application/job
Now that we know a bit of basics about of all of the YARN components – RM, NM, AM, AsM and Containers, let’s see how they participate in lifecycle of a YARN application/job using below image from Apache Hadoop site.
- Client submits a new application/job request to RM
- RM (AsM) accepts the job request. Based on the cluster wide resource stats RM (AsM) finds a slave node that is not too busy.RM (AsM) instructs the NM on this node to launch a container to be used as AM for the given application/job. AM can be thought of as a specialized container which will manage/monitor application/job tasks.
- AM sends request to RM, asking for resources required to run the application/job.
- RM responds back with a list of containers along with list of salve nodes that they can be spawned on.
- AM starts the containers on each of specified salve nodes, it is also responsible for providing progress status to client whenever requested.
I have tried to include all important points and keep brevity at the same time.
Hope this helps.