Hadoop YARN Introduction

Sam Erickson
Updated 20 hours ago
0

Key Takeaways

YARN separates cluster resource allocation from application-specific execution.
The ResourceManager allocates containers; NodeManagers launch them; each ApplicationMaster coordinates its application.
A YARN container is a resource allocation, not automatically a Docker container.
YARN can host streaming frameworks, but it does not provide stream-processing semantics or guarantee low latency.

YARN, short for Yet Another Resource Negotiator, is Hadoop's cluster resource-management layer. It separates the decision about which application gets cluster resources from the framework-specific work of coordinating that application's execution. This lets frameworks beyond MapReduce share a Hadoop cluster without putting every application's task coordination inside one central service.

If you need the broader context first, start with Apache Hadoop architecture. Here, we'll focus on the resource-management components and the path from application submission to completion.

Problems with Classic MapReduce

Classic Hadoop 1.x MapReduce used a central JobTracker and a TaskTracker on each worker node. The JobTracker combined cluster resource management with job coordination: accepting submissions, scheduling tasks, monitoring progress, and arranging retries. TaskTrackers executed the tasks assigned to them.

That combination concentrated cluster-wide scheduling and per-job bookkeeping in the same master. Architecturally, more jobs and tasks meant more coordination work for that process, and its availability affected the whole cluster. The problem wasn't that the map and reduce operations themselves were wrong; it was that their execution framework also owned cluster resource management.

For a refresher on the processing model rather than its cluster infrastructure, see how MapReduce works.

Introducing YARN

YARN splits the JobTracker's combined responsibilities between a global ResourceManager and a separate ApplicationMaster for each application. NodeManagers handle execution on individual worker machines. MapReduce can run on this infrastructure, but it is no longer the only processing framework the resource-management layer can serve.

Responsibility	Classic MapReduce	YARN-based execution
Cluster resources	The JobTracker manages cluster capacity.	The ResourceManager Scheduler allocates containers.
Application coordination	The JobTracker schedules and monitors job tasks.	Each ApplicationMaster coordinates its application.
Worker execution	The TaskTracker runs MapReduce tasks.	The NodeManager launches and manages containers.

This removes the classic combination of global resource management and every job's task coordination from one process. It does not eliminate centralized resource allocation, and it does not automatically make a workload faster. Treat improved scalability and resource sharing as architectural opportunities, not guaranteed performance results.

Where YARN Fits in 2026

YARN is a deployment choice, not a prerequisite for every new data platform. Spark also supports Kubernetes and its own standalone cluster manager, while managed runtimes such as Amazon EMR Serverless let you run analytics without operating a cluster yourself.

For an existing Hadoop platform, understand YARN because its queues, resource policies, and framework integrations determine how your applications get compute capacity. For a new platform, evaluate those operational responsibilities rather than assuming Hadoop requires you to build a YARN cluster.

Apache Hadoop YARN Architecture

The main components are the ResourceManager, NodeManagers, containers, and ApplicationMasters. The diagram combines their placement with the numbered startup sequence explained below; arrows represent control messages, not application data transfers.

[Client] --1 submit--> [ResourceManager]
                      [ApplicationsManager | Scheduler]
                                |
                         2 allocate AM container
                         3 request AM launch
                                v
                      [NodeManager A]
                      [AM container: ApplicationMaster]

ApplicationMaster --4 register / resource requests--> ResourceManager
ApplicationMaster <--5 container allocations--------- ResourceManager
ApplicationMaster --6 work-container launch requests-> NodeManagers A/B
                                                       [work containers]
ApplicationMaster --7 final status / deregister-----> ResourceManager
NodeManagers ------health / container status--------> ResourceManager

ResourceManager: Cluster-Wide Resource Allocation

The ResourceManager, usually abbreviated RM, arbitrates resources across applications. Its two main components have different jobs:

Scheduler: Allocates containers according to resource requests, available capacity, queues, and configured scheduling policies. It does not coordinate application tasks or implement their retry logic.
ApplicationsManager: Accepts application submissions, arranges the initial ApplicationMaster container, and manages ApplicationMaster restart attempts after failures.

For example, the CapacityScheduler lets administrators organize resource sharing through queues and limits. A resource request is therefore not an immediate entitlement to run: allocation depends on the configured policy as well as available resources.

NodeManager: Execution on One Worker

A NodeManager, or NM, is the worker-side agent that launches and manages containers on its machine. It reports node health and container information to the ResourceManager through heartbeats. If its health checks mark the node unhealthy, the ResourceManager stops assigning new containers there.

The important boundary is allocation versus execution: the ResourceManager grants resources, but a NodeManager starts the corresponding process. The NodeManager documentation also explains disk-health checks and restart recovery.

Containers: Resource Allocations on a Node

A YARN container represents resources allocated on a particular node. Memory and virtual CPU cores, or vcores, are the principal resource dimensions; administrators can configure additional countable resource types, such as GPUs. Which dimensions influence scheduling depends on the scheduler configuration.

For example, an application might request a container with 4 GiB of memory and two vcores. That request is not a promise of two dedicated physical cores or a fixed amount of disk and network bandwidth. CPU enforcement depends on operating-system controls and configuration, including Linux cgroups integration.

ApplicationMaster: Coordination for One Application

The ApplicationMaster, or AM, is a framework-specific coordinator for an application. In the normal managed lifecycle, it runs inside a YARN container, registers with the ResourceManager, negotiates further containers, and works with NodeManagers to launch and monitor application work.

It is not a universal task executor shared by every application. Its execution and recovery behavior belongs to the framework using YARN; simply having an ApplicationMaster does not guarantee efficient resource utilization.

YARN Job Process: From Submission to Completion

For a typical managed YARN application, the lifecycle follows these seven steps. Hadoop's application-writing guide describes the client and coordinator interactions in more detail.

Submit: The client submits an application to the ResourceManager, including its queue and the information needed to launch the ApplicationMaster.
Allocate the AM container: The ResourceManager's Scheduler allocates resources for the ApplicationMaster, subject to scheduling constraints.
Launch the AM: The ResourceManager requests that the selected NodeManager launch the ApplicationMaster in that allocated container.
Register and request: The ApplicationMaster registers with the ResourceManager and requests containers for application work.
Receive allocations: The ResourceManager returns container allocations as the Scheduler grants them.
Launch and coordinate work: The ApplicationMaster sends launch requests to the relevant NodeManagers, which start the work containers. The framework coordinates execution and monitors progress.
Complete: The ApplicationMaster reports its final status, deregisters from the ResourceManager, and exits. YARN cleans up the application's resources.

Registration, allocation, and completion are AM-to-RM interactions; starting work containers is an AM-to-NodeManager interaction. Keeping those paths separate explains why obtaining an allocation and successfully starting a process are different events.

To connect this architecture to a familiar batch workload, follow the Hadoop MapReduce WordCount example.

Beyond MapReduce: Spark and Streaming Applications

Apache Spark is a concrete example of a non-MapReduce framework that integrates with YARN. In Spark's cluster deploy mode, the driver runs inside the ApplicationMaster process, while executors run in other containers. Spark's driver schedules Spark tasks; YARN supplies the cluster resource allocations.

That integration belongs to Spark, not to a built-in Spark engine inside YARN. If the framework is new to you, read what Apache Spark does before comparing its deployment options.

The same distinction matters for streaming. A framework such as Spark Structured Streaming supplies the processing model, checkpointing, and recovery behavior. YARN can supply resources to that application, but stream-processing semantics and latency come from the framework, workload, and configuration—not from YARN itself.

Failures, Scheduling, and Operational Tradeoffs

Separating responsibilities also separates failure boundaries. You still need to configure recovery and understand which component owns the failed work.

ResourceManager failure: RM high availability uses an active/standby architecture. Recovering application state also depends on the configured state store and restart behavior; work-preserving RM recovery is designed to reconstruct running state without killing existing applications.
Node or container failure: Replacement work and retries are framework responsibilities, not guarantees from the resource Scheduler. Configured NodeManager restart recovery can preserve active containers across an NM restart, but that is not protection against losing the underlying machine.
ApplicationMaster failure: YARN can start another application attempt within configured attempt limits. Whether that attempt preserves containers or recovers useful application progress depends on the submission settings and framework implementation.

Resource requests and queue policy matter as much as component layout. Oversized requests can waste capacity, while undersized memory allocations can cause container termination. CapacityScheduler limits on parallel applications and AM resource usage can also leave an accepted application waiting to start.

Common Mistakes

Treating acceptance as execution: If an application is waiting, inspect queue capacity, application limits, and AM resource availability before assuming the worker machines are broken.
Treating allocation as successful launch: Check container launch errors and logs separately from scheduler decisions. A granted allocation does not prove the requested command started successfully.
Treating scheduling as security isolation: Before accepting untrusted submissions, review Hadoop authentication, authorization, and container-execution security—not just resource quotas.

Conclusion

YARN's core improvement is separation of responsibility: cluster-wide resource allocation stays with the ResourceManager, local execution stays with NodeManagers, and application coordination belongs to each framework's ApplicationMaster. Use those boundaries to reason about resource sharing and failures, rather than expecting YARN alone to deliver faster processing or real-time behavior.

FAQ

What is YARN in Apache Hadoop?

YARN is Hadoop's cluster resource-management layer. It separates cluster resource allocation from application-specific execution so different processing frameworks can share a cluster.

What is the difference between YARN and MapReduce?

YARN manages cluster resources, while MapReduce is a processing framework that can run on YARN. YARN replaces the classic JobTracker/TaskTracker infrastructure, not the map-and-reduce programming model.

What do the ResourceManager, NodeManager, and ApplicationMaster do?

The ResourceManager allocates resources across applications, a NodeManager launches and manages containers on one worker, and an ApplicationMaster coordinates one application's execution. The ResourceManager's Scheduler does not coordinate every framework's tasks.

What is a YARN container, and what resources does it allocate?

A YARN container is a resource allocation on a node, principally expressed in memory and vcores, with additional resource types available through configuration. It is not automatically a Docker container or a reservation of dedicated physical CPU cores, disk bandwidth, and network bandwidth.

Does YARN itself provide real-time stream processing?

No. YARN can allocate resources to streaming frameworks, but processing semantics, checkpointing, and latency depend on the framework and workload. Moving a batch MapReduce application onto YARN does not make it real-time.