An Introduction to Apache Hadoop

Sam Erickson
Updated 20 hours ago
0

Key Takeaways

Hadoop has four core components: Hadoop Common, HDFS, MapReduce, and YARN.
HDFS stores distributed data; YARN allocates resources for application processes. A DataNode and a NodeManager are separate services, even when they share a machine.
MapReduce distributes batch work through map, shuffle/sort, and reduce stages. Distribution doesn't automatically make a job faster.
HDFS checkpointing, NameNode high availability, and protection of stored data solve different problems; none replaces a backup strategy.

Apache Hadoop is an open-source framework for distributed storage and processing of large datasets across a cluster of computers. Apache Hadoop architecture separates shared utilities, storage, resource management, and batch processing into four components, so you can understand each responsibility independently.

For 2026, treat Hadoop as an important architecture to understand—not an automatic choice for every new data platform. Its strongest case is often an existing HDFS/YARN environment or a workload that justifies operating that infrastructure.

What Is Hadoop? The Four Core Components

The official Hadoop module list names Hadoop Common, HDFS, YARN, and MapReduce. Tools that integrate with them belong to the broader Hadoop ecosystem, not the core framework.

Hadoop Common: Shared Libraries and Utilities

Hadoop Common supplies libraries and utilities used by the other modules. Think of it as shared infrastructure rather than a separate storage service or processing engine.

HDFS: Distributed File Storage

The Hadoop Distributed File System, or HDFS, divides files into blocks distributed across storage workers called DataNodes. The NameNode manages the filesystem namespace—files and directories—and tracks which DataNodes hold each block. DataNodes store and serve the block data.

These names describe services, not interchangeable kinds of machine. A cluster can have dedicated NameNode hosts, storage workers, compute workers, and client machines; a host may run multiple services.

How HDFS Reads and Writes Work

The distinction to remember is metadata versus file contents. As the HDFS user guide explains, clients contact the NameNode for metadata but transfer file data directly with DataNodes.

Read: The client asks the NameNode for block locations, then reads those blocks from appropriate DataNodes.
Replicated write: The client requests file creation and block targets from the NameNode, then streams data through a pipeline of DataNodes that store the replicas.

The NameNode isn't a proxy through which all file bytes pass. Separating metadata operations from data transfer lets multiple clients and DataNodes move data in parallel.

Replication, Rack Awareness, and Erasure Coding

HDFS uses rack topology when placing replicas. A rack is a group of machines that may share networking and other infrastructure; placing copies across racks helps protect against failures affecting an entire rack. This is an HDFS placement responsibility, not YARN resource allocation.

Replication stores additional copies of blocks. HDFS erasure coding instead stores data and parity fragments that can reconstruct missing pieces within the policy's failure tolerance. It can reduce storage overhead compared with replication, but encoding and reconstruction add CPU and network demands.

Neither mechanism guarantees that data can never be lost. Protection depends on the configured policy, placement across failure domains, available capacity, and timely repair after failures.

Secondary NameNode Versus Standby NameNode

A Secondary NameNode periodically merges the filesystem image with the edits log to create a checkpoint. It doesn't automatically take over when the NameNode fails.

For high availability, HDFS can use an active NameNode and standby NameNodes. With the Quorum Journal Manager, the active records metadata changes to a majority of JournalNodes; standbys read those edits to stay synchronized. Automatic failover requires additional coordination and configuration. In this HA arrangement, standbys also perform checkpointing, so a separate Secondary NameNode isn't used.

Hadoop MapReduce: Distributed Batch Processing

MapReduce is Hadoop's framework for running parallel batch jobs. The framework divides input into logical splits, runs map tasks, and can retry failed tasks. A logical input split describes work for a mapper; it isn't the same thing as a physical HDFS block.

A word-count job makes the processing model concrete:

Map: Read input records and emit intermediate key-value pairs, such as (hadoop, 1) for each occurrence of a word.
Shuffle and sort: Partition and transfer mapper output to reducers, sorting and grouping it by key so matching words reach the same reducer.
Reduce: Process each key and its grouped values—for example, sum the counts—and write the results.

Input splits
    |
    v
Map tasks: (word, 1)
    |
    v
Shuffle / sort: word -> [1, 1, ...]
    |
    v
Reduce tasks: (word, total)
    |
    v
Output files

Shuffle and sort are the essential bridge between map and reduce, not an optional implementation detail. They move and organize the intermediate data that reducers consume.

MapReduce isn't inherently a speed boost. Parallel execution can help large, divisible workloads, but scheduling, serialization, network transfers, and disk I/O introduce overhead. For a smaller job, that overhead can outweigh the benefit of adding workers. See our MapReduce quick explanation for another view of the model.

Hadoop YARN: Resource Management

YARN separates cluster resource management from application-specific processing. The older MapReduce JobTracker combined resource management and job coordination; YARN separates these responsibilities so resource allocation isn't tied to one processing model.

The YARN architecture has three central roles:

ResourceManager: Arbitrates cluster resources among applications through its scheduler.
NodeManager: Runs on a worker machine, manages application containers, and reports resource usage to the ResourceManager.
ApplicationMaster: Coordinates one application, negotiates resources, and works with NodeManagers to launch and monitor its tasks.

A YARN container is a resource allocation for an application process, typically including memory and CPU—not a piece of stored data or a memory-only object. The ApplicationMaster itself starts in a container.

This division lets the ResourceManager focus on sharing cluster capacity while ApplicationMasters handle application execution. Our Hadoop YARN introduction explores those relationships further.

Apache Hadoop Architecture: How HDFS and YARN Work Together

HDFS answers “where are the bytes?” YARN answers “where can this application run?” MapReduce uses both when processing files stored in HDFS.

HDFS metadata                  YARN resource management
+-------------------+          +-------------------+
| NameNode          |          | ResourceManager   |
+-------------------+          +-------------------+
          |                              |
          | metadata / reports           | allocations / reports
          v                              v
+-------------------------+  +-------------------------+
| Worker A                |  | Worker B                |
| DataNode: HDFS blocks   |  | DataNode: HDFS blocks   |
| NodeManager: containers |  | NodeManager: containers |
| - ApplicationMaster    |  | - task processes        |
| - task processes       |  |                         |
+-------------------------+  +-------------------------+
ApplicationMaster negotiates with ResourceManager
and coordinates task launches with NodeManagers.

This is a simplified placement example, not a required topology; HA services are omitted. DataNodes and NodeManagers may share worker machines, but storage and compute roles can also be separated.

For an HDFS-backed MapReduce job, the flow is:

The client submits the application to YARN.
YARN starts its ApplicationMaster in a container.
The ApplicationMaster requests resources and coordinates map and reduce tasks on NodeManagers.
Tasks use HDFS clients to read input from DataNodes; locality preferences can help place computation near its input.
Map output is shuffled to reducers, which write final results to the configured output filesystem, such as HDFS.

YARN scheduling may consider data locality, but it doesn't decide where HDFS stores replicas. Storage placement and compute placement are related optimizations owned by different services.

Term	Responsibility	Meaning
HDFS block	Storage	A unit of file data stored on DataNodes.
Input split	Processing	A logical portion of input assigned to a mapper.
YARN container	Resources	An allocation used to run an application process.

The Hadoop Ecosystem: Separate Projects, Different Jobs

HBase, Flume, Spark, and Hive are independent Apache projects that can integrate with Hadoop. They aren't extra Hadoop core modules, and you don't need all of them in a deployment.

HBase: Record-Oriented Database Access

Apache HBase is a distributed, non-relational database designed for random, real-time reads and writes. It commonly uses HDFS beneath its database services. That addresses a different need from scanning files for batch analytics.

Flume: Log and Event Collection

Apache Flume collects, aggregates, and moves log and event data. Its source-channel-sink architecture can deliver events into HDFS. Treat it as an ingestion project to evaluate on its own merits, including maintenance and integration requirements—not a mandatory Hadoop component.

Spark: An Alternative Processing Engine

Apache Spark supports batch processing, streaming, and machine learning. It can read HDFS data and run on YARN, but it can also use other storage systems and cluster managers.

Spark's execution model and caching can benefit workloads that reuse data, but “Spark is always faster” isn't a useful rule. Spark also performs shuffles and can spill data to disk; compare the actual workload, execution plan, data layout, and configuration. Our Hadoop MapReduce versus Spark comparison focuses on the processing-engine distinction.

Hive: SQL Data Warehousing

Apache Hive provides SQL-based reading, writing, and management of datasets in distributed storage. Its storage integrations and execution engine depend on the deployment; it isn't simply a query language running directly on HDFS. For the database-versus-analytics distinction, read Hive versus HBase.

Where Hadoop Fits in 2026

Keeping Hadoop can make sense when you already have substantial HDFS data, established YARN workloads, and the operational expertise to maintain the cluster. Large batch scans and transformations remain consistent with HDFS's throughput-oriented design.

For a new platform, compare that operating model with object storage and managed processing or lakehouse services. Make the decision around workload requirements, data location, recovery needs, and operating cost—not the assumption that large datasets require HDFS.

Cloud deployment doesn't imply a single architecture. You can operate Hadoop on cloud virtual machines or use a managed service: Amazon EMR, for example, supports Hadoop processing with HDFS or Amazon S3 storage. Separately, Spark supports Kubernetes as a cluster manager. That's a different resource-management path, not a standard platform called “YARN on Kubernetes.”

Common Mistakes

Expecting low-latency file access: HDFS emphasizes high-throughput access to large datasets rather than interactive, low-latency filesystem workloads.
Treating more workers as a performance fix: First identify whether the job is constrained by computation, data movement, storage, or scheduling overhead.
Confusing redundancy with recovery: Plan for operator mistakes and wider failures, not just a failed disk or worker.
Leaving security until deployment: Plan authentication, authorization, and network protection together. Hadoop's secure-mode documentation covers Kerberos authentication and protection of Hadoop services.

FAQ

How do HDFS and YARN work together?

HDFS stores files across DataNodes, while YARN allocates resources for application processes. A processing framework such as MapReduce uses YARN to run tasks that read and write HDFS data.

What is the difference between a Secondary NameNode and a standby NameNode?

A Secondary NameNode creates filesystem metadata checkpoints; it doesn't provide automatic failover. A standby NameNode maintains synchronized metadata for takeover in an HA deployment, with automatic failover requiring additional configuration.

Is Hadoop still relevant in 2026?

Yes, particularly for existing HDFS/YARN environments and large batch workloads that justify maintaining a cluster. For a new platform, evaluate object storage and managed processing rather than assuming Hadoop is the default.

How is Spark different from Hadoop MapReduce?

Spark and Hadoop MapReduce are separate processing engines. Spark supports a broader execution model and caching, but performance depends on the workload, data layout, and configuration; it isn't universally faster.

Can Hadoop be used in cloud environments?

Yes. Hadoop can run on cloud virtual machines, and managed services such as Amazon EMR support Hadoop workloads with HDFS or object storage. Running Spark on Kubernetes is a separate deployment pattern, not the same as running YARN.