Key Takeaways
- MapReduce processes large datasets through parallel map tasks, a shuffle that groups records by key, and reduce tasks that process those groups.
- Hadoop can schedule map work near its input data, reducing input transfer. Shuffle can still move substantial data across the network.
- Reducers work in parallel on different key partitions; one reducer doesn't necessarily process the entire dataset.
- MapReduce is a batch model, not an automatic speedup. Startup, network, and disk costs can outweigh parallelism, especially for small or iterative workloads.
MapReduce is a distributed programming model for processing large datasets in parallel. You write a map function that emits intermediate key/value pairs and a reduce function that processes the values associated with each key; the framework handles distributing the work.
Apache Hadoop MapReduce is a concrete implementation. The easiest way to understand it is to follow two text files through map, shuffle, and reduce—not to start with cluster installation.
What is MapReduce?
The central idea is to split a large computation into independent tasks, then bring related intermediate results together. In the MapReduce programming model, the framework handles scheduling, communication between machines, and recovery from worker failures.
This is distributed task parallelism: multiple tasks can run across multiple machines. It isn't simply another name for multithreading within one process.
Keep the model separate from its surroundings. HDFS—the Hadoop Distributed File System—stores data across machines; MapReduce processes data. Our introduction to Apache Hadoop architecture explains how the pieces fit together.
How MapReduce works
For a job with reducers, the logical flow looks like this. The diagram shows data movement, not a requirement that each task occupy a separate machine.
Input splits Map tasks Intermediate records
split A --------> map A --------> (key, value) ...
split B --------> map B --------> (key, value) ...
|
partition + transfer + sort/group
|
+---------+---------+
| |
reducer 0 reducer 1
| |
output part 0 output part 11) Map: emit intermediate records
Hadoop creates a map task for each input split, a logical portion of the input. Each task processes records from that split and emits zero or more intermediate key/value pairs per input record, as described by the Mapper contract.
For word count, the mapper emits (word, 1) for every word it encounters. A file isn't inherently a task: input splitting determines the map-task boundaries.
2) Shuffle: partition, transfer, sort, and group
Shuffle brings records together by key, not by value or by identical key/value pair. Two records such as (Hadoop, 1) and (Hadoop, 3) belong to the same key group despite having different values.
- Partition: assign each intermediate key to a reducer. Hadoop's Partitioner controls that assignment, typically using a hash of the key.
- Transfer: reducers fetch their assigned intermediate records from map outputs.
- Sort and group: Hadoop merge-sorts reducer input by key so values belonging to a key can be processed together.
Hadoop's Reducer documentation describes the fetch-and-sort behavior. Sorting here applies to reducer input; it doesn't automatically make all output files one globally sorted result.
3) Reduce: process one key's values
The reduce function receives a key and an iterable of its intermediate values. For word count, it sums those values and emits (word, count).
Different reducers can process different key partitions concurrently, subject to available resources. A reducer handles the keys in its partition, not arbitrary pairs gathered from the whole cluster. Job output is typically written to a filesystem such as HDFS rather than automatically returned as one result to the submitting client.
A Basic Example: MapReduce word count
Suppose you have two text files, illustrated on different HDFS data nodes. Their placement is simplified; HDFS can maintain replicas on other nodes. We'll split words on whitespace and preserve capitalization.
Input files
Node 1: file-a.txt
Hello from Hadoop
Node 2: file-b.txt
Goodbye from HadoopThe goal is to count occurrences of each word, not produce a single total. This follows the same counting pattern as Hadoop's official WordCount example.
Map outputs
For illustration, let one map task process each file. Each word becomes a key paired with the number 1.
Map A:
(Hello, 1)
(from, 1)
(Hadoop, 1)
Map B:
(Goodbye, 1)
(from, 1)
(Hadoop, 1)Grouped shuffle input
Shuffle gathers the values associated with each word. Here are the logical groups, shown together for readability rather than as a claim about reducer placement or global ordering:
(Goodbye, [1])
(Hadoop, [1, 1])
(Hello, [1])
(from, [1, 1])The brackets illustrate the values supplied to reduce; they aren't a requirement to build an in-memory list. Notice that the two (Hadoop, 1) records become (Hadoop, [1, 1]), not one record that has already been counted.
Reduce outputs
Each reducer sums the values for each word assigned to it:
(Goodbye, 1)
(Hadoop, 2)
(Hello, 1)
(from, 2)These are four per-word frequencies. Adding their counts gives the total number of words: 1 + 2 + 1 + 2 = 6. For an implementation-oriented next step, see our Hadoop MapReduce WordCount example.
What runs the work in Hadoop?
Older explanations often name JobTracker as the coordinator. That describes Hadoop's older architecture, not execution on YARN. For a MapReduce application running on YARN, the current component roles are:
- ResourceManager: allocates cluster resources among applications.
- ApplicationMaster: negotiates resources for the application and coordinates task execution and monitoring.
- NodeManager: manages execution containers and monitors resource use on a worker machine.
The coordinator doesn't personally execute every map function. Worker tasks do the processing. Our Hadoop YARN introduction covers the architecture in more detail.
Data locality reduces input transfer, not every transfer
Data locality means placing computation near the data it reads. When compute workers and HDFS storage share machines, scheduling a map task near an input replica can reduce network reads. The HDFS architecture guide explains this move-computation-to-data approach.
Where a combiner helps
A combiner optionally aggregates intermediate records locally, reducing what must be shuffled. For example, repeated (Hadoop, 1) records from one mapper can become (Hadoop, partialCount) before reaching the reducer.
Integer addition works because regrouping or reordering the additions doesn't change the result. A combiner isn't a replacement for the reducer: partial counts from different mappers still need to be combined. The original MapReduce paper explains this partial-aggregation optimization.
In our tiny example, each mapper sees each word only once, so a combiner wouldn't reduce the number of emitted records. Treat it as an optimization, not a correctness requirement.
When should you use Hadoop MapReduce?
Consider it for large, finite batch jobs—such as scheduled aggregations—when its execution model and your existing Hadoop infrastructure fit the workload. Don't assume that distributing work makes it faster.
Job startup has a cost. Hadoop also materializes intermediate data—writes it out for later consumption—and performs disk I/O during shuffle and merging. Chaining jobs for an iterative algorithm adds repeated output writes and input reads, making interactive queries and repeated passes over the same working data awkward fits.
Runtime depends on the workload, input format, storage, cluster resources, intermediate-data volume, and key distribution. Benchmark representative data against a simpler single-machine approach before paying the operational cost of a cluster.
MapReduce, Spark, and Flink use different execution models
Hadoop MapReduce remains available, but it isn't the only distributed processing engine to evaluate:
| Engine | Execution model | Selection consideration |
|---|---|---|
| Hadoop MapReduce | Batch jobs organized around map tasks, shuffle/sort, and optional reduce tasks. | A possible fit for established Hadoop batch pipelines; repeated job boundaries add overhead. |
| Apache Spark | Composes transformations into computation stages and can cache reusable datasets in memory. | Evaluate for multi-step and iterative processing; shuffle and disk I/O still matter. |
| Apache Flink | Processes bounded and unbounded streams with stateful operators. | Evaluate when continuous processing is central; bounded input also supports batch execution. |
None is universally faster. Spark isn't simply MapReduce with every disk access removed; its shuffle can involve network and disk I/O. Our Hadoop MapReduce vs Spark comparison expands on the distinction.
Google Cloud Dataflow is another distinct case: it executes pipelines defined using Apache Beam's batch and streaming programming model. Shared ideas such as grouping by key don't make it simply a Hadoop MapReduce implementation.
Common Mistakes
- Grouping by value: shuffle groups records by key, even when their values differ.
- Ignoring hot keys: if one key accounts for a disproportionate amount of work, its reducer can become a bottleneck. Adding reducers alone doesn't split that key's group.
- Assuming locality eliminates networking: local input reads don't eliminate shuffle traffic.
- Confusing frequencies with a total: word-count output gives one count per word; a single overall count is a different aggregation.
FAQ
What are the map, shuffle, and reduce phases?
Map emits intermediate key/value pairs. Shuffle partitions, transfers, sorts, and groups those records by key; reduce processes each key's values to produce output.
How does shuffle group records in MapReduce?
Shuffle brings values with the same intermediate key to the same reducer. For example, two records containing (Hadoop, 1) become the logical reducer input (Hadoop, [1, 1]).
Is Hadoop MapReduce still used?
Hadoop MapReduce remains available as a batch-processing framework. Consider it for existing Hadoop pipelines, but evaluate the workload rather than assuming it should be the default for a new system.
What is the difference between MapReduce and Apache Spark?
Hadoop MapReduce organizes processing into map and reduce jobs with intermediate materialization. Spark supports multi-stage computations and reusable in-memory datasets, but it can still incur shuffle and disk costs.
When is MapReduce a poor choice?
MapReduce is often a poor fit for small jobs, low-latency interactive queries, and iterative algorithms. Startup, shuffle, and repeated disk I/O can outweigh the benefit of parallel execution.
Further Reading
- MapReduce: Simplified Data Processing on Large Clusters — the original model and its design motivations.
- Hadoop MapReduce Tutorial — WordCount, task execution, and job behavior.
- Hadoop Mapper contract — input splits and intermediate records.
- Hadoop Reducer contract — shuffle, sorting, and grouped values.
- Hadoop YARN architecture — resource management and application coordination.
- Spark RDD Programming Guide — distributed transformations, persistence, and shuffle costs.
- Flink architecture — bounded and unbounded processing.
- Apache Beam programming model on Dataflow — pipelines beyond the MapReduce model.
