Hadoop Configuration

Sam Erickson
Updated 20 hours ago
0

Key Takeaways

Pseudo-distributed Hadoop runs separate daemons on one machine. It’s useful for learning configuration, not testing multi-host resilience.
Configure core-site.xml, hdfs-site.xml, yarn-site.xml, and mapred-site.xml, plus the daemon environment in hadoop-env.sh.
This walkthrough uses Hadoop 3.5.0 on GNU/Linux with JDK 17, the Java version required by this release’s servers.
Format a new NameNode once, start HDFS before YARN, and verify the cluster with service checks and a MapReduce job—not another format.

Hadoop pseudo-distributed mode configuration puts HDFS storage and YARN resource management on a single Linux host, with each daemon running in its own Java process. You’ll edit four XML files, prepare writable storage directories, initialize HDFS once, and run a small job to check the complete setup.

Hadoop Operating Modes

Mode	How it runs	Useful for
Standalone	The default, non-distributed setup runs as one Java process and uses the local filesystem.	Debugging MapReduce logic without running cluster daemons.
Pseudo-distributed	Separate Hadoop daemons run on one host.	Learning HDFS, YARN, configuration, and job submission with little hardware.
Fully distributed	Hadoop services and workers are deployed across multiple hosts.	Workloads that need a multi-host cluster.

Pseudo-distributed mode exercises daemon communication, but all processes still share one machine’s CPU, memory, disks, and failure domain. Fully distributed describes a topology; it doesn’t make a deployment production-ready by itself.

Prepare the Hadoop User and Java Environment

This walkthrough targets Apache Hadoop 3.5.0, GNU/Linux, Bash, and JDK 17. Hadoop 3.5.0 requires Java 17 on servers; don’t interpret Java compatibility as an open-ended “Java 11 and above” range. Check the release overview and Hadoop Java Versions guidance when using another release.

Assume you’ve unpacked Hadoop at /usr/local/hadoop and created a dedicated hadoop account with home directory /home/hadoop. That account must be able to edit /usr/local/hadoop/etc/hadoop. If installation is unfinished, start with Hadoop Environment Setup, but use the release and Java requirements above.

Open the service user’s shell first

From an administrator account with suitable sudo permissions, run this command by itself. Run all subsequent setup and daemon commands inside the resulting hadoop shell.

sudo -iu hadoop

Append the following environment settings once, then load them into the current shell. Explicitly sourcing the file avoids depending on whether your login profile loads ~/.bashrc.

cat >> ~/.bashrc <<'EOF'
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"
export PATH="$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH"
EOF
source ~/.bashrc
cd "$HADOOP_HOME"

Create and inspect the local storage directories

Create these directories as hadoop, not root. The NameNode stores filesystem metadata locally; the DataNode stores HDFS block data locally. Those are different from directories you’ll later create inside HDFS.

mkdir -p /home/hadoop/hdfs/namenode /home/hadoop/hdfs/datanode \
  /home/hadoop/hadoop-logs /home/hadoop/hadoop-pids
chmod 700 /home/hadoop/hdfs/namenode /home/hadoop/hdfs/datanode \
  /home/hadoop/hadoop-logs /home/hadoop/hadoop-pids
ls -ld /home/hadoop/hdfs/namenode /home/hadoop/hdfs/datanode \
  /home/hadoop/hadoop-logs /home/hadoop/hadoop-pids

Check that the owner is hadoop and that the account has read, write, and directory-search permissions. If existing directories are owned by another account, have an administrator repair ownership on these dedicated lab paths before continuing. Don’t work around ownership errors with world-writable permissions or root-run daemons.

Set JAVA_HOME in hadoop-env.sh

First put your JDK 17 installation’s bin directory on PATH. The following resolves the selected javac executable to its installation directory; inspect the Java version and stop if it isn’t 17.

export JAVA_HOME="$(dirname "$(dirname "$(readlink -f "$(command -v javac)")")")"
"$JAVA_HOME/bin/java" -version

For this fresh lab, append the resolved paths to hadoop-env.sh. Persisting them there gives Hadoop’s daemon scripts their Java, MapReduce, log, and PID settings without relying on an interactive shell’s environment.

cat >> "$HADOOP_CONF_DIR/hadoop-env.sh" <<EOF
export JAVA_HOME="$JAVA_HOME"
export HADOOP_MAPRED_HOME="$HADOOP_HOME"
export HADOOP_LOG_DIR="/home/hadoop/hadoop-logs"
export HADOOP_PID_DIR="/home/hadoop/hadoop-pids"
export PDSH_RCMD_TYPE=ssh
EOF
bin/hadoop version
bin/hdfs envvars

Confirm that Hadoop reports 3.5.0 and that the computed environment contains your intended paths. The PDSH_RCMD_TYPE setting selects SSH if the helper scripts use an installed pdsh utility.

Edit the Hadoop Configuration Files

Use $HADOOP_CONF_DIR for all four files below. For this fresh installation, replace each file’s contents with its corresponding XML document; don’t nest a second configuration element inside an existing one.

core-site.xml: Choose the default filesystem

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

fs.defaultFS supplies the default filesystem URI for Hadoop clients. Here, localhost:9000 is the NameNode’s RPC endpoint—the address used for client requests—not a browser address. Clients must load the same configuration; a client using the default file:/// filesystem operates locally instead.

hdfs-site.xml: Set local storage and lab replication

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///home/hadoop/hdfs/namenode</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///home/hadoop/hdfs/datanode</value>
  </property>
  <property>
    <name>dfs.namenode.http-address</name>
    <value>localhost:9870</value>
  </property>
  <property>
    <name>dfs.namenode.secondary.http-address</name>
    <value>localhost:9868</value>
  </property>
</configuration>

The file:/// values identify storage on the daemon’s local host. Creating writable directories is a separate preparation step; writing a path into XML doesn’t establish correct ownership. A replication factor of 1 suits this single-DataNode lab but provides no replica redundancy.

The two HTTP settings keep the NameNode and SecondaryNameNode web endpoints on localhost. The explicit SecondaryNameNode hostname also gives the startup script a localhost target. The SecondaryNameNode creates metadata checkpoints; it isn’t a standby NameNode.

yarn-site.xml: Configure resource management and shuffle

<configuration>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
  </property>
</configuration>

The ResourceManager allocates cluster resources. The NodeManager manages execution containers on this host; a container is an allocation of resources for an application process. The mapreduce_shuffle auxiliary service serves map outputs to reducers.

The environment allowlist includes HADOOP_MAPRED_HOME, matching the release’s single-node guidance. For the architecture behind these processes, see Hadoop YARN Introduction.

mapred-site.xml: Run MapReduce on YARN

Create mapred-site.xml directly if it’s missing; this walkthrough doesn’t depend on a template-copy command. Set the execution framework and the MapReduce application classpath as follows.

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
</configuration>

mapreduce.framework.name=yarn selects YARN execution. The classpath points application containers at the MapReduce libraries; leave the $HADOOP_MAPRED_HOME references literal in the XML.

Configure Localhost SSH for the Startup Scripts

The multi-daemon startup scripts use remote execution even for a single-host cluster. Install an SSH client and server, ensure sshd is running, and put only localhost in the workers file.

printf 'localhost\n' > "$HADOOP_CONF_DIR/workers"
ssh localhost true

Verify the host fingerprint before accepting a first connection. If passwordless authentication already works, skip key creation. For a fresh lab account without an existing Ed25519 key, use the following; never overwrite an existing private key.

mkdir -p ~/.ssh
chmod 700 ~/.ssh
ssh-keygen -t ed25519 -N '' -f ~/.ssh/id_ed25519
cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
ssh localhost true
ssh -o BatchMode=yes localhost true

The final command must exit successfully without a password or passphrase prompt. A passphrase-free key is a convenience for this isolated lab; protect the account and private key.

Initialize HDFS Once, Then Start the Cluster

This is the single-host topology you’re bringing up. Port 9000 is the RPC choice in this tutorial; the browser interfaces use different ports.

One Linux host
  Hadoop client --RPC :9000--> NameNode --metadata--> DataNode
  Browser       --HTTP :9870-> NameNode UI
  Browser       --HTTP :8088-> ResourceManager UI

  ResourceManager <--resources/status--> NodeManager
                                         |
                                  MapReduce containers

  SecondaryNameNode: metadata checkpoints

Format only a new, empty namespace

cd "$HADOOP_HOME"
# First initialization of this new lab namespace only.
bin/hdfs namenode -format

Start HDFS and create the HDFS user directory

sbin/start-dfs.sh
bin/hdfs dfsadmin -report
bin/hdfs dfsadmin -safemode get

Proceed when the report lists one live DataNode and safe mode is off. If either check fails, investigate before submitting jobs. Then create the user’s directory inside HDFS.

bin/hdfs dfs -mkdir -p /user/hadoop

Start YARN and verify its worker

sbin/start-yarn.sh
bin/yarn node -list
"$JAVA_HOME/bin/jps" -l

Expect one running YARN node. The JVM listing should include NameNode, DataNode, SecondaryNameNode, ResourceManager, and NodeManager classes. Process presence is only an initial check: it doesn’t prove that storage or job execution works.

Verify HDFS and MapReduce End to End

Run this first smoke test against the new cluster. It uploads a tiny local file into HDFS and runs the bundled WordCount example through YARN.

printf 'hadoop yarn\nhadoop hdfs\n' > /home/hadoop/smoke.txt
bin/hdfs dfs -mkdir -p /user/hadoop/config-smoke/input
bin/hdfs dfs -put /home/hadoop/smoke.txt /user/hadoop/config-smoke/input/
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar \
  wordcount /user/hadoop/config-smoke/input /user/hadoop/config-smoke/output \
  && bin/hdfs dfs -cat '/user/hadoop/config-smoke/output/part-*'

Expected word counts are:

hadoop  2
hdfs    1
yarn    1

For another run, choose fresh input and output directories rather than formatting HDFS or deleting storage. MapReduce’s standard file output checks reject an existing output directory. Continue with our Hadoop MapReduce example or Apache’s MapReduce tutorial to understand the job itself.

Check the web interfaces

NameNode: Open http://localhost:9870/ on the Hadoop host to inspect filesystem capacity and live DataNodes. Hadoop 3 uses port 9870 by default; dfs.namenode.http-address can change it.
ResourceManager: Open http://localhost:8088/ to inspect registered nodes and application state. The WordCount application should finish successfully.

If Hadoop runs in a VM, those localhost addresses refer to the VM, not automatically to your laptop. Don’t expose an unauthenticated cluster publicly just to reach its UI.

Stop and restart without reformatting

cd "$HADOOP_HOME"
sbin/stop-yarn.sh
sbin/stop-dfs.sh

For subsequent sessions, start HDFS, check its health, and start YARN again. Skip the formatting step. To explore the filesystem next, follow Hadoop Getting Started With HDFS.

Common Mistakes and Failure Checks

Start with the daemon logs and listening sockets instead of repeatedly rerunning startup scripts. This tutorial explicitly places daemon logs in /home/hadoop/hadoop-logs; Hadoop’s normal default is $HADOOP_HOME/logs.

tail -n 80 /home/hadoop/hadoop-logs/*.log
ss -ltnp
getent hosts localhost
getent hosts "$(hostname)"

Permission denied: Check the configured local storage, log, and PID directories, including access through their parent directories. Repair ownership rather than running the cluster as root.
Java mismatch or immediate exit: Compare bin/hdfs envvars with the JDK selected earlier. Inspect daemon .out files as well as .log files for launch failures.
Address already in use: Use ss -ltnp to identify a conflicting listener. Stop the unintended process or deliberately change the relevant address in configuration.
SSH or hostname failure: Recheck batch-mode localhost SSH, the workers file, and local hostname resolution before investigating HDFS itself.
Already-running message after a crash: Inspect the PID file in /home/hadoop/hadoop-pids and confirm whether the corresponding daemon still exists. Remove a stale PID file only after confirming there’s no matching running daemon.
Job accepted but not progressing: Check registered YARN nodes, available resources, and application diagnostics. Leave memory for the operating system and Hadoop daemons as well as application containers; don’t treat a tiny VM as a production sizing reference.

Moving Beyond a Single Host

Moving to fully distributed mode means more than adding workers. Replace localhost service addresses with resolvable hostnames, distribute consistent configuration, list worker hosts, prepare storage on each host, and plan replication and failure recovery. Follow the cluster setup guide and secure-mode documentation rather than extending this unauthenticated lab unchanged.

There’s no universal CPU, RAM, disk, or node-count minimum that establishes production readiness. Size the cluster for its data volume, replication, concurrent jobs, service placement, and availability requirements—not a generic per-node hardware figure.

FAQ

What is the difference between standalone and pseudo-distributed Hadoop mode?

Standalone mode runs Hadoop as one Java process using the local filesystem. Pseudo-distributed mode runs separate daemons on one host, allowing you to work with HDFS and YARN without a multi-host cluster.

Which Hadoop configuration files are required for this pseudo-distributed setup?

This HDFS-and-YARN walkthrough configures core-site.xml, hdfs-site.xml, yarn-site.xml, and mapred-site.xml. It also sets daemon environment variables in hadoop-env.sh and puts localhost in the workers file for the startup scripts.

What is the default NameNode web UI port in Hadoop 3?

The default NameNode HTTP web UI port is 9870. You can change it with dfs.namenode.http-address; it is separate from the NameNode RPC port configured for clients.

Do I need SSH configured to start Hadoop daemons?

You need working SSH authentication when using the multi-daemon start-dfs.sh and start-yarn.sh helper scripts, including localhost authentication for this lab. SSH is not required when you start each daemon directly on its local host.

Is it safe to run hdfs namenode -format more than once?

Not if you want to preserve the existing namespace. Formatting is an initialization operation that can replace NameNode metadata; use filesystem commands, service reports, and logs for routine verification.