Hadoop Environment Setup

Sam Erickson
Updated 18 hours ago
0

Key Takeaways

Install Java before Hadoop, and choose the JDK supported by your Hadoop release—not simply the newest JDK.
Use a dedicated, non-root Hadoop account and explicitly create writable installation, data, log, and process-ID directories.
Verify the Apache archive before extracting it, then test a real job rather than stopping at hadoop version.
Choose your deployment mode: standalone needs no SSH or HDFS daemons; a pseudo-distributed installation runs separate services on one machine.

To install Apache Hadoop on Linux, create a dedicated account, install a compatible Java runtime, download and verify Hadoop, and configure its environment. Installing the binaries is only the first stage: running HDFS or YARN requires additional configuration and service startup.

This guide uses Hadoop 3.5.0, the latest stable release listed by Apache as of October 17, 2026, and Ubuntu 24.04 LTS on an x86-64 machine. Hadoop 3.5.0 was released on April 2, 2026. If you use another operating system, a Linux VM is a straightforward way to follow the same steps.

Choose Your Hadoop Deployment Mode

HDFS is Hadoop's distributed filesystem; YARN manages resources and launches application workloads. Review Apache Hadoop architecture if those roles are new to you.

Mode	What runs	SSH requirement	Coverage here
Standalone	One Java process using local files	None	A complete local smoke test
Pseudo-distributed	Separate Hadoop daemons on one machine	Optional for direct local startup; used by SSH-based helper scripts	Working HDFS and optional YARN
Fully distributed	Services spread across multiple hosts	Required if you use the SSH-based helper scripts	Next steps, not a production deployment recipe

The distinction matters: the local daemon commands below let you learn HDFS and YARN without first configuring SSH. Apache documents both direct daemon startup and the SSH-based helper scripts.

Prepare Linux and a Dedicated Hadoop User

Start with a fresh Ubuntu VM, Bash, an administrator account with sudo access, and outbound HTTPS access to Ubuntu repositories and Apache downloads. Treat this as a disposable, single-user learning environment.

A dedicated account limits which local files Hadoop processes can modify. It is not a sandbox: the account still has access to anything its Linux permissions allow. See Linux file permissions and user management for the ownership model.

Run these commands from your administrator account. Follow the adduser prompts to create the account; don't grant it administrator privileges. Ubuntu's user-management documentation explains this account-creation workflow.

sudo adduser hadoop
sudo install -d -o hadoop -g hadoop -m 0750 \
  /opt/hadoop /var/lib/hadoop \
  /var/lib/hadoop/name /var/lib/hadoop/data \
  /var/lib/hadoop/checkpoint /var/lib/hadoop/run \
  /var/log/hadoop

These directories are owned by hadoop:hadoop; mode 0750 gives the owner full access, the group read/traverse access, and everyone else no access. Allowing this account to own the installation simplifies the lab; use administrator-controlled binaries for a production deployment.

Install a Compatible Java Runtime

Hadoop 3.5.0 requires Java 17 for servers; its clients support Java 17 and Java 21. Because this machine runs both clients and daemons, use Java 17 throughout. Older advice that any newer JDK is generally safe is incorrect: compatibility is release-specific.

Ubuntu 24.04 provides the OpenJDK 17 JDK package. Install it from your administrator account, then check the specific runtime this guide will use.

sudo apt-get update
sudo apt-get install -y openjdk-17-jdk curl ca-certificates
/usr/lib/jvm/java-17-openjdk-amd64/bin/java -version
sudo -iu hadoop

The Java command should report version 17. After the final command, you are in the Hadoop user's shell; run the remaining installation and Hadoop commands there unless a step explicitly says otherwise.

Download, Verify, and Install Hadoop

Download the Version-Specific Binary Archive

Start at the Apache Hadoop download page, not an old stable2 URL. The following commands use the published x86-64 binary archive in the Hadoop 3.5.0 release directory. For ARM machines, select Apache's corresponding architecture-specific archive and adjust the JDK path.

mkdir -p "$HOME/downloads"
cd "$HOME/downloads"
base=https://downloads.apache.org/hadoop/common/hadoop-3.5.0
curl -fLO "$base/hadoop-3.5.0.tar.gz" &&
curl -fLO "$base/hadoop-3.5.0.tar.gz.sha512" &&
sha512sum --check hadoop-3.5.0.tar.gz.sha512 &&
tar -xzf hadoop-3.5.0.tar.gz -C /opt/hadoop

Continue only if verification reports hadoop-3.5.0.tar.gz: OK. The command chain prevents extraction when a download or checksum check fails. Apache publishes the expected digest in its SHA-512 checksum file; GNU's checksum utilities support checking it.

A matching checksum verifies that the archive matches the published digest; it does not independently establish the publisher's identity. Apache also provides signatures and verification instructions on its download page.

Set JAVA_HOME, HADOOP_HOME, and PATH

Create one reusable shell-environment file. The quoted here-document preserves variable references when writing the file, and the loop avoids adding duplicate PATH entries when you source it again.

cat > "$HOME/.hadoop-env" <<'EOF'
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export HADOOP_HOME=/opt/hadoop/hadoop-3.5.0
export HADOOP_CONF_DIR="$HADOOP_HOME/etc/hadoop"
for dir in "$JAVA_HOME/bin" "$HADOOP_HOME/bin" "$HADOOP_HOME/sbin"; do
  case ":$PATH:" in
    *":$dir:"*) ;;
    *) PATH="$dir:$PATH" ;;
  esac
done
export PATH
unset dir
EOF
. "$HOME/.hadoop-env"
java -version
hadoop version

Add the following line once to the Hadoop user's ~/.bashrc for future interactive Bash shells:

# Fragment to add once to ~/.bashrc
. "$HOME/.hadoop-env"

Also edit $HADOOP_CONF_DIR/hadoop-env.sh, setting or replacing these assignments rather than repeatedly appending them:

# Fragment for etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.5.0
export HADOOP_LOG_DIR=/var/log/hadoop
export HADOOP_PID_DIR=/var/lib/hadoop/run

Hadoop's shell commands load hadoop-env.sh, so this configures the daemon-launch environment rather than relying on an interactive .bashrc. If you later use systemd or another supervisor, explicitly configure its account, executable paths, and environment too.

Verify Hadoop in Standalone Mode

Before changing the XML configuration, run this job against local files. Hadoop's default standalone mode needs neither HDFS services nor SSH.

work="$(mktemp -d "$HOME/hadoop-local.XXXXXX")"
printf 'hadoop hadoop linux\n' > "$work/input.txt"
hadoop jar \
  "$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar" \
  wordcount "$work/input.txt" "$work/output"
cat "$work/output/part-r-00000"

The WordCount example counts occurrences of each word. For this input, the result should contain hadoop with count 2 and linux with count 1. This exercises a job, whereas hadoop version only reports installation information.

Configure Pseudo-Distributed HDFS

Set the Filesystem and Storage Directories

Replace the contents of $HADOOP_CONF_DIR/core-site.xml with:

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

Replace $HADOOP_CONF_DIR/hdfs-site.xml with:

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
  <property>
    <name>dfs.namenode.name.dir</name>
    <value>file:///var/lib/hadoop/name</value>
  </property>
  <property>
    <name>dfs.datanode.data.dir</name>
    <value>file:///var/lib/hadoop/data</value>
  </property>
  <property>
    <name>dfs.namenode.checkpoint.dir</name>
    <value>file:///var/lib/hadoop/checkpoint</value>
  </property>
</configuration>

The NameNode stores filesystem metadata, while the DataNode stores file blocks. These settings put their state in explicit persistent directories; replication 1 is appropriate for this one-DataNode lab but provides no redundant block copy. The checkpoint directory is used by the SecondaryNameNode.

Format Once, Then Start the Services

# Initialization only: do not repeat on an existing filesystem.
hdfs namenode -format

Start the local daemons directly. The SecondaryNameNode creates metadata checkpoints; it is not a standby NameNode.

hdfs --daemon start namenode
hdfs --daemon start datanode
hdfs --daemon start secondarynamenode
hdfs dfsadmin -report

Wait until the report shows one live DataNode, then test an HDFS write and read. The -safemode wait command waits for the NameNode to leave its protective safe mode before you write.

hdfs dfsadmin -safemode wait
printf 'hadoop hadoop linux\n' > "$HOME/hadoop-smoke.txt"
hdfs dfs -mkdir -p /user/hadoop/input
hdfs dfs -put "$HOME/hadoop-smoke.txt" /user/hadoop/input/
hdfs dfs -cat /user/hadoop/input/hadoop-smoke.txt

The final command should return the original line. These filesystem-shell operations verify actual storage access. Continue with practical HDFS commands or explore Hadoop pseudo-distributed mode configuration. The default NameNode web-interface port is 9870; open it inside the VM.

Add YARN and Run a Cluster Job

HDFS alone does not make MapReduce run on YARN. Configure the execution framework and NodeManager shuffle service—the component that serves intermediate map output to reducers.

Replace $HADOOP_CONF_DIR/mapred-site.xml with:

<configuration>
  <property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
  </property>
  <property>
    <name>mapreduce.application.classpath</name>
    <value>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*</value>
  </property>
</configuration>

Replace $HADOOP_CONF_DIR/yarn-site.xml with:

<configuration>
  <property>
    <name>yarn.resourcemanager.hostname</name>
    <value>localhost</value>
  </property>
  <property>
    <name>yarn.nodemanager.aux-services</name>
    <value>mapreduce_shuffle</value>
  </property>
  <property>
    <name>yarn.nodemanager.env-whitelist</name>
    <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
  </property>
</configuration>

These classpath and environment-whitelist settings follow Apache's single-node YARN configuration. Keep the dollar-prefixed classpath references literal in the XML.

yarn --daemon start resourcemanager
yarn --daemon start nodemanager
yarn node -list -all

Proceed when the node list shows one node in RUNNING state. Submit WordCount using the input already stored in HDFS, with an output directory that does not yet exist.

hadoop jar \
  "$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar" \
  wordcount /user/hadoop/input /user/hadoop/output
hdfs dfs -cat '/user/hadoop/output/part-r-*'

Expect the same counts as the standalone test, now produced by a YARN application reading and writing HDFS. The ResourceManager web interface uses port 8088 by default. If the job remains queued, check node health and available container memory before changing scheduler settings.

Stop the Local Services

When finished, stop the daemons as the same Hadoop user. Skip the YARN commands if you only started HDFS.

yarn --daemon stop nodemanager
yarn --daemon stop resourcemanager
hdfs --daemon stop secondarynamenode
hdfs --daemon stop datanode
hdfs --daemon stop namenode

Optional: Configure Passwordless SSH for Hadoop

You don't need this section for the direct startup commands above. Configure SSH if you want Hadoop's start-dfs.sh and start-yarn.sh helper scripts to launch services across configured hosts. SSH keys authorize an account login; they do not start an SSH server or configure hostname resolution.

In a separate administrator terminal, install the client and server and start the SSH service:

sudo apt-get install -y openssh-client openssh-server
sudo systemctl start ssh.service
sudo ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub

Keep the displayed host-key fingerprint for comparison. Back in the Hadoop user's shell, run the following once on the fresh account; do not overwrite an existing private key. This lab key has an empty passphrase, so protect it and do not reuse it elsewhere.

install -d -m 0700 "$HOME/.ssh"
ssh-keygen -t ed25519 -N '' -f "$HOME/.ssh/id_ed25519"
cat "$HOME/.ssh/id_ed25519.pub" >> "$HOME/.ssh/authorized_keys"
chmod 0600 "$HOME/.ssh/authorized_keys"
ssh localhost true
ssh -o BatchMode=yes localhost 'echo SSH_OK'

For the first connection, verify that the host-key fingerprint matches the administrator's output before accepting it. The final command should print SSH_OK without a password prompt. Do not disable host-key checking to make a failing test pass.

A localhost check doesn't prove that every hostname used by the helper scripts is reachable. For multi-node administration, configure account access, trusted host keys, working name resolution, and SSH-server availability on each relevant host; distribute public keys, not private keys.

Common Mistakes and Multi-Node Next Steps

Permission errors: Check ownership of the specific failing directory. In this lab, installation, storage, log, and PID directories belong to hadoop:hadoop. Don't run the daemons with sudo or use chmod 777 as a workaround.
Startup failures: Inspect /var/log/hadoop, the log directory explicitly configured here. Check Java selection and configuration errors before restarting repeatedly.
Repeated smoke tests: Use a fresh MapReduce output path; an existing output directory is rejected. The HDFS upload also needs a fresh destination or deliberate replacement.
Calling the lab production-ready: One machine and replication 1 provide no host-level redundancy. Plan authentication, network controls, backups, capacity, monitoring, and service supervision before storing important data.

For multiple hosts, install compatible Java and Hadoop on every node, replace localhost addresses with reachable service hostnames, distribute consistent configuration, and list worker hosts in etc/hadoop/workers when using the helper scripts. Apache recommends separate service accounts for HDFS and YARN in larger installations; the single hadoop account here is a learning convenience.

FAQ

Why create a separate user for Hadoop?

A dedicated, non-root account gives Hadoop processes controlled ownership of their installation, data, logs, and PID files. It reduces accidental access to your personal files but does not provide sandbox isolation.

Which Java versions does Hadoop 3.5.0 support?

Hadoop 3.5.0 requires Java 17 for server processes and supports Java 17 and Java 21 for clients. Use Java 17 for this single-node setup because the machine runs both roles.

Do I need SSH to run Hadoop in standalone mode?

No. Standalone mode uses local files without SSH, and you can also start local HDFS and YARN daemons directly without it. SSH is needed when you choose Hadoop's SSH-based multi-host startup helpers.

How do I configure Hadoop for pseudo-distributed mode?

Set the HDFS URI in core-site.xml, configure replication and storage paths in hdfs-site.xml, format a new NameNode once, and start the local HDFS daemons. To execute jobs on YARN, also configure mapred-site.xml and yarn-site.xml and start the ResourceManager and NodeManager.

Does hadoop version confirm that HDFS is working?

No. It reports installation information, not filesystem health. Check the DataNode report, write and read an HDFS file, and run a YARN application if you need to verify distributed job execution.

These answers follow Apache's release, single-node, cluster-setup, and command documentation.