Hadoop Getting Started With HDFS

Key Takeaways

• HDFS splits files into blocks stored across DataNodes; the NameNode manages names, permissions, and block-location metadata.
• Clients transfer file data directly with DataNodes, not through the NameNode.
• Format only a new, empty NameNode namespace. Starting an existing filesystem does not require formatting.
• Use HDFS commands to list directories, upload local files, read their contents, and remove your test data.

HDFS, the Hadoop Distributed File System, stores files across a cluster rather than on one machine. Once your cluster is configured and running, use hdfs dfs to perform filesystem operations such as listing, uploading, and reading files. The walkthrough below explains the storage components first, then takes you through a complete file-operation exercise.

HDFS Components: NameNode, DataNodes, and Blocks

NameNode: the filesystem's metadata service

The NameNode manages the filesystem namespace—the directory tree, filenames, permissions, and mapping between files and blocks. It also tracks which DataNodes hold those blocks. Your client contacts it for metadata and permission checks, but file contents don't flow through it.

A basic learning cluster has one NameNode. For production availability, HDFS supports an Active NameNode and one or more Standby NameNodes that maintain state for failover. HDFS federation addresses a different concern: scaling through independent namespaces that share DataNode storage.

DataNode: storage and data transfer

DataNodes store block data on their local filesystems and serve client reads and writes. They send heartbeats to report that they're alive and block reports to describe the blocks they hold; the NameNode uses that information to manage placement and replication.

The simplified client data path looks like this:

Client <---- metadata / block locations ----> NameNode

Read:   DataNode --------------------------> Client
Write:  Client ----> DataNode ----> DataNode ----> DataNode
                              replication pipeline

For a replicated write, the client sends data to the first DataNode in a pipeline, which forwards it to the next. For a read, the client obtains block locations and reads from DataNodes directly. The diagram illustrates a three-replica write; your cluster may use a different replication factor.

Blocks: size and replication are different settings

In the replicated-file layout used here, HDFS divides a file into blocks and stores copies of those blocks across DataNodes. Block size determines the size of each chunk; replication factor determines how many copies HDFS targets.

These are the upstream configuration defaults, not guaranteed values for your installation. Apache's single-node tutorial sets replication to 1, because one DataNode cannot provide copies on separate DataNodes.

To change the default block size, set dfs.blocksize in the hdfs-site.xml loaded by your client before creating files. The value is measured in bytes; changing it affects the default for new files, not the blocks of existing files.

Where HDFS fits

HDFS favors high-throughput, streaming access to large files, not low-latency access to individual records. Large numbers of small files also put pressure on the NameNode because namespace and block metadata occupy memory. Distributing data supports parallel processing; it doesn't make every individual operation faster. See our introduction to Apache Hadoop architecture for the surrounding components.

Before Running HDFS Commands

This tutorial assumes a configured Linux learning installation, with Java available and the Hadoop service account ready. If you haven't reached that point, complete the Hadoop environment setup and Hadoop configuration first. Apache's single-node setup guide is the authoritative reference for configuration prerequisites.

From the root of your configured Hadoop installation, prepare the shell. Keep an existing HADOOP_CONF_DIR if your installation uses a separate configuration directory:

export HADOOP_HOME="$PWD"
export HADOOP_CONF_DIR="${HADOOP_CONF_DIR:-$HADOOP_HOME/etc/hadoop}"
export PATH="$HADOOP_HOME/bin:$PATH"

hdfs getconf -confKey fs.defaultFS

Confirm that fs.defaultFS points to the intended HDFS namespace. An unqualified path such as /test uses the configured default filesystem; with file:///, it addresses the local filesystem instead. The examples below require HDFS as the default.

For script-based startup, configure the NameNode and DataNode storage locations and list worker hosts in $HADOOP_CONF_DIR/workers. The helper scripts require working SSH access to their target hosts; the single-node setup includes access to localhost. Run service commands as the configured Hadoop service account.

Starting HDFS Safely

Initialize only a new, empty namespace

Only when you're intentionally initializing a new, empty learning namespace, run:

# ONE-TIME INITIALIZATION ONLY: new, empty namespace.
hdfs namenode -format

This initializes the NameNode's namespace storage. It is not part of subsequent starts.

Start the configured cluster

For an initialized cluster, skip formatting and run the startup script from your configured installation:

"$HADOOP_HOME/sbin/start-dfs.sh"

The script starts the configured HDFS daemons, including the NameNode and DataNodes. In a non-HA setup, it can also start a configured SecondaryNameNode for metadata checkpointing; that process isn't a Standby NameNode. For a managed or shared cluster, use its established service-management procedure instead.

Check availability before uploading

hdfs dfsadmin -report
hdfs dfsadmin -safemode get
hdfs dfs -ls /

The report shows filesystem statistics and DataNode information. Check that your expected DataNodes are live, then confirm that listing the root succeeds. Safe mode is a read-only NameNode state: a successful listing alone doesn't establish that writes are available.

If these checks fail, resolve configuration, connectivity, authentication, or daemon problems before continuing. Don't reformat the NameNode to fix a connection error.

HDFS Basic Commands: List, Create, and Upload

Run this exercise in your own learning cluster, with permission to create /test. That directory must not already exist. On a shared cluster, ask for a writable directory and replace /test consistently rather than changing root permissions.

List the HDFS root and create a directory

hdfs dfs -ls /
hdfs dfs -mkdir /test

-ls / lists the entries immediately under the HDFS root; -mkdir /test creates the test directory there. These aren't your machine's local root and local directory.

Upload a local file to HDFS

Choose an empty, writable local working directory, then create and upload a small text file:

# Local filesystem: create the source file.
printf 'Hello, HDFS!\n' > testfile.txt

# HDFS: upload it, then inspect the destination.
hdfs dfs -put testfile.txt /test
hdfs dfs -ls /test

-put reads testfile.txt from your local working directory and copies it into HDFS as /test/testfile.txt. It doesn't move or delete the local source. By default, uploading fails if the destination file already exists; -f explicitly allows overwriting, so don't add it automatically.

Reading Data from HDFS

Use -cat to send the HDFS file's contents to your terminal:

hdfs dfs -cat /test/testfile.txt

For the file you just created, the output is:

Hello, HDFS!

Unlike a local cat testfile.txt, this command reads the HDFS copy. Keep -cat for small files during exploration; printing a large dataset to your terminal isn't a useful verification strategy.

Other Useful HDFS Commands

Find help for your installed commands

Use explicit help rather than treating a bare hdfs dfs invocation as a complete reference:

hdfs dfs -help
hdfs dfs -help ls
hdfs dfs -help put

The first command displays command help; the others describe individual operations. Notice the order: -help ls, not -ls -help. Consult Apache's File System Shell guide for the broader reference.

Inspect, rename, and clean up the test file

# Display the last 1 KiB, or the whole file if it's smaller.
hdfs dfs -tail /test/testfile.txt

# Rename the HDFS file, then inspect the directory.
hdfs dfs -mv /test/testfile.txt /test/renamed.txt
hdfs dfs -ls /test

# Delete only the test file, then remove the empty directory.
hdfs dfs -rm /test/renamed.txt
hdfs dfs -rmdir /test

-tail is byte-based, not a request for the last ten lines. -mv moves or renames within a filesystem; it doesn't download a file. -rmdir requires an empty directory. Don't assume deletion is recoverable: Trash behavior depends on configuration.

Stopping HDFS

When you're finished with your own learning cluster, use the same installation, configuration, and service account you used to start it:

"$HADOOP_HOME/sbin/stop-dfs.sh"

This stops the configured HDFS daemons, including the NameNode, DataNodes, and configured checkpointing processes. It doesn't stop YARN. Stop or coordinate dependent workloads before shutting down shared storage; don't run this against a shared cluster without authorization. Apache's cluster operation guide documents startup and shutdown responsibilities.

Common Mistakes

• Formatting on every restart: keep initialization separate from normal service startup.
• Confusing local paths with HDFS paths: verify your default filesystem and remember that -put takes a local source.
• Forcing writes during safe mode: investigate missing DataNodes or other startup problems instead of blindly forcing the NameNode out of safe mode.
• Using a tutorial configuration in production: keep the learning cluster private. Production needs authenticated access, appropriate network restrictions, tested recovery procedures, and an availability design.

Once these storage operations are familiar, our quick explanation of MapReduce introduces the processing model.

FAQ

How do I upload a local file to HDFS?

Run hdfs dfs -put testfile.txt /test after creating the HDFS destination directory. The source is a local file; the destination is in HDFS when your default filesystem points to HDFS.

How do I list files and directories in HDFS?

Run hdfs dfs -ls / to list the HDFS root, or replace / with a directory such as /test. This lists that directory's immediate entries rather than recursively traversing its contents.

What is the difference between a NameNode and a DataNode?

The NameNode manages filenames, permissions, and block-location metadata. DataNodes store block data and serve client reads and writes; file contents don't pass through the NameNode.

What is the default HDFS block size, and how do I change it?

The upstream default is 134217728 bytes, or 128 MiB. Configure dfs.blocksize in the client-loaded hdfs-site.xml to change the default for new files; changing it doesn't resize existing blocks.

Should I format the NameNode every time I start HDFS?

No. Formatting is only for initializing a new, empty namespace, not normal startup. Formatting an existing NameNode can replace metadata needed to access its existing files.

Further Reading

• Apache Hadoop single-node setup — configuration, SSH prerequisites, and the learning-cluster lifecycle.
• HDFS architecture — metadata, blocks, replication, and streaming access.
• File System Shell reference — filesystem commands and their options.
• HDFS Commands Guide — administrative commands, configuration inspection, and safe mode.
• Hadoop cluster setup and operation — service accounts, worker configuration, and lifecycle scripts.
• HDFS high availability with QJM — Active/Standby NameNodes and failover.
• HDFS federation — independent namespaces sharing DataNode storage.
• HDFS configuration defaults — definitions of block-size and replication properties.