What is Apache Spark?

Sam Erickson
Updated 23 hours ago
0

Key Takeaways

Apache Spark is a distributed processing engine, not a general-purpose storage system. Your data lives in external sources and destinations.
A driver coordinates work, while executors process partitions of data in parallel. Shuffles move data between partitions and can dominate execution time.
Start with DataFrames and Spark SQL for tabular workloads. Java, Scala, Python, and R interfaces exist, but their capabilities and lifecycle status differ.
Spark can replace Hadoop MapReduce workloads without replacing HDFS or YARN. It can also run without a Hadoop cluster.
Structured Streaming defaults to micro-batch execution. Lower-latency modes exist, but latency and delivery guarantees depend on the execution mode and integrations.

Apache Spark is an open-source, distributed data-processing engine for batch analytics, SQL queries, streaming pipelines, and machine learning. It divides computations across multiple processors or machines; it does not provide the durable, general-purpose storage system that holds your source data.

That distinction explains why the “Spark vs. Hadoop” debate is often oversimplified. Spark is an alternative to Hadoop’s MapReduce processing engine, but it can work with Hadoop’s storage and resource-management components—or run without them.

Why Spark?

Spark is useful when a data-processing job benefits from parallel execution across machines. Typical workloads include:

Data preparation: cleaning, joining, and aggregating data before loading it into analytical tables.
Large-scale analytics: querying historical datasets through SQL or DataFrame operations.
Streaming analytics: incrementally processing incoming events with Structured Streaming.
Machine learning: preparing features and running supported distributed algorithms through MLlib.

Spark can avoid some repeated disk I/O between processing steps, and caching can help when computations reuse the same data. That can make it faster than MapReduce on suitable workloads, but there is no universal speedup multiplier: data layout, shuffles, execution plans, and available resources matter. For the narrower engine comparison, see Hadoop MapReduce vs Spark.

When a simpler tool is the better choice

Don't start with a cluster just because your dataset sounds large. If a database can perform the transformation where the data already lives, or a single-machine script meets your requirements, start there.

Evaluate Spark when processing time, memory requirements, or pipeline complexity justify distributed execution. Include startup time, scheduling overhead, infrastructure cost, dependency management, monitoring, and recovery procedures in that decision—not just the time spent inside a transformation.

How Spark Works

Drivers, executors, partitions, and tasks

A Spark application has a driver, which coordinates execution, and executors, which run computations. A cluster manager allocates resources to the application; the driver schedules tasks on its executors.

Data is divided into partitions: chunks that Spark can process independently. A task handles a partition within an execution stage, and a job is divided into stages as dependencies require. The cluster overview explains these relationships.

               Driver: plans work and schedules tasks
                      |                 |
                      v                 v
                 Executor A        Executor B
Stage 1:         partition 0       partition 1
                 read + filter     read + filter
                      \                 /
                       shuffle by key
                      /                 \
Stage 2:         aggregate          aggregate
                      \                 /
                       external storage

This is a simplified aggregation job. The data moves between executors during the shuffle; the driver coordinates that work rather than receiving every input row.

Transformations, actions, and lazy evaluation

A transformation, such as filtering rows, describes a new dataset. An action, such as displaying results or writing output, requests computation. Spark DataFrames use lazy evaluation: transformations build a plan instead of immediately computing their results. Reads can still perform planning work, such as inspecting metadata or inferring a schema.

A shuffle redistributes records between partitions. Grouping, many joins, and global sorting can require one. Shuffles involve network transfer, serialization, and disk I/O, so moving data can cost more than computing on it. Unevenly distributed keys—called data skew—can leave a few tasks doing most of the work.

Memory and fault tolerance

Spark does not require the entire dataset to fit in memory. It uses memory and disk, can spill intermediate data to disk, and lets you explicitly cache reusable results.

For lost computed partitions, Spark can use lineage—the transformations that produced them—to recompute the missing data, provided the necessary inputs remain available. Executor-local cache and shuffle files are not substitutes for durable source data. See the RDD programming guide and our explanation of what an RDD is in Spark.

Spark APIs: DataFrames, SQL, and Languages

A DataFrame represents distributed data organized into named columns. Spark SQL lets you express structured computations with SQL, while the DataFrame API expresses them through programming-language methods. Both use Spark SQL’s optimized execution engine.

Start with these APIs for tabular work. RDDs, or resilient distributed datasets, offer lower-level control over distributed collections but expose less structural information to Spark’s SQL optimizer. The Spark SQL guide explains the distinction.

Spark provides Java, Scala, Python through PySpark, and R through SparkR interfaces. Coverage is not equivalent: the typed Dataset API is available in Scala and Java, not Python or R.

A Small PySpark DataFrame Example

This local example uses PySpark 4.2.0, Python 3.10 or newer, and JDK 17. Set JAVA_HOME to your JDK installation, then install PySpark in your Python environment. The installation guide covers environment requirements.

python -m pip install "pyspark==4.2.0"

Save this as spark_demo.py and run python spark_demo.py. It creates its own tiny CSV input; local[2] runs Spark locally with two worker threads.

from pathlib import Path
from tempfile import TemporaryDirectory

from pyspark.sql import SparkSession
from pyspark.sql import functions as F

spark = (
    SparkSession.builder
    .master("local[2]")
    .appName("SparkIntroduction")
    .getOrCreate()
)

try:
    with TemporaryDirectory() as directory:
        path = Path(directory) / "orders.csv"
        path.write_text(
            "country,amount_cents\nUS,1200\nCA,700\nUS,800\nCA,-100\n",
            encoding="utf-8",
        )
        orders = spark.read.csv(
            str(path),
            schema="country STRING, amount_cents LONG",
            header=True,
            mode="FAILFAST",
        )

        # Transformations build a plan.
        totals = (
            orders.filter(F.col("amount_cents") > 0)
            .groupBy("country")
            .agg(F.sum("amount_cents").alias("total_cents"))
        )

        # This action requests and displays results.
        totals.orderBy("country").show()
finally:
    spark.stop()

The positive amounts total 700 cents for CA and 2,000 cents for US. The explicit CSV schema avoids the separate scan used for schema inference; filtering and aggregation remain planned operations until the action requests results.

This is an API demonstration, not a workload that needs Spark. See the PySpark DataFrame quickstart for more operations.

Apache Spark vs Hadoop: Compare the Right Components

Hadoop is a collection of projects, not just a processing engine. Its module overview separates storage, resource management, and computation:

Component	Role	Relationship to Spark
Apache Spark	Distributed processing engine	Runs batch, SQL, streaming, and other computations.
Hadoop MapReduce	Distributed batch-processing engine	An alternative processing engine; suitable workloads can be rewritten for Spark.
HDFS	Distributed, fault-tolerant filesystem	Can store Spark inputs and outputs.
Hadoop YARN	Cluster resource management and scheduling	Can allocate resources for Spark applications.
Kubernetes	Container orchestration	Another supported cluster manager for Spark; it is not a Hadoop component.

Spark can run with its own standalone cluster manager, on YARN, or on Kubernetes. You can keep HDFS and YARN while replacing MapReduce jobs, or choose a deployment that uses neither. For background, read Apache Hadoop architecture.

External storage and table formats

Spark can access supported filesystems, cloud object stores, databases, and streaming sources through compatible APIs and connectors. “Supported” matters: dependencies, credentials, permissions, and deployment configuration determine what your application can actually read or write. The cloud integration guide also explains why object stores and filesystems have different operational behavior.

Table formats such as Apache Iceberg and Delta Lake integrate with Spark through additional libraries and configuration. Match their compatibility requirements to your Spark runtime; they are not automatically supported merely because Spark can access the underlying files.

Also distinguish a table format from a file format: our guide to what Apache Parquet is explains the column-oriented files commonly used in analytical datasets.

Spark Structured Streaming: Micro-Batches and Lower Latency

Structured Streaming incrementally executes queries over incoming data using DataFrame and Dataset APIs. Its default execution model is micro-batching: processing successive small batches rather than guaranteeing immediate execution for every event. A trigger interval is a scheduling setting, not an end-to-end latency promise.

Micro-batch pipelines can achieve exactly-once results with suitable replayable sources, durable checkpoints, and sinks that handle retries correctly. That is not a blanket guarantee for every connector: Spark’s Kafka sink, for example, provides at-least-once writes, meaning duplicates can occur.

Spark also offers Real-time Mode for lower-latency operational workloads, with a defined subset of supported sources, operators, and sinks. Don't confuse it with the older experimental Continuous Processing mode, which has at-least-once guarantees. Neither the label “streaming” nor “real-time” establishes a hard deadline for your application.

A local micro-batch demonstration

Using the same environment, save this as stream_demo.py and run python stream_demo.py. The rate source generates test rows, and the console sink is for debugging—not durable production output.

from pyspark.sql import SparkSession

spark = (
    SparkSession.builder.master("local[2]")
    .appName("StreamingIntroduction")
    .getOrCreate()
)
query = None

try:
    events = spark.readStream.format("rate").option("rowsPerSecond", 2).load()
    query = (
        events.writeStream
        .format("console")
        .outputMode("append")
        .trigger(processingTime="2 seconds")
        .start()
    )
    query.awaitTermination(10)
finally:
    if query is not None:
        query.stop()
    spark.stop()

The processing-time trigger requests a two-second micro-batch interval; a batch can take longer to execute. Before adapting this for production, choose durable output and checkpoint locations and verify recovery semantics in the Structured Streaming guide.

Common Mistakes

Pulling the dataset into the driver. collect() loads all result rows into driver memory. Keep large results distributed; inspect bounded samples instead.
Caching everything or adding machines before inspecting the plan. Measure reuse, shuffle volume, skew, and spill. Use the Spark UI and execution plans to identify the actual bottleneck.
Treating infrastructure as someone else’s problem. Plan for local scratch-disk capacity, failed jobs, dependency compatibility, and access controls. Spark’s security documentation warns that deployments are not secure by default.

Conclusion

Think of Spark as the compute layer in a data platform. Choose storage and cluster management separately, then decide whether distributed processing earns its operational cost. Hadoop components can be part of that platform, but they are not a prerequisite for Spark.

FAQ

Can Apache Spark replace Hadoop?

Spark can replace Hadoop MapReduce for suitable processing workloads, but it does not replace HDFS storage or YARN resource management. Spark is a separate project that can use those components or run without a Hadoop cluster.

What languages does Apache Spark support?

Spark provides Java, Scala, Python through PySpark, and R through SparkR interfaces, but API coverage differs. Typed Datasets are available in Scala and Java, and SparkR has been deprecated since Spark 4.0.0.

Does Apache Spark require HDFS?

No. Spark can use supported filesystems, object stores, databases, and streaming sources through compatible APIs and connectors. Access depends on the required libraries, credentials, permissions, and configuration.

Is Spark real-time or micro-batch?

Structured Streaming uses micro-batches by default, but Spark also offers lower-latency Real-time Mode for supported workloads. Execution mode, sources, sinks, and workload determine latency and delivery guarantees; a trigger interval is not a hard latency bound.