Hadoop Data Formats: Avro or Parquet?

Key Takeaways

• Choose Avro for record-oriented interchange and sequential processing of complete records; choose Parquet for analytical scans that read selected columns. Neither is universally faster.
• Avro object container files include the writer schema. Raw Avro binary records need that schema supplied separately.
• Parquet readers can skip unneeded columns and, where supported, use statistics and indexes to avoid reading irrelevant data.
• HDFS doesn't require a particular file format. Engine compatibility, schema handling, compression, and file layout should drive your choice.

The practical answer to Avro vs Parquet starts with your access pattern: Avro's row-oriented encoding fits record-by-record processing, while Parquet's columnar layout fits analytics over selected fields. Treat those as starting points, not performance guarantees.

You don't have to standardize an entire pipeline on one format. Consider Avro for exchanging events between applications and Parquet for the analytical datasets built from those events.

Hadoop Data Formats

HDFS stores file contents as blocks; it doesn't require CSV, Avro, Parquet, or any other application-level representation. You can store text documents, videos, and structured datasets in the same filesystem. The processing engine must understand the bytes you give it. See Apache Hadoop architecture for the storage and processing distinction, or the official HDFS architecture documentation.

Plain text: CSV and TSV

CSV and tab-separated values are human-readable text serialization formats, not “non-serialized” data. They're useful for simple exports and interchange, but you should explicitly agree on delimiters, escaping, null representation, and field types.

Text files can be compressed. Splittability—whether different workers can independently process portions of a file—depends on the codec and reader. Ordinary gzip text files aren't splittable by Hadoop's TextInputFormat, so each file is processed by one mapper. The MapReduce tutorial explains this constraint.

SequenceFiles

A SequenceFile stores binary key/value pairs and supports uncompressed, record-compressed, and block-compressed representations. It can package records from many inputs into fewer files, but only when your pipeline explicitly writes them that way; it doesn't automatically consolidate existing small files.

For analytical tables, investigate your engine's compaction facilities before introducing another container format. Compaction rewrites small files into larger ones to reduce metadata and file-opening overhead.

Avro

Avro is a serialization system with JSON-defined schemas and a binary encoding. Its types include records, arrays, maps, enums, unions, and fixed-size values, allowing explicit contracts for nested data. For a broader introduction, read what Apache Avro is.

Avro object container files store the writer schema in their header and organize records into optionally compressed blocks. Don't confuse those files with raw Avro messages, whose writer schema must be available separately.

Parquet

Parquet is a columnar file format with schema metadata. It organizes rows into row groups, with a column chunk for each column inside each group; those chunks contain pages. This layout lets readers access selected columns without reading every column's data. Our Apache Parquet overview covers the format separately.

ORC

ORC is another Hadoop-compatible columnar format, not a variation of Avro. It separates column data within stripes and provides indexes and statistics for selective reading. Include Parquet vs ORC in your evaluation when your Hive or other query engine supports both; compare the actual connectors and workload rather than assuming columnar formats behave identically.

Avro vs Parquet: Row-Based and Columnar Access

Avro encodes a record's fields together, then proceeds to the next record. Parquet groups values by column within each row group. The following is a conceptual layout, not a byte-for-byte file representation.

Avro object container file
  Header: writer schema + metadata
  Block: [id, region, amount] [id, region, amount] ...
  Block: [id, region, amount] [id, region, amount] ...

Parquet file
  Row group
    id column chunk     -> pages of id values
    region column chunk -> pages of region values
    amount column chunk -> pages of amount values
  Additional row groups ...
  Footer: schema + metadata locating column chunks

Column projection and predicate pushdown

Column projection, also called column pruning, means reading only the columns a query needs. For example, a regional sales aggregation needs region and amount, but not an unrelated customer profile payload. Parquet's layout lets a reader target those column chunks.

Predicate pushdown passes filter conditions toward the reader so it can avoid unnecessary work. Parquet statistics can rule out ranges of values; optional page indexes can enable finer-grained skipping when the writer supplies them and the reader supports them. If a row group's amount values range from 10 to 90, a filter requiring an amount greater than 100 can exclude that group.

These are opportunities, not guarantees. A selective filter may still read substantial data when values overlap across row groups or the engine can't use the available metadata.

Complete records aren't the same as point lookups

Avro's layout fits sequential processing that consumes complete records, but it doesn't inherently provide an index for finding one record by ID. An Avro reader can also ignore fields absent from its reader schema; that isn't equivalent to Parquet's physical separation of column data.

Conversely, Parquet can read complete records by reading and assembling the required columns. A full-record scan loses some of the advantage of column projection, but that doesn't make Parquet inherently inefficient or guarantee an Avro win.

Which Format Fits Your Workload?

Use this table to choose what to benchmark first, not to predict a universal winner.

For example, consider Avro when a service must replay complete order events. Consider Parquet when analysts repeatedly aggregate a few fields from those orders. Benchmark both when your main job reads and rewrites every field.

Neither format is tied to HDFS: Spark provides readers and writers for both. Deployment details still matter: Spark documents spark-avro as an external module, so confirm that your distribution includes it or add a compatible dependency. Read the Spark Avro data source guide and our introduction to Apache Spark for context.

Schema Evolution Is a Compatibility Contract

Avro reader and writer schemas

Avro schema evolution resolves the schema used to write data against the schema expected by the reader. A reader can ignore writer fields it doesn't request. If the reader adds a field missing from the writer schema, that reader field needs a compatible default; an added field without one fails resolution.

This small example uses the official Avro Python package. In an activated Python virtual environment, install it, then save the program as avro_evolution.py and run it.

python3 -m pip install avro
python3 avro_evolution.py

from pathlib import Path
from tempfile import TemporaryDirectory

import avro.schema
from avro.datafile import DataFileReader, DataFileWriter
from avro.io import DatumReader, DatumWriter

writer_schema = avro.schema.parse('''{
  "type": "record", "name": "Event",
  "fields": [{"name": "id", "type": "long"}]
}''')

reader_schema = avro.schema.parse('''{
  "type": "record", "name": "Event",
  "fields": [
    {"name": "id", "type": "long"},
    {"name": "source", "type": "string", "default": "unknown"}
  ]
}''')

with TemporaryDirectory() as directory:
    path = Path(directory) / "events.avro"
    with path.open("wb") as stream:
        with DataFileWriter(
            stream, DatumWriter(), writer_schema, codec="deflate"
        ) as writer:
            writer.append({"id": 1})

    with path.open("rb") as stream:
        with DataFileReader(
            stream, DatumReader(readers_schema=reader_schema)
        ) as reader:
            print(next(reader))

The container supplies the original writer schema; DatumReader supplies the application's reader schema. The reader fills the missing field from its default.

{'id': 1, 'source': 'unknown'}

Test compatibility in both directions: new readers consuming old data, and old readers consuming new data. Cross-language support doesn't remove the need to validate logical types such as timestamps and decimals.

Parquet schema evolution

Parquet also carries schema information; evolution isn't exclusive to Avro. Reading files with different schemas depends on your engine and table or catalog layer. For example, Spark can merge compatible Parquet schemas, but schema merging is disabled by default and can be expensive. Adding a column, renaming one, and changing its type aren't interchangeable operations.

File Formats and Table Formats Solve Different Problems

Avro, Parquet, and ORC describe data files. Apache Iceberg and Delta Lake manage tables across files, including metadata, snapshots, and schema evolution. They aren't alternatives at the same layer. Iceberg supports Avro, Parquet, and ORC data files; Delta Lake stores data in Parquet and tracks table changes through a transaction log.

Compression and Write Performance

Encoding and compression are different mechanisms. Dictionary encoding, for example, replaces repeated values with dictionary references. A compression codec then compresses the encoded bytes. Parquet applies these mechanisms to pages; Avro object container files apply their selected codec to record blocks.

Both formats support compression, including Snappy and Zstandard in their specifications, but codec availability depends on the implementation and installed dependencies. Test storage size, compression CPU cost, and decompression throughput together; neither format guarantees the better ratio.

Don't assume Parquet always costs more to write. Benchmark the writer, codec, encoding, sorting, and row-group configuration together. Larger Parquet row groups require more buffering in the write path, so configuration matters as well as format.

Production Checklist

• Validate every consumer: round-trip nested data, nulls, decimals, and timestamps through the actual readers you'll deploy.
• Measure representative jobs: include selective-column queries and complete-record scans; record bytes read, CPU, memory, and elapsed time.
• Tune partitioning deliberately: avoid near-unique partition keys that fragment your dataset. Our Hive partitioning guide explains directory-based layouts.
• Inspect file sizes: choosing Parquet doesn't prevent small files. Schedule engine- or table-aware compaction when ingestion produces too many of them; Iceberg maintenance describes this explicitly.
• Verify the physical read: inspect query plans and storage metrics rather than assuming a filter caused data skipping.

The decision isn't “which format is best?” It's “which representation fits this stage of the pipeline?” Start with Avro for record-oriented interchange and Parquet for selective analytical scans, then validate compatibility and performance with your own workload.

FAQ

When should I use Avro instead of Parquet?

Start with Avro for record-oriented interchange or sequential processing that consumes complete records. Its reader/writer schema resolution is useful for evolving contracts, but compatibility must be tested rather than assumed.

Is Parquet better than Avro for Spark analytics?

Parquet is usually the stronger starting point for Spark analytics that read selected columns because readers can target those column chunks. For full-record scans or write-heavy pipelines, benchmark both instead of assuming a universal winner.

Does an Avro file contain its schema?

An Avro object container file includes its writer schema in the header. Raw Avro binary serialization does not embed the full schema, so the consumer needs another way to obtain it.

Can Parquet support schema evolution?

Yes, but behavior depends on the reader and table or catalog layer. Spark can merge compatible schemas across Parquet files, while table formats can provide additional evolution rules; arbitrary renames and type changes aren't automatically safe.

Do both Avro and Parquet support compression?

Yes. Avro object container files compress record blocks, while Parquet compresses page data. Choose a codec supported by every reader and measure the size-versus-CPU trade-off on representative data.

Further Reading

• Apache Avro specification — container files, schema resolution, logical types, and codecs.
• Avro Python getting started — package installation and schema-based file reading and writing.
• Apache Parquet file format — file layout and column-chunk metadata.
• Parquet compression — supported codecs and their format requirements.
• Apache ORC documentation — columnar stripes, indexes, and engine integrations.
• Spark Parquet data source guide — schema merging and read/write behavior.
• Apache Iceberg specification — table metadata, snapshots, schemas, and data-file formats.
• Delta Lake FAQ — the relationship between Parquet data and the transaction log.