Blog

The Apache Hive Metastore

Key Takeaways

  • The Apache Hive Metastore is a metadata catalog and service, not the relational database that backs it.
  • Table data lives in HDFS or another supported storage system; the metastore records how query engines should interpret and locate it.
  • Embedded versus remote describes where the metastore runs. Embedded versus external database is a separate deployment choice.
  • For a shared production catalog, use a supported external database and plan schema migrations, security, backups, and availability together.

Hive lets you query large datasets using SQL-like syntax instead of writing Java MapReduce jobs yourself. The Apache Hive Metastore supplies the catalog metadata behind those tables: their schemas, storage locations, partitions, and related definitions. It commonly persists that metadata in a relational database, while table data remains in the configured storage system.

What Is the Hive Metastore?

Think of the Hive Metastore, often shortened to HMS, as the mapping between a table name and the information needed to use that table. Its catalog is available through a service API; its backing database provides persistent storage for catalog records. Those are different components with different operational responsibilities.

The catalog includes:

  • Databases and tables: names, ownership, column names, and data types.
  • Partitions: registered partition values and their storage descriptions.
  • Storage descriptors: locations, input/output formats, and serialization/deserialization information, which describes how stored bytes become rows.
  • Properties and statistics: table configuration and collected table or column statistics.
  • Other definitions: functions and authorization records, depending on the release and configuration.

The metastore API definitions show these objects explicitly. Hive can create and modify data in underlying storage; the important distinction is that HMS stores catalog metadata rather than the business rows themselves.

How the Hive Metastore Works

A metastore client is the library an engine uses to request metadata. In a remote deployment, that library contacts the HMS service over Thrift, a framework for remote procedure calls. The service reads or updates catalog records in its backing database.

SQL clients
    |
    v
HiveServer2 / Spark / Trino
    |                         |
    | metadata requests       | table reads and writes
    v                         v
Hive Metastore service     HDFS / supported object storage
    |
    | JDBC
    v
Relational database
(catalog metadata)

For a typical catalog-backed query:

  1. The engine resolves the database and table names.
  2. Its metastore client retrieves the required schema, storage, partition, and statistics metadata.
  3. The engine builds an execution plan and reads the relevant data through its storage integration.

HiveServer2's architecture separates metadata lookup from query compilation and execution. HMS does not execute the query or make storage fast by itself. Partition pruning, file layout, statistics, and storage performance still matter; see our guides to Hive partitioning and Apache Parquet.

HDFS is one storage option, not a requirement. Supported deployments can use object storage such as Amazon S3; the appropriate filesystem integration and credentials must be configured independently of HMS connectivity. For the traditional storage model, start with our Apache Hadoop architecture introduction.

Embedded, Local, and Remote Metastore Deployments

Older tutorials often describe three modes. The useful distinction is two independent choices: where the metastore implementation runs, and whether its database is embedded or externally operated.

Embedded Mode

In the simplest development setup, the metastore implementation runs inside the calling process and uses embedded Derby. Derby permits multiple connections within that JVM, but only one JVM can open the same embedded database. That is not the same as “one user at a time.”

Local Mode

In the historical local-mode arrangement, the metastore implementation remains embedded in the calling process, but connects to an external database. For example, HiveServer2 can host the metastore implementation while PostgreSQL runs separately. Each embedding process needs database access and its own connections; an external database does not automatically create a remote HMS service.

Remote Mode

A remote HMS runs as a separate service process. Engine clients contact it rather than accessing the catalog database directly. This is a useful production starting point for a shared catalog because it isolates database credentials and service operations from individual engines; availability still requires service and database redundancy.

Hive Metastore Configuration: A Remote Service

The following templates target the full Apache Hive 4.2.1 distribution, released August 24, 2026. Hive 4.2.x requires JDK 21. Replace the example hosts and paths before use, and follow your distribution's packaging requirements.

Choose the Database and Configuration Locations

Provision an empty metastore database and an account with the permissions required for initialization. PostgreSQL illustrates the wiring below; MySQL is another documented backend. Validate the exact server version, JDBC driver, and schema support against your Hive release and distribution rather than treating historical minimum-version tables as current production recommendations.

For this full-distribution example, use separate hive-site.xml files in each process's configuration directory. The standalone metastore also supports metastore-site.xml; keep configuration sources consistent so another file does not unexpectedly override your settings.

Configure the Engine's Metastore Client

On HiveServer2, place this in its hive-site.xml. The URI identifies the HMS endpoint, not the database. metastore.thrift.uris is the metastore-specific equivalent in Hive 4.2.1.

<!-- Client template: replace the hostname. -->
<configuration>
  <property>
    <name>hive.metastore.uris</name>
    <value>thrift://hms.internal.example:9083</value>
  </property>
</configuration>

Remote clients need service authentication and transport settings too when security is enabled. They do not need the HMS database password merely to make catalog requests.

Configure the Service's JDBC Backend and Warehouse

On the metastore host, save this template as /etc/hive/hive-site.xml. Install the compatible PostgreSQL JDBC driver JAR in the full Hive distribution's lib directory. The database, warehouse, certificate, and credential-store paths must already be provisioned for your environment.

<!-- Service template: replace hosts and deployment-specific paths. -->
<configuration>
  <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:postgresql://db.internal.example:5432/hive_metastore?sslmode=verify-full&amp;sslrootcert=/etc/hive/db-ca.pem</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>org.postgresql.Driver</value>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive_metastore</value>
  </property>
  <property>
    <name>hadoop.security.credential.provider.path</name>
    <value>jceks://file/etc/hive/metastore.jceks</value>
  </property>
  <property>
    <name>metastore.thrift.port</name>
    <value>9083</value>
  </property>
  <property>
    <name>metastore.warehouse.dir</name>
    <value>hdfs://namenode.internal.example:8020/warehouse/hive</value>
  </property>
  <property>
    <name>metastore.schema.verification</name>
    <value>true</value>
  </property>
  <property>
    <name>datanucleus.schema.autoCreateAll</name>
    <value>false</value>
  </property>
  <property>
    <name>metastore.try.direct.sql.ddl</name>
    <value>false</value>
  </property>
</configuration>

metastore.warehouse.dir, also known as hive.metastore.warehouse.dir, controls the default database's warehouse location—not the catalog database's files. Database and table locations can override warehouse placement. The PostgreSQL-specific direct-SQL setting follows the administration guide's recommendation.

The JDBC URL uses sslmode=verify-full to validate the database certificate and hostname. Its certificate file must contain the appropriate trusted certificate. This protects the database connection only; it does not enable TLS on the HMS endpoint.

Supply the Password Outside Checked-In XML

Hive supports Hadoop's CredentialProvider for the javax.jdo.option.ConnectionPassword secret. After preparing the protected directory, run this as the metastore operating-system account; the command prompts for the database password. Restrict access to the credential store and manage its protection password through your deployment's secret controls.

# Requires Hadoop's command-line tools and a writable /etc/hive directory.
hadoop credential create javax.jdo.option.ConnectionPassword \
  -provider jceks://file/etc/hive/metastore.jceks

Initialize the Schema, Validate It, and Start HMS

For a new, empty catalog database only, use the distribution's schema tool. The service account running these commands must have access to the configuration, JDBC driver, certificates, and credential store.

# Prerequisites: full Hive 4.2.1 installed; HIVE_HOME points to it.
# Use -initSchema only for a new, empty metastore database.
set -euo pipefail
: "${HIVE_HOME:?Set HIVE_HOME to the installed Hive distribution}"
export HIVE_CONF_DIR=/etc/hive

"$HIVE_HOME/bin/schematool" -dbType postgres -initSchema
"$HIVE_HOME/bin/schematool" -dbType postgres -validate
"$HIVE_HOME/bin/hive" --service metastore

-initSchema creates the metastore schema; -validate checks it. The full distribution's hive --service metastore command launches the service. Use a process supervisor for production operation.

Sharing HMS with Spark, Trino, and Iceberg

Spark and Trino can use HMS without using Hive's query execution engine. Shared definitions let engines discover the same tables, but they still need compatible metastore clients, storage access, data types, and table features. Our Apache Spark introduction explains Spark's role separately.

For Spark, check the supported client versions and spark.sql.hive.metastore.version and spark.sql.hive.metastore.jars settings. For Trino, the Hive connector uses hive.metastore.uri—singular—and requires separate filesystem configuration. Do not copy property names between engines indiscriminately.

Catalog compatibility is not table-format compatibility. Iceberg's HiveCatalog can use HMS, while Iceberg maintains its own metadata files, snapshots, and manifests in storage. An engine still needs an appropriate Iceberg integration and support for the table's format features.

Production Checklist

  • Recovery: back up the catalog database and test restoration. Coordinate catalog recovery with table-data recovery; preserving data files alone is not a catalog backup. Use your database's supported backup and restore procedures.
  • Availability: deploy redundant HMS instances and an available database. Hive clients can list multiple HMS URIs; test discovery and failover with every engine. More HMS replicas do not remove a database single point of failure.
  • Connection capacity: budget database connections across service replicas and embedded callers. Review datanucleus.connectionPool.maxPoolSize alongside database capacity instead of increasing it blindly.
  • Security: restrict network access, configure compatible service authentication and TLS, and separately protect JDBC connections. Verify authorization in every engine and permissions in underlying storage; catalog access is not a universal authorization policy.
  • Monitoring: alert on metadata-request failures, latency, pool exhaustion, database health, and schema mismatch errors. Test representative partition lookups as well as simple connectivity.

Common Mistakes

  • Confusing endpoints: a Thrift HMS URI and a JDBC database URL serve different connections.
  • Ignoring configuration ownership: remote engine clients should not receive service-only database credentials.
  • Assuming discovery guarantees readability: seeing a table in HMS does not prove an engine can read its format or access its files.
  • Fixing schema errors by bypassing checks: investigate the actual database and binary versions before changing verification settings.

FAQ

Is the Hive Metastore a database or a service?

The Hive Metastore is a metadata catalog and service, commonly backed by a relational database. The service exposes catalog operations; the database persists catalog records rather than table data.

How do I configure Hive to use a remote metastore?

Set hive.metastore.uris in the Hive process's configuration to the remote HMS endpoint, such as thrift://hms.internal.example:9083. Configure the service's JDBC backend separately, and match authentication and transport settings on both sides.

Can Spark or Trino use the Hive Metastore?

Yes, both can use HMS through their Hive integrations without using Hive's execution engine. Check each engine's metastore-client compatibility, storage configuration, and table-format support before sharing a catalog.

Which database versions are supported for my Hive Metastore release?

Consult the documentation and compatibility guidance for your exact Hive release or distribution, including its JDBC driver and metastore schema requirements. Do not interpret historical minimum database versions as recommendations for a new production deployment.

Why is embedded Derby unsuitable as a production metastore backend?

Only one JVM can open a given embedded Derby database, although that JVM can make multiple connections. It is not the independently operated database service expected for a shared production catalog, and Derby's developers voted to retire the project on October 10, 2025.

Further Reading

Written by Sam Erickson

Software engineer. I write every article on StackChief myself. Author of Mastering the Tech Interview. More about me

Related reading