Migrate on Premise HBase Cluster to HDInsight

Former Employee

Nov 04, 2021

HDInsight HBase is offered as a managed cluster that is integrated into the Azure environment. The clusters are configured to store data directly in Azure Storage, which provides low latency and increased elasticity in performance and cost choices. This property enables customers to build interactive websites that work with large datasets. To build services that store sensor and telemetry data from millions of end points. And to analyze this data with Hadoop jobs. HBase and Hadoop are good starting points for big data project in Azure. The services can enable real-time applications to work with large datasets. In this article we will see the approach to Migrate on Premise HBase to HDInsight HBase.

Please follow the below documentation for Best Practice and the Benefit for Migrating to Azure HDInsight.
https://docs.microsoft.com/en-us/azure/hdinsight/hadoop/apache-hadoop-on-premises-migration-motivation

Use Case: In this example we will demonstrate the two approach for Migration.

Copy the hbase folder: With this approach, you copy all HBase data, without being able to select a subset of tables or column families. Subsequent approaches provide greater control. HBase uses the default storage selected when creating the cluster. HBase stores its data and metadata files under the following path: /hbase. In this approach there is more downtime based on data size.

Snapshot: This enable you to take a point-in-time backup of data in your HBase datastore. Snapshots have minimal overhead and complete within seconds, because a snapshot operation is effectively a metadata operation capturing the names of all files in storage at that instant. At the time of a snapshot, no actual data is copied. Snapshots rely on the immutable nature of the data stored in HDFS, where updates, deletes, and inserts are all represented as new data. You can restore (clone) a snapshot on the same cluster, or export a snapshot to another cluster. This can be done table by table but the advantage is of less downtime.

Incremental Load: is common for both the approach

Lets run this step by step.

Option 1: Using Copying hbase folder approach

Step 1: Create Table in Source Cluster and insert some data

Step 2: Note down the timestamp before stopping the HBase service from Ambari and taking the backup.

Step 3: Copy the HBase Folder from source to Destination cluster

Step 4: Rename the HBase directory to hbase_mig in destination cluster

Step 5: Copy the Source HBase Data folder to destination HBase Data Folder

Note: Please find the link for copying the data as there are multiple option based upon the business requirement. Here we have used Hadoop copy command.

Data migration: On-premises Apache Hadoop to Azure HDInsight | Microsoft Docs

Step 6: Once the Data is copied Restart the HBase Services From Ambari in destination Cluster.

Step 7: Verify the destination Cluster by listing the table and scanning the table.

There may be scenario which takes time to copy the data in that case your on premise cluster still receives some data. Please refer to Incremental load option as mentioned in below document in incremental load section.

Option 2: Using Snapshot Based Approach

Step 1: Create Table in Source Cluster and insert some data