Manual Recovery Guide for SAP HANA on Azure Large Instance from storage snapshot with AzAcSnap

[GeertVanTeylingen]

Table of Contents

Version 

Authors 

Overview 

Assumptions 

Terms and Definitions 

System status 

Recover the database to its most recent state 

Recover the database to the following point in time 

Recover the database to a specific data (snapshot) backup 

Appendix - SAP HANA Data Volume locations 

Version

This article is for the SAP HANA on Azure Large Instances using the Microsoft AzAcSnap version 4.2 or later.

Authors

Phil Jensen, Principal Software Engineer at Microsoft.

Overview

This article provides guidance on using SAP HANA Studio to recover SAP HANA on Azure Large Instances. This article has step-by-step screenshots to follow to understand the three primary methods of recovering SAP HANA using HANA Studio from a snapshot taken using the Microsoft provided snapshot tools.

The screenshots in this document are from SAP HANA Studio session accessing SAP HANA 2.0SPS04. Be aware that user interfaces may vary, dependent on the specific SAP HANA release.

Disclaimer: This article and the associated screenshots are taken from an SAP HANA v2.0 system recovery as set up in the Microsoft test environment for SAP HANA on Azure Large Instance. Anyone following this guide is responsible for ensuring the recovery process works in their own environment as expected.

Assumptions

The administrator following this artcile has experience with SAP HANA and HANA Studio because not all details are provided as screenshots to follow (e.g. logging in to HANA Studio, etc.).

The administrator is familiar with SAP HANA backup processes, including the Backup Catalog and Storage Snapshots.

The administrator has the appropriate permissions at a Linux shell to copy files as the <sid>adm user into the SAP HANA Data Area.

Terms and Definitions

Terms used in this documentation: 

• SID: A System Identifier for SAP HANA installation, typically 3 characters long. 
• HLI: SAP HANA on Azure Large Instance Unit.

System status

The system layout used for this documentation has a “primary” SID (C31) and another second tenant (C32). 

The second tenant (C32) was created using the SQL commands:

• CREATE DATABASE C32 SYSTEM USER PASSWORD <SomePassword>

The primary data area is under “/hana/data/C31/mnt00001”.  Further explanation of the SAP HANA persistent data storage area is in the Appendix.

Recover the database to its most recent state

In this case the goal is to restore the complete system (SYSTEMDB, C31, C32) from a snapshot to the most recent database state, including any log replay.

1. First step is to stop the database
   
   
   When this is finished, the Processes tab should display as follows:
   
2. Start the recovery process from the menu.
   

   Note, the recovery wizard can take several seconds to launch (see the following status)

   

3. Choose the recovery type, in this case “Recover the database to its most recent state”
   

4. Choose the location of the backup catalog, which is needed for recovery.
   

5. The backup catalog will be fetched to display the appropriate backup to recover from (this can take a minute or two to load).
   

6. The first time the backup catalog is refreshed, its likely no suitable snapshot will be found to restore from.  This is because the administrator will need to copy/restore the files from the snapshot into the data area.
   

7. In this example, the files are copied from the “hidden” snapshot location in the filesystem.
   

   # su - c31adm
   
   > cp -pr /hana/data/C31/mnt00001/.snapshot/hana_hourly.2019-09-15_2100.1/* /hana/data/C31/mnt00001/.

8. When the copy is complete, refresh the view of the backup catalog to ensure the snapshot we are restoring from is listed.
   

9. Now select the available SNAPSHOT shown in green to recover from.
   

10. Choose the location of the Log Backups.
    

11. Check any appropriate “Other Settings”, the following screen is the defaults.
    

12. On the summary page, review any final details and press Finish to restore the system database.
    

13. When the recovery has finished a Recovery Execution Summary provides details of the recovery.  The following screen shows a completed recovery of the SYSTEMDB.
    
    

    (!) NOTE:
    
    Note the message stating “recovering the system database from a storage snapshot invalidates all the tenant databases”. Tenant databases must now be recovered.

14. Start the recovery of the Tenant database.
    

15. Choose the Tenant to recover from.
    

16. Choose to recover the tenant database to its most recent state (same as for the system database).
    

17. Provide the location of the Backup Catalog (same as for the system database).
    

18. Allow the tenant database to be stopped for recovery.
    

19. Wait for the Backup Catalog to be refreshed and displayed.
    

22. When recovering the tenant database there should already be a valid snapshot to recover from (unlike the system database where we needed to restore the snapshot files into the data area and refresh the view).  Select this snapshot and click next.
    
23. Specify any locations for log backups to include in the recovery process.
    
24. Check any appropriate “Other Settings”, the following screen is the defaults.
    
25. On the summary page, review any final details and press Finish to restore the tenant database. Select Finish to proceed with the recovery.
    
26. The recovery process can take a few minutes, depending on database size and log files to process.
    
    
27. When the recovery has finished a Recovery Execution Summary provides details of the recovery. The following screen shows a completed recovery of the TENANT DB.
    
    
28. The following screenshot shows the database after recovery with some services running.
    Note, there is no process for C32 running, this tenant still needs to be recovered.

Repeat the steps 14-25 to recover any other tenants.

In our example, after recovering tenant C32, the process list looks like the following:

A process listing can also be retrieved form the command line when logged in as the <sid>adm user.

> /usr/sap/hostctrl/exe/sapcontrol -nr 00 -function GetProcessList

15.09.2019 23:51:43
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2019 09 15 23:08:57, 0:42:46, 28998
hdbcompileserver, HDB Compileserver, GREEN, Running, 2019 09 15 23:09:28, 0:42:15, 29598
hdbindexserver, HDB Indexserver-C31, GREEN, Running, 2019 09 15 23:21:34, 0:30:09, 31935
hdbindexserver, HDB Indexserver-C32, GREEN, Running, 2019 09 15 23:37:51, 0:13:52, 36538
hdbnameserver, HDB Nameserver, GREEN, Running, 2019 09 15 23:08:57, 0:42:46, 29017
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2019 09 15 23:09:28, 0:42:15, 29601
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2019 09 15 23:09:29, 0:42:14, 29648
hdbxsengine, HDB XSEngine-C31, GREEN, Running, 2019 09 15 23:21:53, 0:29:50, 32071

Recover the database to the following point in time

This process allows recovery of the database to a specific point in time, perhaps just prior to an invalid transaction.

1. First step is to stop the database.
   
   

   When this is finished, the Processes tab should display as follows:

   
   
   
2. Start the recovery process from the menu.
   

   Note, the recovery wizard can take several seconds to launch (see the following status):

   
   
   
3. Choose the recovery type, in this case “Recover the database to the following point in time”, in this example the time stamp chosen is 16-September-2019 05:00:00 (in 24 hour UTC/GMT).
   
   

   (!) NOTE:
   
   The time used is based on UTC/GMT.

4. Confirm the recovery to continue, noting the potential for lost data.
   
   
   
5. Choose the location of the backup catalog, which is needed for recovery.
   
   
   
6. The backup catalog will be fetched to display the appropriate backup to recover from (this can take a minute or two to load).
   
   
   
7. The first time the backup catalog is refreshed, its likely no suitable snapshot will be found to restore from.  This is because the administrator will need to copy/restore the files from the snapshot into the data area.
   
   
   
8. In this example, the files can be copied from the “hidden” snapshot location in the filesystem.
   

   su – c31adm
   
   > cp -pr /hana/data/C31/mnt00001/.snapshot/hana_hourly.2019-09-15_2100.2/* /hana/data/C31/mnt00001/.
   

   
   
9. When the copy is complete, refresh the view of the backup catalog to ensure the snapshot we are restoring from is listed.
   
   
   
10. Now select the available SNAPSHOT shown in green to recover from.
    
    
    
11. Choose the location of the Log Backups.
    
    
    
12. Check any appropriate “Other Settings”, the following screen is the defaults.
    
    
    
13. On the summary page, review any final details and press Finish to restore the system database.
    
    
    
14. When the recovery has finished a Recovery Execution Summary provides details of the recovery.  The following screen shows a completed recovery of the SYSTEMDB.
    
    

    (!) NOTE:
    
    Note the message stating “recovering the system database from a storage snapshot invalidates all the tenant databases”. Tenant databases must now be recovered.

15. Start the recovery of the Tenant database.
    
    
    
16. Choose the Tenant to recover from.  At the time of writing, only a single tenant database is supported by SAP to recover from. 
    
    
    
17. Choose to recover the tenant database to the following point in time (same as for the system database). 
    

    (!) NOTE:
    
    The time used is based on UTC/GMT.

18. Provide the location of the Backup Catalog (same as for the system database).
    
    
    
19. Allow the tenant database to be stopped for recovery.
    
    
    
20. Wait for the Backup Catalog to be refreshed and displayed.
    
    
    
21. When recovering the tenant database there should already be a valid snapshot to recover from (unlike the system database where we needed to restore the snapshot files into the data area and refresh the view).  Select this snapshot and click next.
    
    
    
22. Specify any locations for log backups to include in the recovery process.
    
    
    
23. Check any appropriate “Other Settings”, the following screen is the defaults.
    
    
    
24. On the summary page, review any final details and press Finish to restore the tenant database. Select Finish to proceed with the recovery.
    
    
    
25. The recovery process can take a few minutes, depending on database size and log files to process.
    
    
    
26. When the recovery has finished a Recovery Execution Summary provides details of the recovery. The following screen shows a completed recovery of the TENANT DB.
    
    
    
27. The following screenshot shows the database after recovery with some services running.
    

    Note, there is no process for C32 running, this tenant still needs to be recovered.
    

Repeat the steps 14-25 to recover any other tenants.

In our example, after recovering tenant C32, the process list looks like the following:

A process listing can also be retrieved form the command line when logged in as the <sid>adm user.

> /usr/sap/hostctrl/exe/sapcontrol -nr 00 -function GetProcessList

15.09.2019 23:51:43
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2019 09 15 23:08:57, 0:42:46, 28998
hdbcompileserver, HDB Compileserver, GREEN, Running, 2019 09 15 23:09:28, 0:42:15, 29598
hdbindexserver, HDB Indexserver-C31, GREEN, Running, 2019 09 15 23:21:34, 0:30:09, 31935
hdbindexserver, HDB Indexserver-C32, GREEN, Running, 2019 09 15 23:37:51, 0:13:52, 36538
hdbnameserver, HDB Nameserver, GREEN, Running, 2019 09 15 23:08:57, 0:42:46, 29017
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2019 09 15 23:09:28, 0:42:15, 29601
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2019 09 15 23:09:29, 0:42:14, 29648
hdbxsengine, HDB XSEngine-C31, GREEN, Running, 2019 09 15 23:21:53, 0:29:50, 32071

Recover the database to a specific data (snapshot) backup

This process recovers the database to a specific snapshot only (i.e. no log replay).

1. First step is to stop the database. 
   

   When this is finished, the Processes tab should display as follows:

   
   
   
2. Start the recovery process from the menu.
   

   Note, the recovery wizard can take several seconds to launch (see the following status):

   
   
   
3. Choose the recovery type, in this case “Recover the database to a specific data backup”.
   
   
   
4. Confirm the recovery to continue, noting the potential for lost data.
   
   
   
5. As there will be no log replay, continue to “Recover without the backup catalog”.
   
   
   
6. Specify the Backup to Recover, Destination Type = Snapshot.
   
   
   
7. Note this restore method will Initialize Log Area. Check any appropriate “Other Settings”, the following screen is the defaults.
   
   
   
8. Restore the snapshot files to the data area.  In this example, the files can be copied from the “hidden” snapshot location in the filesystem.
   

   # su – c31adm
   > cp -pr /hana/data/C31/mnt00001/.snapshot/hana_hourly.2019-09-15_2100.2/* /hana/data/C31/mnt00001/.

   
   
9. On the summary page, review any final details.  Make sure you have copied/restored the snapshot files to the data area, if the copy has completed then press Finish to restore the system database.
   
   
   
10. When the recovery has finished a Recovery Execution Summary provides details of the recovery.  The following screen shows a completed recovery of the SYSTEMDB.
    
    

    (!) NOTE:
    
    Note the message stating “recovering the system database from a storage snapshot invalidates all the tenant databases”. Tenant databases must now be recovered.

11. Start the recovery of the Tenant database.
    
    
    
12. Choose the Tenant to recover from.  At the time of writing, only a single tenant database is supported by SAP to recover from.
    
    
    
13. Choose to recover the tenant database to a specific data backup.
    
    
    
14. As there will be no log replay, continue to “Recover without the backup catalog”.
    
    
    
15. Specify the Backup to Recover, Destination Type = Snapshot.
    
    
    
16. Note this restore method will Initialize Log Area. Check any appropriate “Other Settings”, the following screen is the defaults.
    
    
    
17. There is no need to restore the snapshot files to the data area as this was done when recovering the system database.
    
    
18. On the summary page, review any final details and press Finish to restore the system database.
    
    
    
19. When the recovery has finished a Recovery Execution Summary provides details of the recovery.  The following screen shows a completed recovery of the tenant database.
    
    
    
20. The following screenshot shows the database after recovery with some services running.
    

    Note, there is no process for C32 running, this tenant still needs to be recovered.

Repeat the steps 11-20 to recover any other tenants.

In our example, after recovering tenant C32, the process list looks like the following:

A process listing can also be retrieved form the command line when logged in as the <sid>adm user.

> /usr/sap/hostctrl/exe/sapcontrol -nr 00 -function GetProcessList

15.09.2019 23:51:43
GetProcessList
OK
name, description, dispstatus, textstatus, starttime, elapsedtime, pid
hdbdaemon, HDB Daemon, GREEN, Running, 2019 09 15 23:08:57, 0:42:46, 28998
hdbcompileserver, HDB Compileserver, GREEN, Running, 2019 09 15 23:09:28, 0:42:15, 29598
hdbindexserver, HDB Indexserver-C31, GREEN, Running, 2019 09 15 23:21:34, 0:30:09, 31935
hdbindexserver, HDB Indexserver-C32, GREEN, Running, 2019 09 15 23:37:51, 0:13:52, 36538
hdbnameserver, HDB Nameserver, GREEN, Running, 2019 09 15 23:08:57, 0:42:46, 29017
hdbpreprocessor, HDB Preprocessor, GREEN, Running, 2019 09 15 23:09:28, 0:42:15, 29601
hdbwebdispatcher, HDB Web Dispatcher, GREEN, Running, 2019 09 15 23:09:29, 0:42:14, 29648
hdbxsengine, HDB XSEngine-C31, GREEN, Running, 2019 09 15 23:21:53, 0:29:50, 32071

Appendix – SAP HANA Data Volume locations

A detailed explanation of persistent data storage can be found in the “SAP HANA Administration Guide for SAP HANA Platform” - “Persistent Data Storage in the SAP HANA Database” section.

The following diagram is taken from the “Data and Log Volumes” sub-section. This shows the Directory Hierarchy for Persistent Data Storage (System with Multitenant Database Containers) for SAP HANA.  Note the separation of System DB and Tenant DB files into logically grouped sub-directories.  The volume names of tenant databases have a suffix to represent the database. For example, the indexserver volume for the first tenant database is hdb00002.00002, for the second database hdb00002.000003, and so on.  For example, Tenant DB 1 data storage is grouped into both “hdb00002.00003” and “hdb00003.00003” sub-directories for the indexserver and xsengine respectively.