Step 1: Introduction
Customers moving SAP on Oracle systems to Azure are strongly advised to follow clear guidance from Oracle, Microsoft and SAP to move to Automatic Storage Management (ASM). There are many performance, administration and support benefits with ASM. Customers that have used a migration from on-prem to Azure as an opportunity to move to ASM have given consistent positive feedback that ASM is the best storage solution for Oracle.
Oracle DBAs who are not familiar with ASM can follow this procedure to learn more about ASM.
For Oracle 19c and higher the SAP Basis team should ensure the Oracle DBA team move away from LVM and use ASM for any new installation or migration to Azure.
Jan Klokkers – Oracle Corporate Vice President responsible for SAP:
“ASM is the default storage architecture for Oracle databases for many years. It is being used on Oracle Engineered Systems and for all RAC deployments on UNIX/Linux systems.
ASM is the only storage layer which has been optimized for Oracle databases. Filesystems such as EXT4 and XFS have never been optimized for Oracle databases as they are designed for general purpose use and not for database workload.
Close to ASM comes only Azure NetApp Files with Direct NFS where Direct NFS has been designed to run an Oracle database more efficiently.
All other solutions are not robust and reliable for large databases and do not provide the throughput to run large databases (especially large SAP systems).”
- Moving your SAP Database to Oracle Automatic Storage Management 11g Release 2 - A Best Practices Gui...
- SAP on Oracle – Automatic Storage Management (ASM) - SAP on Oracle - Support Wiki
- Oracle Azure Virtual Machines DBMS deployment for SAP workload - Azure Virtual Machines | Microsoft Learn
- 1550133 - Using Oracle Automatic Storage Management (ASM) with SAP NetWeaver based Products - SAP ON...
- 1565179 - SAP software and Oracle Linux - SAP ONE Support Launchpad
Step 2: Installation Details & Inputs
The setup and configuration in this document is detailed below:
- OS = Oracle Linux 8.6 Azure Gallery Image
- DB = Oracle 19c Patch 16
- SAP Release = NetWeaver 7.5 Application Server
- SAP SID = C11
- *Oracle DBMS ORACLE_SID=C11
- DB_SID=C11
- Oracle GRID ORACLE_BASE=/oracle/BASE
- Oracle GRID ORACLE_HOME=/oracle/GRID/19.0.0
- ORA_INVENTORY=/oracle/oraInventory
- Oracle GRID ORACLE_SID = +ASM
- Oracle DBMS ORACLE_HOME=/oracle/C11/19.0.0
- Oracle DBMS ORACLE_BASE=/oracle/C11
- Oracle ORACLE_SID=C11
- Azure VM = E16v5
- *the Oracle SID should be the same as the SAP SID, but it is permitted to be a different value
Experienced Oracle DBAs will note that the installation is using the same user “oracle” to own both the GRID and the DBMS instance. The Oracle Home and Oracle Base values may also not conform to previous standards, however the values are correct.
Disk Layout:
Oracle ASM can be used with Azure Premium SSD and Azure Premium SSDv2. Oracle ASM is not supported on ANF. Oracle ASM has not been tested with Azure UltraDisk. In this example the newly introduced Azure Premium SSDv2 storage has been used. Premium SSDv2 has several advantages compared to Premium SSDv1 and can be very attractive for Oracle customers as well. The new Premium SSDv2 storage is supported by SAP to be used for SAP deployments. Including database deployments.
Step 3: Operating System Preparation for SAP on Oracle ASM on Azure
- Configure the /etc/hosts. Enter the IP address and the hostname of the VM in the hosts file
|
sudo vi /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 xx.xx.xx.xx <<hostname>>.localdomain <<hostname>> |
- Either open ports required for Oracle, SWPM and SAP, or alternatively stop and disable the firewall
|
sudo systemctl disable firewalld sudo systemctl stop firewalld |
- set the sysctl parameters
|
sudo vi /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6 = 1 net.core.somaxconn = 4096 net.core.netdev_max_backlog = 300000 net.ipv4.tcp_rmem = 4096 131072 16777216 net.ipv4.tcp_wmem = 4096 16384 16777216 net.ipv4.tcp_max_syn_backlog = 16348 net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_slow_start_after_idle = 0 net.ipv4.conf.all.rp_filter = 0 net.ipv4.tcp_dsack = 1 net.ipv4.tcp_sack = 1 net.ipv4.tcp_no_metrics_save = 1 net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_syn_retries = 8 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_timestamps = 1 ### added by the SAP installation fs.file-max = 6815744 kernel.shmmni = 4096 kernel.shmall = 1073741824 kernel.shmmax = 4398046511104 kernel.panic_on_oops = 1 net.core.rmem_default = 262144 net.core.rmem_max = 16777216 net.core.wmem_default = 262144 net.core.wmem_max = 16777216 net.ipv4.conf.all.rp_filter = 2 net.ipv4.conf.default.rp_filter = 2 fs.aio-max-nr = 1048576 net.ipv4.ip_local_port_range = 9000 65500 kernel.sem= 32000 1024000000 500 32000 vm.max_map_count=2147483647
Activate the parameter sudo sysctl -p /etc/sysctl.conf
|
- Disable selinux
|
sudo vi /etc/selinux/config ... # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled |
- Set Huge Pages
Review the Microsoft DBMS Guide for Oracle and this link 7 easy steps to configure HugePages for your Oracle Database Server | CarajanDB
- Create a temporary installation user/password for using SWPM in dialog mode. Most companies do not allow the use of root
|
As user root/admin run sudo useradd <username> sudo passwd <username> sudo usermod -aG wheel <username> sudo visudo (add user to sudoers) |
- Increase swapfile to 20GB as per Oracle recommendation. Follow SAP documentation for SAP application servers. Server Configuration Checklist for Oracle Database Installation
As user with root access run
|
sudo vi /etc/waagent.conf
ResourceDisk.Format=y Provisioning.Agent= waagent
# Set the property ResourceDisk.EnableSwap to y # Create and use swapfile on resource disk. ResourceDisk.EnableSwap=y
# Set the size of the SWAP file with property ResourceDisk.SwapSizeMB #ResourceDisk.SwapSizeMB=2000 ResourceDisk.SwapSizeMB=20480
|
- Update the system and install required and useful packages. The entire list below can be copied and pasted into a terminal session. This list will run for 5-10 minutes and then restart the OS
|
cd /tmp ### update all packages sudo dnf update -y
### tuned packages sudo dnf install -y tuned-profiles-oracle.noarch
### required packages sudo dnf install -y oracle-database-preinstall-19c sudo dnf install -y oracleasm sudo dnf install -y binutils sudo dnf install -y compat-libstdc++-33 sudo dnf install -y elfutils-libelf sudo dnf install -y elfutils-libelf-devel sudo dnf install -y fontconfig-devel sudo dnf install -y glibc sudo dnf install -y glibc-devel sudo dnf install -y ksh sudo dnf install -y libaio sudo dnf install -y libaio-devel sudo dnf install -y libXrender sudo dnf install -y libXrender-devel sudo dnf install -y libX11 sudo dnf install -y libXau sudo dnf install -y libXi sudo dnf install -y libXtst sudo dnf install -y libgcc sudo dnf install -y librdmacm-devel sudo dnf install -y libstdc++ sudo dnf install -y libstdc++-devel sudo dnf install -y unixODBC
### OL8 packages sudo dnf install -y net-tools sudo dnf install -y nfs-utils sudo dnf install -y libnsl sudo dnf install -y make sudo dnf install -y targetcli sudo dnf install -y smartmontools sudo dnf install -y sysstat sudo dnf install -y xorg-x11-server-Xorg sudo dnf install -y xorg-x11-xauth sudo dnf install -y libXaw.x86_64 sudo dnf install -y xorg-x11-fonts-misc sudo dnf install -y uuidd
###Install the X11 Tools (make it easier to test the X11 environment) cd /tmp sudo wget https://vault.centos.org/centos/8/PowerTools/x86_64/os/Packages/xorg-x11-apps-7.7-21.el8.x86_64.rpm sudo yum localinstall -y xorg-x11-apps-7.7-21.el8.x86_64.rpm
### packages required for SAP sudo dnf install -y csh sudo dnf install -y motif sudo dnf install -y libXtst-devel sudo dnf install -y libXp-devel sudo dnf install -y libX11-devel sudo dnf install -y libSM-devel sudo dnf install -y libICE-devel
sudo reboot |
- Start the Tuned Daemon and configure Tuned profile
|
sudo tuned-adm list Available profiles: - accelerator-performance - Throughput performance based tuning with disabled higher latency STOP states - balanced - General non-specialized tuned profile - desktop - Optimize for the desktop use-case - hpc-compute - Optimize for HPC compute workloads - intel-sst - Configure for Intel Speed Select Base Frequency - latency-performance - Optimize for deterministic performance at the cost of increased power consumption - network-latency - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance - network-throughput - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G+ networks - optimize-serial-console - Optimize for serial console use. - oracle - Optimize for Oracle RDBMS - powersave - Optimize for low power consumption - throughput-performance - Broadly applicable tuning that provides excellent performance across a variety of common server workloads - virtual-guest - Optimize for running inside a virtual guest - virtual-host - Optimize for running KVM guests
sudo tuned-adm profile oracle throughput-performance virtual-guest
tuned-adm active Current active profile: oracle throughput-performance virtual-guest |
- Create the mountpoints for Oracle and SAP
Replace “C11” with the SAP SID and Oracle SID
|
sudo mkdir -p /oracle /oracle/C11/sapbackup sudo mkdir -p /usr/sap/C11 /sapmnt/C11 /usr/sap/trans /usr/sap/software |
Create the LVM Physical Volumes, Logical Volumes and Volume Groups for the ORACLE_HOME and the SAP file systems
Check which disks are used buy the boot disks and identify disk(s) that can be used for the Oracle Home (/oracle), SAP and installation
|
lsblk
sudo pvcreate /dev/sda sudo vgcreate software_vg /dev/sda |
Create Logical Volumes. LV can be checked with command lvscan
|
sudo lvcreate -L 10G -n software software_vg sudo lvcreate -L 50G -n oracle_home software_vg sudo lvcreate -L 20G -n usr_sap software_vg sudo lvcreate -L 20G -n sapmnt software_vg sudo lvcreate -L 10G -n saptrans software_vg sudo lvcreate -l 100%FREE -n sapbackup software_vg |
Create the xfs file systems
|
sudo mkfs -t xfs /dev/software_vg/software sudo mkfs -t xfs /dev/software_vg/oracle_home sudo mkfs -t xfs /dev/software_vg/usr_sap sudo mkfs -t xfs /dev/software_vg/sapmnt sudo mkfs -t xfs /dev/software_vg/saptrans sudo mkfs -t xfs /dev/software_vg/sapbackup |
Add these entries to the /etc/fstab
|
sudo vi /etc/fstab
# /dev/software_vg/software /usr/sap/software xfs relatime,inode64 0 0 /dev/software_vg/usr_sap /usr/sap/C11 xfs relatime,inode64 0 0 /dev/software_vg/sapmnt /sapmnt xfs relatime,inode64 0 0 /dev/software_vg/saptrans /usr/sap/trans xfs relatime,inode64 0 0 # /dev/software_vg/oracle_home /oracle xfs relatime,inode64 0 0 /dev/software_vg/sapbackup /oracle/C11/sapbackup xfs relatime,inode64 0 0 |
Mount the file systems. The mounted filesystems should look similar to the below
|
sudo mount -a df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 63G 0 63G 0% /dev tmpfs 63G 0 63G 0% /dev/shm tmpfs 63G 25M 63G 1% /run tmpfs 63G 0 63G 0% /sys/fs/cgroup /dev/mapper/rootvg-rootlv 19G 3.3G 16G 18% / /dev/mapper/rootvg-crashlv 10G 104M 9.9G 2% /var/crash /dev/sda1 794M 538M 257M 68% /boot /dev/sda15 495M 5.1M 490M 2% /boot/efi /dev/sdb1 590G 32K 560G 1% /mnt tmpfs 13G 0 13G 0% /run/user/1000 /dev/mapper/software_vg-software 10G 104M 9.9G 2% /usr/sap/software /dev/mapper/software_vg-usr_sap 20G 175M 20G 1% /usr/sap/C11 /dev/mapper/software_vg-sapmnt 20G 175M 20G 1% /sapmnt /dev/mapper/software_vg-saptrans 10G 104M 9.9G 2% /usr/sap/trans /dev/mapper/software_vg-oracle_home 40G 318M 40G 1% /oracle /dev/mapper/software_vg-sapbackup 28G 232M 28G 1% /oracle/C11/sapbackup |
Create directories under the ORACLE_HOME /oracle path
Replace “C11” with the SAP SID and Oracle SID
| sudo mkdir -p /oracle/C11/sapreorg /oracle/C11/oraInventory /oracle/C11/saptrace /oracle/C11/19.0.0 /oracle/stage /oracle/client /oracle/C11/saparch /oracle/GRID/19.0.0 /oracle/BASE |
Step 4: Install Oracle ASM Libraries & Configure ASM Disks
Note: If you are installing on OL lower than 8.6 you need to download the oracleasm-support RPM manually from the oracle website
ASM Lib download link Oracle ASMLib Downloads for Oracle Linux 8
ASM Support download link OL8: How To Install ASMLib (Doc ID 2720215.1)
|
cd /tmp sudo wget https://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.17-1.el8.x86_64.rpm sudo yum localinstall -y ./oracleasmlib-2.0.17-1.el8.x86_64.rpm
sudo dnf install -y oracleasm-support |
Check if all ASM packages are installed. Do not proceed until both packages are installed
|
rpm -qa |grep oracleasm oracleasmlib-2.0.17-1.el8.x86_64 oracleasm-support-2.1.12-1.el8.x86_64 |
Prepare Oracle ASM
|
sudo chmod -R 775 /oracle sudo chown -R oracle:oinstall /oracle |
Setting up the Oracle ASM library driver involves defining the default user (oracle) and default group (oinstall) as well as configuring the drive to start on boot (choose y) and to scan for disks on boot (choose y). You need to answer the prompts from the following command:
|
sudo oracleasm configure -i Default user to own the driver interface []: oracle Default group to own the driver interface []: oinstall Start Oracle ASM library driver on boot (y/n) [n]: y Scan for Oracle ASM disks on boot (y/n) [y]: y Writing Oracle ASM library driver configuration: done
You can also use the below command to configure oracleasm in non-interactive mode: sudo oracleasm configure -u oracle -g oinstall -e -s y |
Make sure the following entries are present in /etc/sysconfig/oracleasm
- ORACLEASM_ENABLED=true
- ORACLEASM_SCANBOOT=true
If the above parameters are not set, a reboot will undo the oracledisk group initialization.
if security/permission errors are seen on this step then the user/group ownership on /oracle and /dev/sd* are likely a problem
|
sudo oracleasm init Creating /dev/oracleasm mount point: /dev/oracleasm Loading module "oracleasm": oracleasm Configuring "oracleasm" to use device physical block size Mounting ASMlib driver filesystem: /dev/oracleasm
sudo systemctl enable oracleasm This will ensure oracleasm is loaded as part of reboot. |
Check the status if ASM is loaded successfully. Do not continue if ASM is not loaded
|
sudo oracleasm status Checking if ASM is loaded: yes Checking if /dev/oracleasm is mounted: yes |
It is required to create partitions on disks for ASMLib as documented here https://docs.oracle.com/en/database/oracle/oracle-database/19/cwlin/configuring-disk-devices-to-use-oracle-asmlib.html#GUID-5464167D-B8D6-4204-8C43-8F1E56E0ACC0
Create new primary partitions on the disks that are to be used for ASM. Use the command lsblk to determine which disk is used as a boot disk and which disks are already used by LVM
Check which disks are used with ASM with command lsblk. Ensure there is only one large partition on the disk used for creating ASM disk groups.
|
sudo fdisk /dev/sdd # 1024 DATA sudo fdisk /dev/sde # 1024 DATA sudo fdisk /dev/sdf # 1024 DATA sudo fdisk /dev/sdg # 1024 DATA sudo fdisk /dev/sdh # 512 OLOG sudo fdisk /dev/sdi # 512 OLOG sudo fdisk /dev/sdj # 3096 ARCH sudo fdisk /dev/sdk # 2048 RECO Answer the prompts from the above fdisk commands with: n for new partition p for primary partition press 3x enter for the partition number, default first cylinder and last cylinder press w to write the changes to the partition table |
Change the permission on the disks that are to be used for ASM
Check which disks are used with ASM with command lsblk and adjust the commands below
|
sudo chmod -R 775 /oracle sudo chown -R oracle:oinstall /oracle
sudo chmod 600 /dev/sdd1 sudo chmod 600 /dev/sde1 sudo chmod 600 /dev/sdf1 sudo chmod 600 /dev/sdg1 sudo chmod 600 /dev/sdh1 sudo chmod 600 /dev/sdi1 sudo chmod 600 /dev/sdj1 sudo chmod 600 /dev/sdk1
sudo chown oracle:oinstall /dev/sdd1 sudo chown oracle:oinstall /dev/sde1 sudo chown oracle:oinstall /dev/sdf1 sudo chown oracle:oinstall /dev/sdg1 sudo chown oracle:oinstall /dev/sdh1 sudo chown oracle:oinstall /dev/sdi1 sudo chown oracle:oinstall /dev/sdj1 sudo chown oracle:oinstall /dev/sdk1 |
Create ASM disks with the naming convention below. Example DATA_<SID>_NN
Replace “C11” with the Oracle SID. This clearly identifies the ASM disks on VMs with multiple instances
|
sudo oracleasm createdisk DATAC1101 /dev/sdd1 sudo oracleasm createdisk DATAC1102 /dev/sde1 sudo oracleasm createdisk DATAC1103 /dev/sdf1 sudo oracleasm createdisk DATAC1104 /dev/sdg1 sudo oracleasm createdisk ARCHC1101 /dev/sdh1 sudo oracleasm createdisk OLOGC1101 /dev/sdi1 sudo oracleasm createdisk OLOGC1102 /dev/sdj1 sudo oracleasm createdisk RECOC1101 /dev/sdk1 |
Verify the status of the ASM disks
|
sudo oracleasm listdisks ARCHC1101 DATAC1101 DATAC1102 DATAC1103 DATAC1104 OLOGC1101 OLOGC1102 RECOC1101 |
Step 5: Install SAP ASCS, Database and SAP Application Server(s)
Download the required packages from the SAP marketplace
The Oracle installation DVD and patches are located under “Databases”
The SAP Kernel required is located under: “Support Packages & Patches” à ”N” à “SAP Netweaver” à “SAP Netweaver 7.5” à “Application Server ABAP” à
Copy the SAP and Oracle software over from your desktop or jumpserver to /usr/sap/software.
The process flow for installing a NetWeaver 7.5 Distributed System is as follows:
- Create Azure VM and disk resources.
- Download required software.
- Follow the Operating System Preparation (Step #3 in this blog).
- Install ASM Libraries (Step #4).
- Install Oracle GRID on database server.
- Adapt the UID and GUID for oracle user and groups.
- Install GRID software with X11 Installation Tool.
- Install SAP ASCS, Database and SAP Application Server(s) (Step #5).
- Run sapinst and install ASCS.
- Run sapinst and install SAP Database Instance.
- When Database Installation pauses install Oracle DBMS.
- Patch Oracle DBMS and GRID.
- Continue Database Installation with sapinst.
- Run sapinst and install SAP PAS/Application Server(s).
- Perform SAP post installation tasks.
1. Install Oracle Grid with Oracle X11 Based Installer
Oracle Grid software should be installed prior to starting the SAP Database Instance with SAPInst
- Unpack the DB client and the DB (as root)
|
cd /usr/sap/software/Oracle19/ ll oracle oinstall 711317528 Oct 20 08:34 client.zip oracle oinstall 2815340105 Oct 20 08:32 GIRU19P_2208-70004508.ZIP oracle oinstall 2815340105 Oct 24 15:02 GIRU19P_2208-70004550.ZIP oracle oinstall 124138770 Oct 20 08:29 OPATCH19P_2208-70004508.ZIP oracle oinstall 9931078768 Oct 20 08:41 oracle19.zip oracle oinstall 27092 Oct 21 06:08 sudo oracleasmlib-2.0.17-1.el8.x86_64.rpm oracle oinstall 55698 Oct 20 08:30 README19P_2208-70004508.HTM oracle oinstall 24265636 Oct 20 08:30 SAP19P_2208-70004508.ZIP oracle oinstall 58 Oct 24 15:54 SAPBundle oracle oinstall 128461935 Oct 24 15:56 SBPJDK19P_2208-70004508.ZIP
mkdir client DB OPatch SAPBundle GRIDupdate cd client unzip ../client.zip ... cd ../DB/ unzip ../oracle19.zip ... cd ../OPatch/ unzip ../OPATCH19P_2208-70004508.ZIP ... cd ../SAPBundle/ unzip ../SAP19P_2208-70004508.ZIP ... cd ../GRIDupdate unzip ../GIRU19P_2208-70004550.ZIP ... cd .. sudo chmod -R 775 * sudo chown -R oracle:oinstall * |
X11 requires software such as xming or mobaxterm.
Review this article and test X11 before continuing Enabling X11 forwarding on Linux | SAP Blogs
Enable the X11 forwarding (as root) – The X11 package xorg-x11-apps-7.7-21.el8.x86_64.rpm should automatically update this file. If X11 does not work check this parameter.
|
sudo vi /etc/ssh/sshd_config ForwardX11 yes |
Restart the sshd – only required if the sshd_config was changed.
| sudo systemctl restart sshd |
Login directly as user oracle without doing su - to keep the xterm settings from putty.
If it is not possible to logon directly using the oracle user, follow the procedure below to keep the X11 settings:
|
sudo cp /home/<azureuser>/.Xauthority /home/oracle/.Xauthority sudo chown oracle:oinstall /home/oracle/.Xauthority |
Test the sample X11 application xclock
|
ssh oracle@<host> xauth list #Test the X11 environment xclock |
- Unzip the Grid archive in the GRID Oracle Home and run gridSetup.sh. Grid must be unzipped into the final installation directory i.e
|
/oracle/GRID/19.0.0 cd /oracle/GRID/19.0.0 sudo unzip /usr/sap/software/Oracle19/DB/LINUX_X86_64/grid_home/LINUX.X64_193000_grid_home.zip sudo chmod -R 775 /oracle sudo chown -R oracle:oinstall /oracle |
Install the latest GRID - Cluster Verification Utility (as user with sudo access)
Installing the cvuqdisk RPM for Linux (oracle.com)
|
cd /oracle/GRID/19.0.0 sudo rpm -ivh ./cv/rpm/cvuqdisk-1.0.10-1.rpm Verifying... ####################### [100%] Preparing... ####################### [100%] Using default group oinstall to install package 1:cvuqdisk-1.0.10-1 ####################### [100%] |
Create the asm groups asmadmin and asmoper
|
sudo groupadd -g 54327 asmadmin sudo groupadd -g 54328 asmoper sudo groupadd -g 54329 asmdba sudo usermod -a -G asmdba oracle sudo usermod -a -G asmadmin oracle sudo usermod -a -G asmoper oracle |
Set the user- and group ID for the installation software directory
|
cd /usr/sap/software/ sudo chown -R oracle:oinstall Oracle19 sudo chmod -R 755 Oracle19 |
Set the owner for the /dev/sudo oracleasm directory
| sudo chown -R oracle:oinstall /dev/oracleasm |
Logon directly as oracle (without doing su -) or prepare the .Xauthority file in /user/home/oracle as detailed above
|
export ORACLE_BASE=/oracle/BASE export ORACLE_HOME=/oracle/GRID/19.0.0 export ORA_INVENTORY=/oracle/oraInventory export CV_ASSUME_DISTID=OEL8.1 export DISPLAY=localhost:10.0 cd /oracle/GRID/19.0.0 ./gridSetup.sh Launching Oracle Grid Infrastructure Setup Wizard... |
The gridSetup.sh summary screen should look like the screenshot below. Detailed step by step screenshots are attached to this blog.
- Run asmca to create OLOG, ARCH and RECO Disk Groups and Change Compatibility of DATA, OLOG, ARCH and RECO to 19.0.0.0.0. Azure storage is already highly redundant and it is supported to use External redundancy. Alternatively use ASM redundancy and/or Log Mirroring if required.
|
cd /oracle/GRID/19.0.0/bin ./asmca |
Add the disk groups OLOG, RECO and ARCH
Right mouse click on “Disk Groups” and select “Create”
Edit the Disk and change DBMS release to 19.0.0.0 – Right click on disk group in asmca and “Edit Attributes“ -> “Database Compatibility“ = 19.0.0.0.0
2. Install ASCS with SAPInst
Install the ASCS. The screenshots attached to this blog show step by step installation of a NetWeaver 7.5 Distributed ASCS.
Select the desired installation type. Step by step screenshots are attached to this blog showing installation of a NetWeaver 7.5 Distributed System.
Start the SWPM as root
|
cd /usr/sap/software/NW7.5/SWPM/ sudo ./sapinst SAPINST_REMOTE_ACCESS_USER= <username created in step 3.5> [==============================] / extracting... done! INFO: Native locale set to en_US.UTF-8 INFO 2022-10-20 13:57:54.180 (mainThread) [sixxcreate.cpp:349] ******************************************************************************** Initial log directory: /root/.sapinst/ralforadb01/8791 ********************************************************************************
SAPinst build information: -------------------------- Version: 753.0.2 ... Open your browser and paste the following URL address to access the GUI https://<<hostname.domain_name>>:4237/sapinst/docs/index.html |
3. Install SAP Database Instance with SAPInst
Start the SAP installer to install the SAP Database Instance (as root)
Detailed step by step screenshots of the Database Instance installation are attached to this blog
|
cd /usr/sap/software/NW7.5/SWPM sudo ./sapinst SAPINST_REMOTE_ACCESS_USER= <username created in step 3.5> |
Choose the following options during the installation:
- Oracle Database Installation for = Single Instance
- Filesystem or Oracle ASM = Oracle ASM
- Database Server Version = 19
- Database Client Version = 19
- Advanced DB Configuration – For Oracle Experts Only = tick
- Oracle Database Software Owner = oracle
- An existing database was detected and can be made available = Remove database and recreate
- Stop after step ‘createDB’ for patch update
- ORACLE_HOME for GRID = /oracle/GRID/19.0.0
- ORACLE_SID for GRID = +ASM
- Oracle GRID Software Not Installed = untick
- Oracle ASM Disk Group = add OLOG
When SAPInst stops proceed to step 4 below
4. Install Oracle Database Instance with RUNINSTALLER
Prepare the environment for the oracle user. Add the lines at the end of the bashrc.
Then logoff and relogin again.
|
su – oracle vi .bashrc ... export ORACLE_HOME=/oracle/C11/19.0.0 export ORACLE_SID=C11 export DB_SID=C11 export ORACLE_BASE=/oracle/C11 export ORA_INVENTORY=/oracle/C11/oraInventory export DISPLAY=localhost:10.0 export CV_ASSUME_DISTID=OEL8.1 export TNS_ADMIN=$ORACLE_HOME/network/admin
logoff exit |
Change the owner of the oracle DB installation media (as root)
|
sudo chown -R oracle:oinstall /usr/sap/software/Oracle19 sudo chmod -R 755 /usr/sap/software/Oracle19 |
logon as oracle user (X11 enabled)
|
cd /usr/sap/software/Oracle19/DB/LINUX_X86_64/db_home/SAP ./RUNINSTALLER |
Follow the inputs as per SAP note 2660017 to install Oracle. Step by step screenshots are attached to this blog
When RUNINSTALLER finishes, press continue in SAPInst. Do NOT do Oracle and GRID SBP patching immediately after installation of the Oracle DBMS
When SAPInst stops after CREATE DB it is then usually safe to apply patches or apply patches after SWPM has completely finished. Customers report that it is more reliable and safe to apply patches after the SAPInst installation finishes.
5. Run OPatch to Patch GRID and Oracle DBMS
Oracle patching can either be done after the creation of the database in SAPInst or after the SAP Database Installation process has completely finished. Either option will work.
Check the actual OPatch version as user = oracle
|
$ORACLE_HOME/OPatch/opatch version OPatch Version: 12.2.0.1.17 |
OPatch update for DBMS Instance
Now update the OPatch executables as user = oracle
|
cd /usr/sap/software/Oracle19 unzip ../OPATCH19P_2208-70004508.ZIP
pwd /usr/sap/software/Oracle19/Opatch/
mv /oracle/C11/19.0.0/OPatch /oracle/C11/19.0.0/OPatch-back cd /usr/sap/software/Oracle19
cp -r OPatch /oracle/C11/19.0.0/ $ORACLE_HOME/OPatch/opatch version ======================================================== GENERIC OPATCH VERSION - FOR USE IN SAP ENVIRONMENT ONLY ======================================================== OPatch Version: 12.2.0.1.30 OPatch succeeded. |
OPatch update for GRID
Update the OPatch as user = oracle
|
cd /usr/sap/software/Oracle19 unzip ../OPATCH19P_2208-70004508.ZIP mv /oracle/GRID/19.0.0/OPatch /oracle/GRID/19.0.0/OPatch.backup cp -r OPatch /oracle/GRID/19.0.0/ |
MOpatch update for DBMS instance
We extracted the SAP bundle patch earlier as user = oracle
|
cd /usr/sap/software/Oracle19/SAPBundle/SBP_191600220719_202208
cp -r MOPatch /oracle/C11/19.0.0/ |
MOpatch update for GRID
We extracted the SAP bundle patch earlier as user = oracle
|
cd /usr/sap/software/Oracle19/SAPBundle/SBP_191600220719_202208
cp -r MOPatch /oracle/GRID/19.0.0/ |
Install the SAP Bundle Patch
Both files are required in the Patch directory – as user = oracle
|
SAP19P_2208-70004508.ZIP SBPJDK19P_2208-70004508.ZIP This will take some time cd /usr/sap/software/Oracle19 $ORACLE_HOME/MOPatch/mopatch.sh -v -s SAP19P_2208-70004508.ZIP ... .. Log file: $ORACLE_HOME/cfgtoollogs/mopatch/mopatch-2022_10_24-16-00-36.log MOPatch completed successfully. Overall Status: COMPLETE, NON-SBP PATCHES LOST |
Before you start the GRID patch verify the ORACLE_LISTS:
Display the ORACLE_HOME names, there must be two. One for the DB and one for GRID.
|
cat /oracle/oraInventory/ContentsXML/inventory.xml <HOME_LIST> <HOME NAME="OraDB19Home1" LOC="/oracle/OR1/19.0.0" TYPE="O" IDX="1"/>
if you see only one ORACLE_HOME_NAME create a second one. In my case for GRID (as oracle user) cd $ORACLE_HOME/oui/bin ./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=/oracle/GRID/19.0.0 ORACLE_HOME_NAME=OraGRID19Home1 CRS=true
List now the ORACLE_HOME list – you now have two entries cat /oracle/oraInventory/ContentsXML/inventory.xml <HOME_LIST> <HOME NAME="OraDB19Home1" LOC="/oracle/OR1/19.0.0" TYPE="O" IDX="1"/> <HOME NAME="OraGRID19Home1" LOC="/oracle/GRID/19.0.0" TYPE="O" IDX="2" |
GRID update
Run as user with sudo access
|
export ORACLE_BASE=/oracle/BASE export ORACLE_HOME=/oracle/GRID/19.0.0 export ORA_INVENTORY=/oracle/oraInventory export OHGRID=/oracle/GRID/19.0.0 cd /oracle/GRID/19.0.0 sudo /oracle/GRID/19.0.0/crs/install/roothas.sh -prepatch |
|
Now apply the GRID Patch as user = oracle cd /usr/sap/software/Oracle19
$OHGRID/MOPatch/mopatch.sh -v -s SGR19P_2208-70004550.ZIP
==Following patches were SUCCESSFULLY applied:
Patch: /usr/sap/software/Oracle19/GRIDupdate/34130714/34160635 Log: /oracle/OR1/19.0.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-10-24_16-52-31PM_1.log
OPatchauto session completed at Mon Oct 24 16:53:00 2022 Time taken to complete the session 1 minute, 10 seconds |
Post patch activities as a user with sudo access
|
sudo /oracle/GRID/19.0.0/rdbms/install/rootadd_rdbms.sh sudo /oracle/GRID/19.0.0/crs/install/roothas.sh -postpatch . CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service' . CLSRSC-672: Post-patch steps for patching GI home successfully completed. |
Once the patching is completed, press continue in SAPInst.
6. Installation of SAP Application Server(s)
Run SAPInst and install SAP Application Server(s).
Step by step screenshots can be found attached to this document
Appendix, Notes, Utilities and Links
1. Azure Storage Types and Oracle DataGuard
With SSDv2 there is a support limitation for Oracle Data Guard. Oracle Data Guard does not support mixing different disk sector sizes. Azure Premium Storage v1 has a 512b sector size and Azure Premium Storage v2 is 4k Native. It is not supported to have different disk sector sizes within a DataGuard configuration.
2. ASMlib
Document 2720215.1 (oracle.com) OL8: How To Install ASMLib (Doc ID 2720215.1)
3. Oracle Database Installation Selection Parameters for RUNINSTALLER
4. SAP Notes and Downloads
- Oracle ASMLib Downloads for Oracle Linux 8
- 2477472 - Oracle Database Upgrade with Grid Infrastructure (12.2) - SAP ONE Support Launchpad
- 2800021 - Upgrading to Oracle Grid Infrastructure 19c - SAP ONE Support Launchpad
- 2879781 - How to Apply a Grid Infrastructure Patch Before Grid Infrastructure Configuration (gridsetup.sh/gridsetup.bat) is executed - SAP ONE Support Launchpad
- 1521371 - Setting of ORACLE_BASE in SAP environments - SAP ONE Support Launchpad
- 2879781 - How to Apply a Grid Infrastructure Patch Before Grid Infrastructure Configuration (gridsetup.sh/gridsetup.bat) is executed - SAP ONE Support Launchpad
- 2660017 - Oracle Database Software Installation on Unix - SAP ONE Support Launchpad
- 1738053 - SAPinst for Oracle ASM installation SAP ONE Support Launchpad
- 2896926 - ASM disk group compatibility NetWeaver SAP ONE Support Launchpad
- 1550133 - Using Oracle Automatic Storage Management (ASM) with SAP NetWeaver based Products
- 888626 - Redo log layout for high-end systems SAP ONE Support Launchpad
- 105047 - Support for Oracle functions in the SAP environment SAP ONE Support Launchpad
- 2799920 - Patches for 19c: Database SAP ONE Support Launchpad
- 974876 - Oracle Transparent Data Encryption (TDE) SAP ONE Support Launchpad
- 2936683 - Oracle Linux 8: SAP Installation and Upgrade SAP ONE Support Launchpad
- 1672954 - Oracle 11g, 12c, 18c and 19c: Usage of hugepages on Linux
- 1171650 - Automated Oracle DB parameter check
5. Check ASM File Locations with ASMCMD
ASMCMD is a simple too that can be used to browse the ASM file system
|
export ORACLE_SID=+ASM export DB_SID=OR1 export ORACLE_BASE=/oracle/BASE export ORACLE_HOME=/oracle/GRID/19.0.0
[oracle@ol86asm18 bin]$ export ORACLE_HOME=/oracle/GRID/19.0.0 [oracle@ol86asm18 bin]$ export ORA_INVENTORY=/oracle/oraInventory [oracle@ol86asm18 bin]$ export OHGRID=/oracle/GRID/19.0.0 [oracle@ol86asm18 bin]$ export ORACLE_SID=+ASM [oracle@ol86asm18 bin]$ cd /oracle/GRID/19.0.0/bin [oracle@ol86asm18 bin]$ ./sqlplus /NOLOG
SQL*Plus: Release 19.0.0.0.0 - Production on Sat Nov 19 19:32:13 2022 Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle. All rights reserved.
SQL> connect / as sysasm; Connected to an idle instance. SQL> startup ASM instance started
Total System Global Area 1137173320 bytes Fixed Size 8905544 bytes Variable Size 1103101952 bytes ASM Cache 25165824 bytes ASM diskgroups mounted SQL> exit Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.3.0.0.0
[oracle@ol86asm18 bin]$ ./asmcmd ASMCMD> ls -la State Type Rebal Name MOUNTED EXTERN N ARCH/ MOUNTED EXTERN N DATA/ MOUNTED EXTERN N OLOG/ MOUNTED EXTERN N RECO/ ASMCMD> exit |
6. Oracle IO Calibrate
ORACLE-BASE - Measuring Storage Performance For Oracle Systems
Before we start the IO Calibrate we stop the SAP system
|
sapcontrol -nr 01 -function StopSystem sapcontrol -nr 01 -function GetProcessList
name, description, dispstatus, textstatus, starttime, elapsedtime, pid disp+work, Dispatcher, GRAY, Stopped, , , 1872566 igswd_mt, IGS Watchdog, GRAY, Stopped, , , 1872567 |
|
su – c11adm sqlplus / as sysdba
SQL*Plus: Release 19.0.0.0.0 - Production on Wed Nov 2 10:51:56 2022 Version 19.16.0.0.0
Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.16.0.0.0
SQL> SELECT INSTANCE_NAME, STATUS, DATABASE_STATUS FROM V$INSTANCE;
INSTANCE_NAME STATUS DATABASE_STATUS ---------------- ------------ ----------------- OR1 OPEN ACTIVE |
Enable the TIMED_STATISTICS and TIMED_OS_STATISTICS
|
ALTER SYSTEM SET TIMED_STATISTICS=true SCOPE=SPFILE; ALTER SYSTEM SET TIMED_OS_STATISTICS=60 SCOPE=SPFILE; |
*After the test you may want to set the TIMED_OS_STATISTICS back to 0
Check the ASYNC status
|
SQL> SELECT d.name,i.asynch_io FROM v$datafile d,v$iostat_file I WHERE d.file# = i.file_no AND i.filetype_name = 'Data File';
NAME ASYNCH_IO ---------------------------------------------------------- +DATA/OR1/DATAFILE/system.262.1119616513 ASYNC_ON +DATA/OR1/DATAFILE/sysaux.263.1119616515 ASYNC_ON +DATA/OR1/DATAFILE/psapundo.264.1119616515 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3.267.1119617687 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3.268.1119617689 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3.269.1119617693 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3.270.1119617695 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3.271.1119617699 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3.272.1119617701 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3750.273.1119617703 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3750.274.1119617707 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3750.275.1119617709 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3750.276.1119617711 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3750.277.1119617715 ASYNC_ON +DATA/OR1/DATAFILE/psapsr3usr.278.1119617715 ASYNC_ON
15 rows selected. |
To turn on asynchronous I/O (if not already done, issue the following command and restart the database.
| ALTER SYSTEM SET filesystemio_options=setall SCOPE=SPFILE; |
Run the IO Calibrate test
|
SET SERVEROUTPUT ON DECLARE l_latency PLS_INTEGER; l_iops PLS_INTEGER; l_mbps PLS_INTEGER; BEGIN DBMS_RESOURCE_MANAGER.calibrate_io (num_physical_disks => 1, max_latency => 20, max_iops => l_iops, max_mbps => l_mbps, actual_latency => l_latency);
DBMS_OUTPUT.put_line('Max IOPS = ' || l_iops); DBMS_OUTPUT.put_line('Max MBPS = ' || l_mbps); DBMS_OUTPUT.put_line('Latency = ' || l_latency); END; /
Result 3 DATA Disks max_iops = 61745 latency = 2.269 max_mbps = 3792 Max IOPS = 61745 Max MBPS = 3792 Latency = 2 |
7. Adding a New Azure Disk to a ASM Disk Group – Online Balancing
Oracle ASM supports online addition of storage and online data rebalancing.
This is a key advantage of ASM over LVM. LVM requires a full backup/restore.
|
SELECT name, header_status, path FROM V$ASM_DISK;
NAME HEADER_STATU PATH -------------------------------------------------------------------- DATA_0001 MEMBER /dev/sudo oracleasm/disks/DATA_DISK_02 DATA_0000 MEMBER /dev/sudo oracleasm/disks/DATA_DISK_01 ARCH_0000 MEMBER /dev/sudo oracleasm/disks/ARCH_DISK_01 RECO_0001 MEMBER /dev/sudo oracleasm/disks/RECO_DISK_02 RECO_0000 MEMBER /dev/sudo oracleasm/disks/RECO_DISK_01 DATA_0002 MEMBER /dev/sudo oracleasm/disks/DATA_DISK_03 |
Add another disk from the Azure portal
|
lsblk -l NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 1T 0 disk sda1 8:1 0 1024G 0 part sdb 8:16 0 1T 0 disk sdb1 8:17 0 1024G 0 part sdc 8:32 0 1T 0 disk sdc1 8:33 0 1024G 0 part sdd 8:48 0 1T 0 disk sdd1 8:49 0 1024G 0 part sde 8:64 0 1T 0 disk sde1 8:65 0 1024G 0 part sdf 8:80 0 1T 0 disk sdf1 8:81 0 1024G 0 part sdg 8:96 0 1.5T 0 disk sdh 8:112 0 1.5T 0 disk sdi 8:128 0 30G 0 disk sdi1 8:129 0 800M 0 part /boot sdi2 8:130 0 28.7G 0 part sdi14 8:142 0 4M 0 part sdi15 8:143 0 495M 0 part /boot/efi sdj 8:144 0 2.4T 0 disk sdj1 8:145 0 2.4T 0 part /mnt sdk 8:160 0 1T 0 disk |
format the new disk
|
fdisk /dev/sdk
Command (m for help): n Select (default p): p Partition number (1-4, default 1): First sector (256-268435455, default 256): Last sector, +sectors or +size{K,M,G,T,P} (256-26835455, default 26435455): Created a new partition 1 of type 'Linux' and of size 1024 GiB. Command (m for help): w The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks. |
Oracle ASM Disk Groups can either be extended by adding additional disks or by extending current disks. It is strongly recommended to add additional disks rather than extending existing disks. Review these MOS articles and links
MOS Notes 1684112.1 and 2176737.1
As root
|
sudo chown oracle:oinstall /dev/sdk1 cd /oracle/GRID/19.0.0/bin
sudo oracleasm listdisks ARCH_DISK_01 DATA_DISK_01 DATA_DISK_02 DATA_DISK_03 RECO_DISK_01 RECO_DISK_02
sudo oracleasm createdisk DATA_DISK_04 /dev/sdk1 Writing disk header: done Instantiating disk: done |
As oracle
|
su – oracle cd /oracle/GRID/19.0.0/bin
./asmca -silent -addDisk -diskGroupName DATA -disk '/dev/sudo oracleasm/disks/DATA_DISK_04' |
System will automatically redistribute the data
|
ps -ef | grep rbal oracle 1524207 1 0 Nov01 ? 00:00:11 asm_rbal_+ASM oracle 1610444 1 0 Nov01 ? 00:00:03 ora_rbal_OR1 |
Once this process is finished, we start the new IO Calibrate to see if there is a difference
Thanks to Ralf Klahr and Vamshi Polasa for contributing and work to get to this blog
Microsoft