Reconfigure OS Parameters After Linux Server Memory Upgrade

Adjust OS memory parameters after more or less memory available

One of the Oracle database server ‘s memory is to be upgraded from 48GB to 72GB, so some parameters should be adjusted to take advantage of the newly added memory.

Change memlock Parameter

Modify “memlock” parameter in /etc/security/limits.conf to 70G, which is for hugepage purpose. This parameter can be bigger than SGA, up to or even bigger than memory size.

oracle  soft      memlock  73400320
oracle  hard      memlock  73400320
  grid  soft      memlock  73400320
  grid  hard      memlock  73400320

Change nr_hugepages Parameter

Change parameter ‘nr_hugepages’  in /etc/sysctl.conf.
43GB memory( 60% of whole memory ) is allocated to hugepages.

vm.nr_hugepages = 22016

Change shmmni, shmall and shmmax  values in /etc/sysctl.conf

 a)Keep kernel.shmmni = 4096

b) kernel.shmall=13369344 (51GB)  = 70% of whole memory divided by page size 4096.

c ) kernel.shmmax=38654705664( 36GB) =  50% of the whole memory size.

kernel.shmmni = 4096
kernel.shmall = 13369344
kernel.shmmax = 38654705664

Reconfigure SGA_TARGET and SGA_MAX_SIZE

Make sure the total SGA usage is less than vm.nr_hugepages = 22016( 43GB).

High Swap Usage On Oracle Database Server

SITUATION

When investigating into client’s Oracle database performance issue, we found the swap space usage is constantly very high on this Linux server.

OS: RHEL 7.3
DB: Oracle 12.2.0.1

FINDINGS

1)top
Tasks: 352 total, 2 running, 350 sleeping, 0 stopped, 0 zombie
Cpu(s): 13.4%us, 4.1%sy, 0.0%ni, 79.3%id, 2.2%wa, 0.3%hi, 0.8%si, 0.0%st
Mem: 32172820k total, 32015956k used, 156864k free, 14528k buffers
Swap: 16777208k total, 7435428k used, 9341780k free, 11129844k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
137049 oracle 15 0 16.2g 5.3g 5.3g S 20.6 17.4 10:14.17 oracle
 72457 oracle 15 0 16.2g 4.7g 4.7g S 15.3 15.3 10:50.15 oracle
...
..
.
2) pmap
$ pmap -x 137049
137049: oracleRACTEST1 (LOCAL=NO)
Address          Kbytes   RSS     Dirty Mode Mapping
0000000000400000 96356    11704   0     r-x-- oracle
0000000006419000 444      140     4     rwx-- oracle
0000000006488000 148      100     80    rwx-- [ anon ]
000000001966e000 532      176     92    rwx-- [ anon ]
0000000060000000 16779264 5444888 1485768 rwxs- [ shmid=0x670005 ]
00000032b6a00000 112      108     0     r-x-- ld-2.5.so
00000032b6c1c000 4        0       0     r-x-- ld-2.5.so
...
..
.
00007fff4c504000 160 136 132 rwx-- [ stack ]
00007fff4c5d2000 12 4 0 r-x-- [ anon ]
ffffffffff600000 8192 0 0 ----- [ anon ]
---------------- ------ ------ ------
total kB 16954204 5477952 1490448
3) swappiness
$ cat /proc/sys/vm/swappiness
10

Subscribe to get access

Read more of this content when you subscribe today.

How to Configure HugePages for Oracle Database on Linux

Oracle strongly recommends to configure HugePages for servers with more than 8GB memory used by Oracle database(s).

a) Setting up  memlock in /etc/security/limits.conf. Make this parameter slightly smaller than available physical memory ( 512G ).

oracle soft memlock 475830802
oracle hard memlock 475830802

b) Check memlock setting is effective by logging as oracle user.

$ ulimit -l
475830802

c) Copy shell script hugepages_settings.sh as per My Oracle Support (MOS) note 401749.1), and run this script as user oracle.

$ chmod 744 ./hugepages_setting.sh 
$./hugepages_settings.sh

 . . . Recommended setting: vm.nr_hugepages = 86016

--- --- hugepages_settings.sh 
--- --- #!/bin/bash 
# # hugepages_settings.sh 
# # Linux bash script to compute values for the 
# recommended HugePages/HugeTLB configuration 
# on Oracle Linux 
# # Note: This script does calculation for all shared memory 
# segments available when the script is run, no matter it 
# is an Oracle RDBMS shared memory segment or not. 
# # This script is provided by Doc ID 401749.1 from My Oracle Support 
# http://support.oracle.com 
# Welcome text
echo " This script is provided by Doc ID 401749.1 from My Oracle Support (http://support.oracle.com) where it is intended to compute values for the recommended HugePages/HugeTLB configuration for the current shared memory segments on Oracle Linux. Before proceeding with the execution please note following:  
* For ASM instance, it needs to configure ASMM instead of AMM.  
* The 'pga_aggregate_target' is outside the SGA and    you should accommodate this while calculating SGA size.  
* In case you changes the DB SGA size, as the new SGA will not fit in the previous HugePages configuration,    it had better disable the whole HugePages,    start the DB with new SGA size and run the script again. And make sure that:  
* Oracle Database instance(s) are up and running  
* Oracle Database 11g Automatic Memory Management (AMM) is not setup    (See Doc ID 749851.1)  
* The shared memory segments can be listed by command:      
# ipcs -m 
Press Enter to proceed..."
read 
# Check for the kernel 
version KERN=`uname -r | awk -F. '{ printf("%d.%d\n",$1,$2); }'` 
# Find out the HugePage size 
HPG_SZ=`grep Hugepagesize /proc/meminfo | awk '{print $2}'` 
if [ -z "$HPG_SZ" ];
then     
  echo "The hugepages may not be supported in the system where the script is being executed."     
  exit 1 
fi 
# Initialize the counter NUM_PG=0 
# Cumulative number of pages required to handle the running shared memory segments 
for SEG_BYTES in `ipcs -m | cut -c44-300 | awk '{print $1}' | grep "[0-9][0-9]*"` 
do     
MIN_PG=`echo "$SEG_BYTES/($HPG_SZ*1024)" | bc -q`     
if [ $MIN_PG -gt 0 ]; 
then         
 NUM_PG=`echo "$NUM_PG+$MIN_PG+1" | bc -q`     
 fi 
done 
RES_BYTES=`echo "$NUM_PG * $HPG_SZ * 1024" | bc -q` 
# An SGA less than 100MB does not make sense 
# Bail out if that is the case 
if [ $RES_BYTES -lt 100000000 ]; 
then     
  echo "***********"     
  echo "** ERROR **"     
  echo "***********"     
  echo "Sorry! There are not enough total of shared memory segments allocated for HugePages configuration. HugePages can only be used for shared memory segments that you can list by command:      
# ipcs -m of a size that can match an Oracle Database SGA. Please make sure that:  
 * Oracle Database instance is up and running  
 * Oracle Database 11g Automatic Memory Management (AMM) is not configured"     exit 1 
fi 
# Finish with results 
case $KERN in     
'2.2') 
  echo "Kernel version $KERN is not supported. Exiting." ;;     
'2.4') 
  HUGETLB_POOL=`echo "$NUM_PG*$HPG_SZ/1024" | bc -q`;            
  echo "Recommended setting: vm.hugetlb_pool = $HUGETLB_POOL" ;;     
'2.6') 
  echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;     
'3.8') 
  echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;     
'3.10') 
  echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;;     
'4.1') 
  echo "Recommended setting: vm.nr_hugepages = $NUM_PG" ;; 
esac 
# End

d) Disable AMM by setting the values of both of the initialization parameters memory_target and memory_max_target to 0. Instead we use ASMM sga_target and sga_max_size for the SGA and pga_aggregate_target for PGA.

SQL> alter system set memory_target=0 scope=spfile; 

System altered. 

SQL>alter system set memory_max_target=0 scope=spfile; 

System altered.
SQL> alter system set sga_target=xxxxxx scope=spfile;
System altered.
SQL> alter system set sga_max_size=xxxxxx scope=spfile;
System altered.
SQL> alter system set pga_aggregate_target =xxxxxx scope=spfile;

e) Modify file /etc/sysctl.conf,  and set the value of the vm.nr_hugepages parameter to the recommended value.

$cat /etc/sysctl.conf
...
...
vm.nr_hugepages = 86016

f) Shut down all the instances, and reboot the Linux server to make HugePages configuration effective.

$ grep ^Huge /proc/meminfo
HugePages_Total: 86016
HugePages_Free:   6557
HugePages_Rsvd:    492
HugePages_Surp: 0
Hugepagesize: 2048 kB

Please note the sum of the HugePages_Free and HugePages_Rsvd values is likely to be smaller than the total HugePages size because  the database instances allocate pages dynamically and proactively as required.

g) Check the database instances are using HugePages in Alert.log when instances starts up.

Starting ORACLE instance (normal)
************************ Large Pages Information *******************
Per process system memlock (soft) limit = UNLIMITED

Total Shared Global Region in Large Pages = 8194 MB (100%)

Large Pages used by this instance: 4097 (8194 MB)
Large Pages unused system wide = 61442 (120 GB)
Large Pages configured system wide = 86016 (168 GB)
Large Page size = 2048 KB
********************************************************************