OEM 13c Corrective Action to extend tablespace size: Wrong ASM projected safely usable free space (MB)

In Oracle Enterprise Manager 13c ( 13.2.0.0.0),  auto extending tablespace corrective action is not working as expected, specially when calculating space usage.

*** BEGIN Parameters ***
Increase Bigfile Size (%): 5
Maximum Disk Usage (%): 95
How to Increase Space: Increase by %
...
..
.
ASM disk group is: DATA
Last created datafile: +DATA/RACTEST/3F9C860784456287E053530F040ADB20
                                  /DATAFILE/test.372.1018794017
Tablespace total size (MB): 899964.00
Largest numeric suffix: 0
Datafile filename: test.372
Datafile directory: +DATA/RACTEST/3F9C860784456287E053530F040ADB20/
                                                          DATAFILE/
Datafile suffix: .1018794017
ASM space usage for disk group DATA is free(MB): 5817030.000, 
                           total(MB): 28311415.000, required 
                     for mirroring(MB): 0.000, redundancy: 1
ASM projected safely usable free space (MB) is: 1317210.000
ASM projected Space Used (%) is: 95.35
Not enough disk space left in disk group DATA to extend datafile, 
                                                     95.35% > 95%
Disconnected from database

From the CA job trace file, we can see:

Increase Bigfile Size (%): 5
Maximum Disk Usage (%): 95
How to Increase Space: Increase by %
ASM space usage for disk group DATA is free(MB):  5817030.000,
                                      total(MB): 28311415.000
Current big tablespace size : 899964.00
ASM projected safely usable free space (MB) is: 1317210.000
ASM projected Space Used (%) is: 95.35
Not enough disk space left in disk group DATA to extend datafile, 
95.35% > 95%
Disconnected from database

ASM projected safely usable free space (MB) should be :

Current disk group free space - Current Tablespace Size * Increase % 
= 5817030.000 - 899964*5/100 = 5772031.8 MB

While the Corrective Action trace file shows below ASM projected safely usable free space (MB):

ASM projected safely usable free space (MB) is: 1317210.000

Let’s check the source code from Oracle Enterprise Agent, we can see the code missed Percentage ( %) for tablespace increase rate.

$cd /u01/app/oracle/product/agent/agent_13.2.0.0.0/plugins/
       oracle.sysman.db.agent.plugin_13.2.1.0.0/scripts/db

$ls -l dbAddSpaceTS.pl

...
..
.
 # Case 1: Increase existing datafile by %
    if ($bIncrByPct)
    {
      $diskFreeMB = $dirAvailMB - ($tsSizeMB * $incrPct);
    }
...
..
.

# Case 1: Increase by %
  if ($bIncrByPct)
  {
    $safelyUsableMB = ($dirAvailMB - ($tsSizeMB * $incrPct) 
                                   - $dgReqMirror) / $dgRedundancy;
  }
...
..
.

so the right code should be :

 # Case 1: Increase existing datafile by %
if ($bIncrByPct)
{
$diskFreeMB = $dirAvailMB - ($tsSizeMB * $incrPct)/100;
}
...
..
.

# Case 1: Increase by %
if ($bIncrByPct)
{
$safelyUsableMB = ($dirAvailMB - ($tsSizeMB * $incrPct)/100 
- $dgReqMirror) / $dgRedundancy;
}
...
..
.

After modifying the code, everything works fine for Corrective Action job.

Use Corrective Action ( CA ) to Automate Archivelog Backup When FRA Usage is Over Threshold of Metric Extension ( ME ) in OEM

Take advantage of OEM Corrective Action ( CA ) to automate archive logs backup after FRA usage over threshold.

In Previous posts, we have created following related posts:

  1. Create a Metric Extension (ME) to Monitor FRA Usage on OEM
  2. Create an Incident Rule for Metric Extension ( ME ) to Monitor FRA Used in OEM

Now and then, FRA usage is easily over threshold, DBA needs logon system and manually run an archivelog backup.  It is easily to make database hung if FRA usage is 100%.

Here we use “Corrective Actions” to run a RMAN archivelog backup when the FRA usage is over warning / Critical threshold.

Subscribe to get access

Read more of this content when you subscribe today.

10) After a little while, an alert is received about “Corrective action status=Succeed”

Corrective action=RMAN_AUTO_BACKUP_ARCHIVELOGS
Corrective action owner=SYSMAN 
Corrective action status=Succeeded 
Corrective action output=SID: RACTEST1
HOME: /u01/app/oracle/product/11.2.0/dbhome_1
BASE: /u01/app/oracle
SID: RACTEST1
HOME: /u01/app/oracle/product/11.2.0/dbhome_1
BASE: /u01/app/oracle

Host=RACTEST1.ractest 
Target type=Cluster Database 
Target name=RACTEST-CLUSTER 
Categories=Capacity 
Message=The value of Flash Recovery Area % Used for +FRA is 82
Severity=Warning 
Event reported time=Dec 22, 2016 9:01:54 PM EST 
Target Lifecycle Status=Development 
Comment=RAC TEST
Operating System=Linux
Platform=x86_64
Associated Incident Id=209879 
Associated Incident Status=New 
Associated Incident Acknowledged By Owner=No 
Associated Incident Priority=High 
Associated Incident Escalation Level=0 
Event Type=Metric Alert 
Event name=ME$RACRecoveryAreaSpace:FRA_PERCENT_USED 
Metric Group=ME$RACRecoveryAreaSpace
Metric=FRA_PERCENT_USED
Metric value=82
Key Value=+FRA
Key Column 1=FRA_NAME
Rule Name=Flash Recovery Area Percent Used for ME