Blog

ORA-00600: internal error code, arguments: [kwqitnmphe:ltbagi]

There are a lot ORA-00600 [kwqitnmphe:ltbagi] errors in alert log, this error is caused by a QMON slave background process:

*** 2019-10-05 18:43:28.563
*** SESSION ID:(1906.11789) 2019-10-05 18:43:28.563
*** CLIENT ID:() 2019-10-05 18:43:28.563
*** SERVICE NAME:(SYS$BACKGROUND) 2019-10-05 18:43:28.563
*** MODULE NAME:(Streams) 2019-10-05 18:43:28.563
*** ACTION NAME:(QMON Slave) 2019-10-05 18:43:28.563

kwqitmmit during kwqitmmit:time mgr IOT proc: Error 600 in Queue 
                                Table SYS.SYS$SERVICE_METRICS_TAB
kwqicaclcur: Error 600
Cursor Session Number : 1904
Cursor Session Serial : 24687
Cursor Pin Number : 24585
kwqitmmit during kwqitmmit:end proc: Error 600 in Queue Table 
                                        SYS.SYS$SERVICE_METRICS_TAB
kwqitptm: error 600 encountered during serving 10008981

*** 2019-10-05 18:43:28.564
ORA-00600: internal error code, arguments: [kwqitnmphe:ltbagi], 
                        [1], [0], [], [], [], [], [], [], [], [], []
OPIRIP: Uncaught error 447. Error stack:
ORA-00447: fatal error in background process
ORA-00600: internal error code, arguments: [kwqitnmphe:ltbagi], [1], 
                             [0], [], [], [], [], [], [], [], [], []

Subscribe to get access

Read more of this content when you subscribe today.

Invalid Queue Rule Set After DBMS_AQADM.PURGE_QUEUE_TABLE

In 11.2, after manually purged the message queue, some queue rule sets might become Invalid.

before purging the queue.

SQL> select object_type, object_name       
      from dba_objects       
      where owner='SYS' and status!='VALID'; 

no rows selected

Purge the queue table.

SQL> DECLARE         
       po dbms_aqadm.aq$_purge_options_t;      
     BEGIN         
       po.block := FALSE;         
       DBMS_AQADM.PURGE_QUEUE_TABLE
            ( queue_table => 'SYS.SYS$SERVICE_METRICS_TAB',
          purge_condition => NULL,         
            purge_options => po); 
     END; 

/  

PL/SQL procedure successfully completed.

Check the Invalid objects again.

SQL> select object_type, object_name       
       from dba_objects      
      where owner='SYS' and status!='VALID'; 

OBJECT_TYPE           OBJECT_NAME
 -------------------  --------------------------
 RULE SET             SYS$SERVICE_METRICS_N

Recompile the invalid objects.

SQL> @$ORACLE_HOME/rdbms/admin/utlrp.sql

SQL> select object_type, object_name
from dba_objects
where owner='SYS' and status!='VALID';

no rows selected

How to Get Database, Tablespace,Table or Index Size in PostgreSQL

1) Database Size in PostgreSQL.

testdb=> SELECT pg_size_pretty ( pg_database_size ('testdb') );

pg_size_pretty
----------------
136 GB
(1 row)
testdb=>select pg_database.datname,
      pg_size_pretty(pg_database_size(datname)) AS DBSIZE
      from pg_database;
datname    | dbsize
-----------+---------
template0  | 7873 kB
rdsadmin   | 8125 kB
template1  | 8037 kB
postgres   | 8037 kB
testdb     | 136 GB
(5 rows)

2) Tablespace size in PostgreSQL.

testdb=> select pg_size_pretty ( pg_tablespace_size ('pg_default'));

pg_size_pretty
----------------
136 GB
(1 row)
testdb=> select pg_tablespace.spcname,
pg_size_pretty(pg_tablespace_size(spcname)) AS "TBLSPACESIZE" 
 from pg_tablespace;

  spcname   | TBLSPACESIZE
------------+--------------
 pg_default | 136 GB
 pg_global  | 774 kB
(2 rows)

3)Table size in PostgreSQL.

—  Specify the disk space used by the table OID or table name, excluding the index (but containing TOAST, free space mapping, and visual mapping).

testdb=> select pg_size_pretty( pg_table_size('testuser.test'));
pg_size_pretty
----------------
10 GB
(1 row)
testdb=> select pg_size_pretty( pg_relation_size('testuser.test'));
pg_size_pretty
----------------
10 GB
(1 row)

— Specify the total disk space used by the table OID or table name, including all indexes and TOAST data.

testdb=> select pg_size_pretty( pg_total_relation_size('testuser.test'));
pg_size_pretty
----------------
16 GB
(1 row)

4)Index size in PostgreSQL.

— total disk space associated with table index of specified table OID or table name.

testdb=> select pg_size_pretty( pg_indexes_size('testuser.test'));
pg_size_pretty
----------------
5377 MB
(1 row)
-- get all index names of table testuser.test
testdb=>\d testuser.test
...
..
.

-- get size of index
testdb=> select pg_size_pretty( pg_relation_size('testuser.ix_test_id'));
pg_size_pretty
----------------
942 MB
(1 row)

5) Top 10 biggest tables.

testdb=> select schemaname,tablename, 
 pg_size_pretty( pg_table_size(schemaname||'.'||tablename)) TABSIZE, 
 pg_table_size(schemaname||'.'||tablename) TABSIZEINBYTES
from pg_tables order by 4 desc limit +10;

schemaname  | tablename         | tabsize | tabsizeinbytes
------------+-------------------+---------+---------------
testuser    | cdr               | 44 GB   | 47675359232
testuser    | test              | 10 GB   | 11257372672
...
..
.
(10 rows)

6) Top 10 tables with biggest indexes.

testdb=> select schemaname,tablename,
pg_size_pretty( pg_indexes_size(schemaname||'.'||tablename)) IDXSIZE,
pg_indexes_size(schemaname||'.'||tablename) IDXSIZEINBYTES 
from pg_tables order by 4 desc limit +10;

schemaname | tablename | idxsize  | idxsizeinbytes
-----------+-----------+----------+----------------
testuser   | cdr       | 29 GB    | 31294144512
testuser   | test      | 5379 MB  | 5640585216
...
..
.
(10 rows)

7) Top 10 biggest indexes.

testdb=> select schemaname,tablename,indexname,
pg_size_pretty(pg_relation_size(schemaname||'.'||indexname)) idxsize,
pg_relation_size(schemaname||'.'||indexname) indexsizeinbytes 
from pg_indexes 
order by 5 desc limit +10;

schemaname  | tablename | indexname | idxsize | indexsizeinbytes
------------+-----------+-----------+---------+------------------
testuser    | cdr       | pk_cdr    | 4045 MB | 4241883136
testuser    | test      | ix_pk     | 2777 MB | 2912100352
...
..
.
(10 rows)

8) Top 10 biggest tables ( including indexes, TOAST…).

testdb=> select schemaname,tablename,
pg_size_pretty( pg_total_relation_size(schemaname||'.'||tablename)) relsize,
pg_total_relation_size(schemaname||'.'||tablename) relsizeinbytes 
from pg_tables order by 4 desc limit +10;

schemaname  | tablename | relsize | relsizeinbytes
------------+-----------+---------+----------------
testuser    | cdr       | 74GB    | 78969503744
testuser    | test      | 16 GB   | 16900882432
testuser    | test1     | 13 GB   | 13817872384
...
..
.
(10 rows)

Functions of database object

Function nameReturn typedescribe
pg_column_size(any)intThe number of bytes required to store a specified value (possibly compressed)
pg_database_size(oid)bigintSpecify the disk space used by the OID database
pg_database_size(name)bigintDisk space for specified name database
pg_indexes_size(regclass)bigintUse total disk space associated with table index of specified table OID or table name
pg_relation_size(relation regclass, fork text)bigintSpecify the table or index of OID or name by specifying fork(‘main‘, Disk space used by’fsm’or’vm’
pg_relation_size(relation regclass)bigintAbbreviation for pg_relation_size(…,’main’)
pg_size_pretty(bigint)textConverts a size in bytes expressed as a 64-bit integer into a human-readable format with size units
pg_size_pretty(numeric)textConverting a byte-based value into a human readable unit of size
pg_table_size(regclass)bigintSpecify the disk space used by the table OID or table name, excluding the index (but containing TOAST, free space mapping, and visual mapping)
pg_tablespace_size(oid)bigintSpecify the disk space used by the table space of the OID
pg_tablespace_size(name)bigintDisk space used by the specified name table space
pg_total_relation_size(regclass)bigintSpecify the total disk space used by the table OID or table name, including all indexes and TOAST data

OEM 13c Agent Creates Many OraInstall_Timestamp Directories in /tmp

For Oracle Enterprise Manager 13c agent, it creates many directories and files under /tmp of Linux server:

...
..
.
drwxr-----. 2 oracle oinstall 46 Sep 18 10:05 OraInstall2019-09-18_10-05-53AM
drwxr-----. 2 oracle oinstall 46 Sep 18 10:05 OraInstall2019-09-18_10-05-57AM
drwxr-----. 2 oracle oinstall 46 Sep 18 10:06 OraInstall2019-09-18_10-06-02AM
drwxr-----. 2 oracle oinstall 46 Sep 19 10:05 OraInstall2019-09-19_10-05-49AM
drwxr-----. 2 oracle oinstall 46 Sep 19 10:05 OraInstall2019-09-19_10-05-53AM
drwxr-----. 2 oracle oinstall 46 Sep 19 10:05 OraInstall2019-09-19_10-05-57AM
drwxr-----. 2 oracle oinstall 46 Sep 19 10:06 OraInstall2019-09-19_10-06-01AM
$ cd OraInstall2019-09-19_10-06-01AM

$ ls -ltr
total 4
-rw-r-----. 1 oracle oinstall 129 Sep 19 10:06 nextgen2019-09-19_10-06-01AM.log

$cat nextgen2019-09-19_10-06-01AM.log
2019-09-19 10:06:01,829 INFO [1] oracle.sysman.oii.oiio.oiiol.OiiolTextLogger - OiicStandardInventorySession(OPatch,13.8.0.0.0)

CAUSES

It is a bug for OEM 13c agent which tries to collect ORACLE_HOME configurations, etc.

SOLUTION

  1.  This issue is a bug which is fixed in EM Cloud Control Ver 13.4. Upgrade the OMS/Agents to Ver 13.4 to fix the issue permanently.
  2. Manually or using a cronjob to delete those files.

OEM 13c Corrective Action to extend tablespace size: Wrong ASM projected safely usable free space (MB)

In Oracle Enterprise Manager 13c ( 13.2.0.0.0),  auto extending tablespace corrective action is not working as expected, specially when calculating space usage.

*** BEGIN Parameters ***
Increase Bigfile Size (%): 5
Maximum Disk Usage (%): 95
How to Increase Space: Increase by %
...
..
.
ASM disk group is: DATA
Last created datafile: +DATA/RACTEST/3F9C860784456287E053530F040ADB20
                                  /DATAFILE/test.372.1018794017
Tablespace total size (MB): 899964.00
Largest numeric suffix: 0
Datafile filename: test.372
Datafile directory: +DATA/RACTEST/3F9C860784456287E053530F040ADB20/
                                                          DATAFILE/
Datafile suffix: .1018794017
ASM space usage for disk group DATA is free(MB): 5817030.000, 
                           total(MB): 28311415.000, required 
                     for mirroring(MB): 0.000, redundancy: 1
ASM projected safely usable free space (MB) is: 1317210.000
ASM projected Space Used (%) is: 95.35
Not enough disk space left in disk group DATA to extend datafile, 
                                                     95.35% > 95%
Disconnected from database

From the CA job trace file, we can see:

Increase Bigfile Size (%): 5
Maximum Disk Usage (%): 95
How to Increase Space: Increase by %
ASM space usage for disk group DATA is free(MB):  5817030.000,
                                      total(MB): 28311415.000
Current big tablespace size : 899964.00
ASM projected safely usable free space (MB) is: 1317210.000
ASM projected Space Used (%) is: 95.35
Not enough disk space left in disk group DATA to extend datafile, 
95.35% > 95%
Disconnected from database

ASM projected safely usable free space (MB) should be :

Current disk group free space - Current Tablespace Size * Increase % 
= 5817030.000 - 899964*5/100 = 5772031.8 MB

While the Corrective Action trace file shows below ASM projected safely usable free space (MB):

ASM projected safely usable free space (MB) is: 1317210.000

Let’s check the source code from Oracle Enterprise Agent, we can see the code missed Percentage ( %) for tablespace increase rate.

$cd /u01/app/oracle/product/agent/agent_13.2.0.0.0/plugins/
       oracle.sysman.db.agent.plugin_13.2.1.0.0/scripts/db

$ls -l dbAddSpaceTS.pl

...
..
.
 # Case 1: Increase existing datafile by %
    if ($bIncrByPct)
    {
      $diskFreeMB = $dirAvailMB - ($tsSizeMB * $incrPct);
    }
...
..
.

# Case 1: Increase by %
  if ($bIncrByPct)
  {
    $safelyUsableMB = ($dirAvailMB - ($tsSizeMB * $incrPct) 
                                   - $dgReqMirror) / $dgRedundancy;
  }
...
..
.

so the right code should be :

 # Case 1: Increase existing datafile by %
if ($bIncrByPct)
{
$diskFreeMB = $dirAvailMB - ($tsSizeMB * $incrPct)/100;
}
...
..
.

# Case 1: Increase by %
if ($bIncrByPct)
{
$safelyUsableMB = ($dirAvailMB - ($tsSizeMB * $incrPct)/100 
- $dgReqMirror) / $dgRedundancy;
}
...
..
.

After modifying the code, everything works fine for Corrective Action job.