ORA-600 [kwqmnuji: bad option]

SYMPTOMS

There are a lot of ORA-600 [kwqmnuji: bad option] errors in alert.log of 11.2.0.4 RAC database.

ORA-00600: internal error code, arguments: [kwqmnuji: bad option], [2], [], [], [], [], [], [], [], [], [], []
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x119190D, kwqmncif()+31] [flags: 0x0, count: 1]
Exception [type: SIGSEGV, SI_KERNEL(general_protection)] [ADDR:0x0] [PC:0x119190D, kwqmncif()+31] [flags: 0x0, count: 1]

CAUSES

This is “Bug 14746914 – ORA-600 [kwqmnuji: bad option] using AQ in RAC (Doc ID 14746914.8)”.

The fix for 14746914 is first included in

WORKAROUND

We used below workaround to make ORA-600 [kwqmnuji: bad option] error disappeared from alert.log.

1) Bounce the instance, and still see cored umps:

Errors in file /u01/app/oracle/diag/rdbms/ractest/RACTEST1/trace/RACTEST1_ora_2954.trc:
Errors in file /u01/app/oracle/diag/rdbms/ractest/RACTEST1/trace/RACTEST1_ora_2954.trc:
Errors in file /u01/app/oracle/diag/rdbms/ractest/RACTEST1/trace/RACTEST1_ora_2954.trc:

2) Check OS process 2954 and its related session. then kill this process.

$kill -9 2954

3) Check alert.log, and find there is no more ORA-00600 errors and core dumps.

Drop Tablespace With Datafile Missing

shutdown database.
startup database in mount status.
alter database datafile ‘PATH/filename’ offline drop.
open database.
drop tablespace tablespace_name including contents.

Note, the following errors could be seen in alert.log:

ORA-01135: file 4 accessed for DML/query is offline
ORA-01110: data file 4: ''

ORA-01157: cannot identify/lock data file 4 - see DBWR trace file
ORA-01110: data file 4: ''
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3

Reconfigure OS Parameters After Linux Server Memory Upgrade

Adjust OS memory parameters after more or less memory available

One of the Oracle database server ‘s memory is to be upgraded from 48GB to 72GB, so some parameters should be adjusted to take advantage of the newly added memory.

Change memlock Parameter

Modify “memlock” parameter in /etc/security/limits.conf to 70G, which is for hugepage purpose. This parameter can be bigger than SGA, up to or even bigger than memory size.

oracle  soft      memlock  73400320
oracle  hard      memlock  73400320
  grid  soft      memlock  73400320
  grid  hard      memlock  73400320

Change nr_hugepages Parameter

Change parameter ‘nr_hugepages’ in /etc/sysctl.conf.
43GB memory( 60% of whole memory ) is allocated to hugepages.

vm.nr_hugepages = 22016

Change shmmni, shmall and shmmax values in /etc/sysctl.conf

a)Keep kernel.shmmni = 4096

b) kernel.shmall=13369344 (51GB) = 70% of whole memory divided by page size 4096.

c ) kernel.shmmax=38654705664( 36GB) = 50% of the whole memory size.

kernel.shmmni = 4096
kernel.shmall = 13369344
kernel.shmmax = 38654705664

Reconfigure SGA_TARGET and SGA_MAX_SIZE

Make sure the total SGA usage is less than vm.nr_hugepages = 22016( 43GB).

“Database option mismatch: PDB installed version NULL.” in PDB_PLUG_IN_VIOLATIONS

Since 12.2.0.1, CDB can have same or more options installed than PDB

There are warning messages in PDB_PLUG_IN_VIOLATIONS :

SQL>select NAME,CAUSE,MESSAGE,TYPE,CON_ID 
      from PDB_PLUG_IN_VIOLATIONS;

NAME         CAUSE  MESSAGE                                                                                              
----------  ------- ---------------------------------------------
                                                 TYPE      CON_ID
------------ ---------- -------------------- ------------------
RACTESTPDB   OPTION     Database option DV mismatch: PDB installed version NULL. CDB installed version 12.2.0.1.0.           WARNING    3 
RACTESTPDB   OPTION     Database option OLS mismatch: PDB installed version NULL. CDB installed version 12.2.0.1.0.          WARNING    3
RACTESTPDB   OPTION     Database option SDO mismatch: PDB installed version NULL. CDB installed version 12.2.0.1.0.          WARNING    3
RACTESTPDB   APEX       APEX mismatch: PDB installed version NULL CDB installed version 5.0.4.00.12                          WARNING    3

Check what options installed in CDB ( CDB$ROOT ):

SQL> show con_name

CON_NAME
--------------
CDB$ROOT

SQL> select COMP_NAME from dba_registry;

COMP_NAME
----------------------------------------
Oracle Database Catalog Views
Oracle Database Packages and Types
JServer JAVA Virtual Machine
Oracle XDK
Oracle Database Java Packages
Oracle Real Application Clusters
Oracle XML Database
Oracle Workspace Manager
Oracle Text
Oracle Multimedia
Spatial                    <--- Not in PDB
Oracle Label Security      <--- Not in PDB
Oracle Application Express <--- Not in PDB
Oracle Database Vault      <--- Not in PDB

14 rows selected.

Check what options installed in PDB:

SQL> show con_name

CON_NAME
-----------------
RACTESTPDB

SQL> select COMP_NAME from dba_registry;

COMP_NAME
-------------------------------------
JServer JAVA Virtual Machine
Oracle Database Catalog Views
Oracle Database Java Packages
Oracle Database Packages and Types
Oracle Multimedia
Oracle Real Application Clusters
Oracle Text
Oracle Workspace Manager
Oracle XDK
Oracle XML Database

10 rows selected.

We can see CDB has four more options installed than PDB.

CONCLUSION

It is accepted that CDB not necessarily has same options installed as PDB. But CDB should have same or more options installed than PDB.

So in this situation, the warning messages in PDB_PLUG_IN_VIOLATIONS can be ignored.

How to Configure SHMMAX SHMMNI SHMALL ?

Every DBA is so familiar with those three kernel parameters in Linux — SHMMAX, SHMMNI and SHMALL. Most of the time, the default values are used by majority clients. But sometime, those three parameters need to be reconfigured to reflect the system memory sizing and total databases SGA sizing.

What are SHMMAX SHMMNI SHMALL ?

SHMMAX : The maximum size in bytes of a single shared memory segment that a Linux process can allocate in its virtual address space.

SHMMNI: The system wide maximum number of shared memory segments.

SHMALL: The total amount of shared memory pages that can be used system wide.

How to check current SHMMAX SHMMNI SHMALL values ?

run “ipcs -lm”

$ipcs -lm

------ Shared Memory Limits --------
max number of segments = 4096   <-------------- SHMMNI
max seg size (kbytes) = 273678336  <----------- SHMMAX
max total shared memory (kbytes) = 371195904 <- SHMALL * PAGE_SIZE
min seg size (bytes) = 1

Check the values in proc file system.

$ getconf PAGE_SIZE
4096

$ cat /proc/sys/kernel/shmall
92798976


$ cat /proc/sys/kernel/shmmax
280246616064


$ cat /proc/sys/kernel/shmmni
4096

How to change SHMMAX SHMMNI SHMALL values ?

Change them in the proc file system without reboot

# echo 92798976 > /proc/sys/kernel/shmall

# echo 4096 >  /proc/sys/kernel/shmmni

# echo 280246616064 > /proc/sys/kernel/shmmax

Use sysctl without reboot

# sysctl -w kernel.shmall=92798976

# sysctl -w kernel.shmmni=4096

#  sysctl -w kernel.shmmax=280246616064

To make the change permanent

This file is used during the boot process.

Edit /etc/sysctl.conf with below lines:

kernel.shmmni = 4096
kernel.shmall = 92798976
kernel.shmmax = 280246616064

What are the optimal values for SHMMAX SHMMNI SHMALL ?

SHMMAX : Either SGA size of any individual database or half size of the system memory, whichever is the higher.

SHMMNI: Normally default value 4096 should be sufficient.

SHMALL: The sum of all the SGAs on the system, divided by the page size. The page size can be gotton by :

 $ getconf PAGE_SIZE
 4096