How to Add a Disk into ASM DiskGroup Safely

For a client, there are situations when adding a disk into ASM disk group failed, finally rebooting the server is required to resolve this issue.

According to Oracle support suggestion, we can create a test diskgroup by using this new disk, if everything is successful, then we can drop this test diskgroup, and add the new disk into target diskgroup.

It is a good practice specially for critical production environment.

Check the new disk is a “CANDIDATE”

$sqlplus / as sysasm 


SQL>select INST_ID,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,
MODE_STATUS,STATE,REDUNDANCY,OS_MB,OS_MB,TOTAL_MB,NAME,
FAILGROUP,PATH
from gv$asm_disk
where PATH like '/dev/mapper/prod_data_303p1'
order by 1,2,3;

Create a Test Diskgroup

Create a test diskgroup “TEMP_TEST”, and check disk status is “MEMBER”

SQL> CREATE DISKGROUP TEMP_TEST EXTERNAL REDUNDANCY disk '/dev/mapper/prod_data_303p1' NAME TEMP_TEST_0001;


Diskgroup created.
SQL>select INST_ID,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,
MODE_STATUS,STATE,REDUNDANCY,OS_MB,OS_MB,TOTAL_MB,NAME,
FAILGROUP,PATH
from gv$asm_disk
where PATH like '/dev/mapper/prod_data_303p1'
order by 1,2,3;

Mount the Test Diskgroup on Other Nodes

Mount the test diskgrop on all the rest ASM nodes to make sure all working fine.

$sqlplus / as sysasm

SQL>set pagesize 100
SQL>set linesize 300
SQL>select INST_ID,GROUP_NUMBER,NAME,STATE,TOTAL_MB,FREE_MB,
COMPATIBILITY,DATABASE_COMPATIBILITY
from gv$asm_diskgroup
where name='TEMP_TEST'
order by 1,2;
SQL> alter diskgroup TEMP_TEST mount;


Diskgroup altered.
SQL>select INST_ID,GROUP_NUMBER,NAME,STATE,TOTAL_MB,FREE_MB,
COMPATIBILITY,DATABASE_COMPATIBILITY
from gv$asm_diskgroup
where name='TEMP_TEST'
order by 1,2;

Dismount the Test Diskgroup on Other Nodes

Dismount the test diskgroup on all the other nodes except the first node for dropping in the next step.

SQL> alter diskgroup TEMP_TEST dismount;


Diskgroup altered.

SQL>select INST_ID,GROUP_NUMBER,NAME,STATE,TOTAL_MB,FREE_MB,
COMPATIBILITY,DATABASE_COMPATIBILITY
from gv$asm_diskgroup
where name='TEMP_TEST'
order by 1,2;

Drop the Test Diskgroup

Drop the test diskgroup on first node where the test diskgroup is still mounted.

SQL> DROP DISKGROUP  TEMP_TEST ;


Diskgroup dropped.

And check the new disk header status is “FORMER”, which is available for adding into target diskgroup now.

SQL>select INST_ID,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,
MODE_STATUS,STATE,REDUNDANCY,OS_MB,OS_MB,TOTAL_MB,NAME,
FAILGROUP,PATH
from gv$asm_disk
where PATH like '/dev/mapper/prod_data_303p1'
order by 1,2,3;

Add Disk into Target Diskgroup

Now it is time to add the new disk into target diskgroup.

SQL> set time on

08:08:17 SQL> ALTER DISKGROUP PROD_DATA ADD DISK '/dev/mapper/prod_data_303p1' NAME PROD_DATA_0012;

Diskgroup altered.

Then check disk and diskgroup status

09:09:01 SQL>select INST_ID,GROUP_NUMBER,DISK_NUMBER,MOUNT_STATUS,
HEADER_STATUS,MODE_STATUS,STATE,REDUNDANCY,
OS_MB,OS_MB,TOTAL_MB,NAME,FAILGROUP,PATH
from gv$asm_disk
where PATH like '%prod_data%'
order by 1,2,3;
09:09:14 SQL>  select INST_ID,GROUP_NUMBER,NAME,STATE,TOTAL_MB,FREE_MB,
COMPATIBILITY,DATABASE_COMPATIBILITY
from gv$asm_diskgroup
where name='PROD_DATA' ;

Monitor and Check Rebalance

00:10:35 SQL> select GROUP_NUMBER,OPERATION,STATE,POWER,ACTUAL,
SOFAR,EST_WORK,EST_RATE,EST_MINUTES
from v$asm_operation;

ASM disk group state shows as ‘CONNECTED’

As we know, normally disk group state is ‘MOUNTED’ or ‘DISMOUNTED’, etc. Today, when I ran the following query, two disk group state shows as ‘CONNECTED’:

SQL> select NAME,STATE from v$asm_diskgroup ;

NAME       STATE 
---------- ---------
DATA       CONNECTED                                                      
RECO       CONNECTED 
...
..
.

According to Oracle doc, ‘CONNECTED’ means disk group is in use by the database instance:

STATEVARCHAR2(11)State of the disk group relative to the instance:
CONNECTED – Disk group is in use by the database instance
BROKEN – Database instance lost connectivity to the Oracle ASM instance that mounted the disk group
UNKNOWN – Oracle ASM instance has never attempted to mount the disk group
DISMOUNTED – Disk group was cleanly dismounted by the Oracle ASM instance following a successful mount
MOUNTED – Instance is successfully serving the disk group to its database clients
QUIESCING – CRSCTL utility attempted to dismount a disk group that contains the Oracle Cluster Registry (OCR). The disk group cannot be dismounted until Cluster Ready Services (CRS) exits, because the disk group contains the OCR.

database spfile created in wrong place under ASM

SYMPTOMS

DBA is trying to create a new spfile from a pfile as following :

SQL> create SPFILE='+DATAC1/TESTDB/' from pfile='/home/oracle/testdb.pfile';
File created.

Under ASM , the spile is link to the wrong location under ASM:

ASMCMD> ls -alr
Type Redund Striped Time Sys Name
N spfile =>
+DATAC1/DB_UNKNOWN/PARAMETERFILE/SPFILE.272.613162051
ASMCMD>

OR

ASMCMD> ls -alr
Type Redund Striped Time Sys Name
N spfile =>
+DATAC1/ASM/PARAMETERFILE/spfile.4186.1126528877
ASMCMD>

CAUSE

This is because the database instance has no open client session to the ASM instance at the time of the ‘create spfile’ command. That is, the ASM instance is not aware of the db name and therefore uses “DB_UNKNOWN” or ‘ASM’.

SOLUTION

When database in OPEN or MOUNT status, make sure  there is at least one connection to the ASM instance. 

SQL>select * from v$asm_client;

GROUP_NUMBER INSTANCE_NAME
------------ ----------------------------------------------------------------
DB_NAME  CLUSTER_NAME                    STATUS
-------- ------------------------------- ------------
SOFTWARE_VERSION
------------------------------------------------------------
COMPATIBLE_VERSION                                               CON_ID
------------------------------------------------------------ ----------
           0 +ASM1
TESTDB_ usa1234clu07cb8                 CONNECTED
19.0.0.0.0
19.0.0.0.0                                                            0

— Create spfile from a pfile

SQL>  create spfile='+datac1' from pfile='/home/oracle/testdb.pfile';

File created.

— check spfile is created in the right directory now

ASMCMD> pwd
+datac1
ASMCMD> cd TESTDB/
ASMCMD> ls PARAMETERFILE/
spfile.4186.1126530135

— configure TESTDB database with new spfile

$ srvctl modify  database -d TESTDB -spfile  '+DATAC1/TESTDB/PARAMETERFILE/spfile.4186.1126530135'

asmcmd pwcopy password file into ‘DB_UNKNOWN’ directory

When trying to copy password file of primary databases for standby database, unfortunately the password file was located into a directory called “DB_UNKNOWN”.

$ asmcmd pwcopy /tmp/orapwTESTDB +DATAC1/TESTDB/PASSWORD/
copying /tmp/orapwTESTDB -> +DATAC1/TESTDB/PASSWORD/orapwTESTDB
$ asmcmd
ASMCMD> cd +DATAC1/TESTDB/PASSWORD/
ASMCMD> ls -lt
Type      Redund  Striped  Time             Sys  Name
PASSWORD  HIGH    COARSE   JUL 16 14:00:00  Y    pwdtestdb.372.1078064659
PASSWORD  HIGH    COARSE   JUL 16 17:00:00  N    orapwtestdb => +DATAC1/DB_UNKNOWN/PASSWORD/pwddb_unknown.266.1078077567

SOLUTION

Subscribe to get access

Read more of this content when you subscribe today.

Sorry! This product is not available for purchase at this time.

Finally you need modify the database configuration to use the new password file:

$ srvctl modify database -d TESTDB  -pwfile +DATAC1/TESTDB/PASSWORD/pwdtestdb.372.1078078773

How to Relocate ASM ControlFiles

After database name has been changed, the controlfiles are still staying in the directory of old database, so we need relocate them into directory of new database name.

Subscribe to get access

Read more of this content when you subscribe today.