OEM Blackout Stuck in “Stop Pending” Status

OEM OMS Version13.2.0.0.0

There is one blackout stuck in “Stop Pending” Status, when manually stopped it with errors:

Error: The blackout is already pending stop

1) Implemented “Blackout Stuck in ‘Stop Pending’ Status ( Doc ID 2014040.1 )”, but it does not work.

$ ./emctl set property -name oracle.sysman.core.conn.maxConnForJobWorkers -value 100 
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved. 
SYSMAN password: 
Property oracle.sysman.core.conn.maxConnForJobWorkers has been set to value 100 for all Management Servers 
OMS restart is required to reflect the new property value 

$./emctl get property -name oracle.sysman.core.conn.maxConnForJobWorkers 
Oracle Enterprise Manager Cloud Control 13c Release 2 
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved. 
SYSMAN password: 
Value for property oracle.sysman.core.conn.maxConnForJobWorkers at Global level is 100 

$cd <OMS_HOME>/bin
$./emctl stop oms -all
$./emctl start oms

2)Tried to use emcli to stop or delete it, still no luck.

$emcli login -username="sysman" -password="password"

$ emcli get_blackout_details -name="DB Weekend Pager Notification Blackout"
Status            Status ID Run Jobs Next Start Duration Reason                            Frequency Repeat Days Months Start Time          End Time TZ Region        TZ Offset Type
Stop In Progress  5         yes      none       57:00    Apps: New Application Monitoring  weekly    none   [7]  none   2017-06-03 23:00:00 none     Australia/Sydney +11:00    Blackout

$emcli stop_blackout -name="DB Weekend Pager Notification Blackout"
Error: The blackout is already pending stop

$emcli delete_blackout -name="DB Weekend Pager Notification Blackout"
Error: Cannot delete a blackout that has not ended or was not stopped

3) Download and Upgrade EMDIAG REPVFY KIT as per  EM 13c, 12c: EMDIAG REPVFY Kit for Cloud Control – Download, Install/De-Install and Upgrade (Doc ID 1426973.1).

$ unzip repvfy12_2017.1130.zip -d $ORACLE_HOME/emdiag

$ cd <EMDIAG_HOME>/bin
$ chmod +x repvfy

$ ./repvfy upgrade 
Please enter the SYSMAN password: xxxxxx
...
..
.
COMPONENT            INFO
-------------------- --------------------
EMDIAG Version       17.11.30
EMDIAG Edition       2
Repository Version   13.2.0.0.0
Database Version     12.1.0.2.0
Test Version         17.12.05
Repository Type      CENTRAL
Verify Tests         589
Object Tests         235
Deployment           MEDIUM

9 rows selected.

4) Tried to use EMDIAG to stop or kill this blackout, still no luck.

SQL> select BLACKOUT_GUID,BLACKOUT_NAME from mgmt_blackouts where BLACKOUT_NAME like 'DB%Black%';

BLACKOUT_GUID                    BLACKOUT_NAME
-------------------------------- ---------------------------------------------
32665ABC0B4F323575601C3DF8C77BE3 DB Weekend Pager Notification Blackout


SQL> exec MGMT_BLACKOUT.stop_blackout('32665ABC0B4F323575601C3DF8C77BE3',null);
BEGIN MGMT_BLACKOUT.stop_blackout('32665ABC0B4F323575601C3DF8C77BE3',null); END;
*
ERROR at line 1:
ORA-20706: The blackout is already pending stop
ORA-06512: at "SYSMAN.MGMT_BLACKOUT_ENGINE_NG", line 2701
ORA-06512: at "SYSMAN.MGMT_BLACKOUT", line 145
ORA-06512: at line 1

SQL> exec MGMT_BLACKOUT.delete_blackout('32665ABC0B4F323575601C3DF8C77BE3');
BEGIN MGMT_BLACKOUT.delete_blackout('32665ABC0B4F323575601C3DF8C77BE3'); END;
*
ERROR at line 1:
ORA-20704: Cannot delete a blackout that has not ended or was not stopped
ORA-06512: at "SYSMAN.MGMT_BLACKOUT_ENGINE_NG", line 2818
ORA-06512: at "SYSMAN.MGMT_BLACKOUT", line 172
ORA-06512: at line 1

5) Check and Update Repository Tables As per Oracle Support Instructions.

a) Check BLACKOUT_STATUS in Table EM_BLACKOUTS_NG.
BLACKOUT_STATUS =4    ACTIVE
BLACKOUT_STATUS=5     STOP PENDING
BLACKOUT_STATUS=10   STOPPED
BLACKOUT_STATUS=11   ENDED

SQL>select BLACKOUT_GUID,BLACKOUT_NAME,BLACKOUT_STATUS 
    from EM_BLACKOUTS_NG_E
    where BLACKOUT_GUID='32665ABC0B4F323575601C3DF8C77BE3';

BLACKOUT_GUID                    BLACKOUT_NAME              BLACKOUT_STATUS
-------------------------------- -------------------------- ---------------
32665ABC0B4F323575601C3DF8C77BE3 DB Weekend Pager          5
                                 Notification Blackout

b) Manually Update both EM_BLACKOUTS_NG_E and EM_BLACKOUT_TARGET_STATE_E.

SQL>update EM_BLACKOUTS_NG_E set BLACKOUT_STATUS=10 
    where BLACKOUT_GUID='32665ABC0B4F323575601C3DF8C77BE3';
1 row updated.

SQL> commit;
Commit complete.

SQL> update EM_BLACKOUT_TARGET_STATE_E set BLACKOUT_STATUS=10 
     where BLACKOUT_GUID='32665ABC0B4F323575601C3DF8C77BE3';

60 rows updated.

SQL> commit;
Commit complete.

5) Check from OEM console again, the problem is gone.

2 thoughts on “OEM Blackout Stuck in “Stop Pending” Status”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.