Blog

OEM Agent Throws Error “Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620”

Apply some hidden parameters into emd.properties file, then bounce the agent. It might fix some internal issues.

In OEM 13c, a couple of OEM agents reported some internal errors  like :

Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620

Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:621

From: oracle 
Sent: Monday, 13 February 2017 10:41 AM
To: 
Subject: EM Event: Critical:ractest:3872 - Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620.

Host=ractest
Target type=Agent 
Target name=ractest:3872 
Categories=Diagnostics 
Message=Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620. 
Severity=Critical 
...
..
.
Update Details:
Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620.

Extracted from gcagent.log :

2017-02-13 10:10:00,329 [256:7761FFD4] WARN - IntervalSchedule: Skip schedule [rac_database:RACTEST:observer_11g] - skipping due to execution time exceeding interval
2017-02-13 10:10:00,329 [257:B85D2018:GC.Executor.6 (oracle_database:RACTEST:Response)] WARN - IntervalSchedule: Skip schedule [oracle_database:RACTEST:Response] - skipping due to execution time exceeding interval
2017-02-13 10:10:00,329 [258:5A4306F7:GC.Executor.7 (rac_database:RACTEST:observer_11g)] WARN - Action result processing failure rac_database.RACTEST::observer_11g
java.lang.InterruptedException 
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)
...
..
.
oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:620)
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:643)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at oracle.sysman.gcagent.task.TaskFutureImpl.run1(TaskFutureImpl.java:380)
...
..
. 

 at java.lang.Thread.run(Thread.java:662)
2017-02-13 10:10:00,331 [257:GC.Executor.6] ERROR - Critical error:
oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:620)
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:643)

WORKAROUND

Subscribe to get access

Read more of this content when you subscribe today.

ME ( Metric Extension ) Metric Is Not Collected For Some Databases

By default, ME metric will not be collected if associated target has a severity on response:state metric. So override this default by explicitly choosing Advanced option while creating ME.

For some databases, ME ( Metric Extension ) is not collecting data. But for the rest databases, it is working fine.

me_not_collecting

Checking the ME ( Metric Extension ) definition and found an advanced option is not selected.

selection

After creating a new version of this ME ( Metric Extension ) by checking this mentioned option, it works fine.

The reason is  this database has frequent Warning/Critical response metric alert. like  :

…..
….

Event Type=Metric Alert
Event name=Server_Adaptive_Threshold_Metric:instance_efficiency__response_time_pt
Metric Group=Server_Adaptive_Threshold_Metric
Metric=Response Time Per Txn
Metric value=258.497253691275
Key Value=SYSTEM
Rule Name=DEFAULT_RULESET_FOR_ALL_TARGETS,METRIC_ALERT_INCIDENT_CREATION
Rule Owner=
Update Details:
Metrics “Response Time Per Txn” is at 258.497
Incident created by rule (Name = Incident management rule set for all targets, Create incident for critical metric alerts [System generated rule]).

CRS-4535: Cannot communicate with Cluster Ready Services

This error is caused by crsd not running, while both cssd and evmd processes are running. Also all the database instances are running.

Environment is 12.1.0.2 GI and 11.2.0.4 databases.

$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services  
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
$ps -eaf|grep crsd 

No process found

Try to start up CRS:

# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

Subscribe to get access

Read more of this content when you subscribe today.

Oracle Proxy User Configurations

Oracle Proxy User allows to access a schema via a different username/password combination, which is done by using the GRANT CONNECT THROUGH clause on the destination user.

A proxy user is a user who is able to connect as another user without password required. In this example, user “TESTUSER” needs connect as user “APPDBA” without knowing the password of user “APPDBA”.

  • Create destination user with appropriate role(s).
SQL> create user appdba identified by Password;
User created.

SQL> grant connect,resource,dba to appdba;
Grant succeeded.
  • Create the proxy user account.
SQL> create user testuser identified by PasswordProxy;
User created.
  • Alter the destinatiom user ( APPDBA) to connect through the proxy user (TESTUSER).
SQL> alter user appdba grant connect through testuser;
User altered.
  • Test the connection for proxy user.
SQL> select count(*) from dba_users;

 COUNT(*)
----------
 31


SQL> connect testuser[appdba]/PasswordProxy
Connected.
SQL> show user
USER is "APPDBA"


SQL> select * from proxy_users;

PROXY      CLIENT       AUT FLAGS
---------- ------------ --- -----------------------------------
TESTUSER   APPDBA       NO  PROXY MAY ACTIVATE ALL CLIENT ROLES


SQL> select sys_context('userenv','session_user') session_user,
            sys_context('userenv','session_schema') session_schema,
            sys_context('userenv','current_schema') current_schema,
            sys_context('userenv','proxy_user') proxy_user
    from dual;

SESSION_USER    SESSION_SCHEMA   CURRENT_SCHEMA PROXY_USER
--------------- --------------- --------------- ---------------
APPDBA          APPDBA          APPDBA          TESTUSER


SQL>select USERNAME,ACTION_NAME,COMMENT_TEXT,PROXY_SESSIONID 
      from dba_audit_trail 
      where PROXY_SESSIONID is not null

USERNAME ACTION_NAM COMMENT_TEXT               PROXY_SESSIONID
-------- ---------- -------------------------- ---------------
APPDBA   LOGON      Authenticated by: PROXY;    222208723
                    EXTERNAL NAME: oracle      


SQL> select USERNAME,ACTION_NAME,COMMENT_TEXT ,PROXY_SESSIONID 
     from dba_audit_trail 
     where SESSIONID=222208723;

USERNAME  ACTION_NAME               COMMENT_TEXT     PROXY_SESSIONID
--------  ------------------------  ---------------- ---------------
TESTUSER  PROXY AUTHENTICATION ONLY Authenticated by: 
                                    DATABASE

  • If you don’t want the proxy user (testuser ) has all the privilege of user APPDBA, alter the proxy user by giving the certain roles ( CONNECT, RESOURCE ). In this case, DBA_USERS is not accessible to user TESTUSER.
SQL> alter user appdba grant connect through testuser with role connect, resource;

User altered.

SQL> connect testuser[appdba]/PasswordProxy
Connected.

SQL> select count(*) from dba_users;
select count(*) from dba_users
 *
ERROR at line 1:
ORA-00942: table or view does not exist
  • The proxy authentication can be revoked using the following command.
ALTER USER appdba REVOKE CONNECT THROUGH testuser;

How to Apply Database Component Of a Patch Onto ORMB Database

Always make a backup of CISADM schema or whole database before applying database component of a patch onto ORMB database.

Here is an example of applying an ORMB patch onto ORMB Oracle database.  There working environment is as below ;

Oracle Database 12c
Oracle Client 32bit 12c
ORMB 2.5.0.3
JDK 8

Before applying patching, make a backup of database in any way you prefer:

Set the environment variables:

C:\>set TNS_ADMIN=d:\oracle
C:\>set TOOLSBIN=d:\dbpatch_tools\bin
C:\>set JAVA_HOME=d:\java

Ensure to be able to connect to the database from the application server workstation:

C:\>tnsping ormb
TNS Ping Utility for 32-bit Windows: Version 12.1.0.2.0 - Production on 13-JAN-2017 10:19:11
Copyright (c) 1997, 2014, Oracle. All rights reserved.
Used parameter files:
d:\oracle\sqlnet.ora
Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ractest-scan.ractest)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVIC
E_NAME = ORMB)))
OK (10 msec)

Unzip the patch files, change to the CDXPatch directory and execute the ouafDatabasePatch.cmd utility. The utility will prompt you for the value of the following parameters:

if it prompts you for the target database type, enter O for Oracle.
Enter the username that owns the schema (e.g., CISADM)
The password for the user (in silent mode)(e.g., XXXXXXX)
Enter the name of the Oracle Database Connection String : //ractest-scan.ractest:1521/ORMB

Below is the output from applying patch 25054594 ( with three child patches -24586912, 24604908, 24963398  ).

D:\PATCHES\p25054594_25030\MultiPlatform\V2.5.0.3.0-25054594_MultiPlatform\database\ORACLE\CDXPatch>ouafDatabasePatch.cmd
"CMDLINE::: d:\java\bin\java.exe -cp d:\dbpatch_tools\lib\*;d:\dbpatch_tools\config com.oracle.ouaf.database.patch.OUAFPatch "
Enter the target database type (O/M/D) [O]: O
Enter the username that owns the schema: CISADM
Enter the password for the CISADM user: XXXXXXX
Enter the name of the Oracle Database Connection String: //ractest-scan.ractest:1521/ORMB
Target Schema is a Production Schema
Ready to process patches, Do you want to continue? (Y/N): Y
Working Directory: ORMB001
***********************************
Setting up language file: ORMB001\CDXPatch.lang
exit value: 0

Applying 24586912 ...
Writing to log file: ORMB001\log24586912.log
-----------------------------------------------------------
--Applying patch 24586912 at 01-13-2017 14:20:49using $LastChangedRevision: 42479 $
---------------------------------------------------------------
--Copying language information
-----------------------------------------------------------
--Patch 24586912 applied successfully at 01-13-2017 14:21:20
---------------------------------------------------------------

Applying 24604908 ...
Writing to log file: ORMB001\log24604908.log
-----------------------------------------------------------
--Applying patch 24604908 at 01-13-2017 14:21:20using $LastChangedRevision: 42479 $
---------------------------------------------------------------
--Copying language information
-----------------------------------------------------------
--Patch 24604908 applied successfully at 01-13-2017 14:21:31
---------------------------------------------------------------

Applying 24963398 ...
Writing to log file: ORMB001\log24963398.log
-----------------------------------------------------------
--Applying patch 24963398 at 01-13-2017 14:21:31using $LastChangedRevision: 42479 $
---------------------------------------------------------------
--Copying language information
-----------------------------------------------------------
--Patch 24963398 applied successfully at 01-13-2017 14:21:31
---------------------------------------------------------------
Patch applied successfully...

Check the successful installations of the patches as per How to Check the Successful Installation of a Database Patch Onto ORMB Database

We can see only the child patches ( 24586912, 24604908, 24963398 ) are stored in repository. The parent patch ID is NOT installed into database patch repository )

SQL>  select sr_no,run_status_flg,END_DTTM 
        from cisadm.ci_ut_instl 
       where sr_no in ( '25054594','24586912','24604908','24963398') 
         and run_status_flg = '02' 
         and END_DTTM IS Not NULL;

SR_NO                          RU END_DTTM
----------------               -- ---------
24586912                       02 13-JAN-17
24604908                       02 13-JAN-17
24963398                       02 13-JAN-17

Finally run Security utility as per How to Run Security Utility in ORMB Multiple Instances Environment