Blog

OEM Agent Throws “Internal error detected: java.lang.IllegalStateException:oracle.sysman.gcagent.target.interaction.execution.ConfigStateMgr:827”

There are some issues for agent to run host_storage collection every 24 hours.

Some of the OEM agents throw the internal errors in OEM alerts:

From: oracle 
Sent: Tuesday, 14 February 2017 6:08 PM
To: 
Subject: EM Event: Critical:racnode1:3872 - Internal error detected: java.lang.IllegalStateException:oracle.sysman.gcagent.target.interaction.execution.ConfigStateMgr:827.
...
..
.

Error messages from agent log file:

2017-02-14 18:07:32,288 [265:GC.Executor.6 (host:racmode1:host_storage) (host:racnode1:host_storage:storage_reporting_data)] ERROR - null
javax.xml.bind.UnmarshalException
 - with linked exception:
[org.xml.sax.SAXParseException: <Line 1, Column 9207>: XML-20221: (Fatal Error) Invalid char in text.]

Workaround:

1) Remove <agent_inst>/sysman/emd/state/configstate/host/racnode1/storage_*.xml files.

2) Restart the agent.

This will restart the collection of the storage data. As these data are collected every 24 hours, please wait for at least 24 hours to check the report.

You can also run the collection manually:

$ emctl status agent scheduler | grep host | grep -i storage
2017-02-16 18:25:52.117 : host:racnode1:host_storage
2017-02-16 18:34:37.318 : host:racnode1:HostStorageSupport

$ emctl control agent runCollection racnode1:host host_storage
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully


$ emctl control agent runCollection racnode1:host HostStorageSupport
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully

OEM Agent Throws Error “Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620”

Apply some hidden parameters into emd.properties file, then bounce the agent. It might fix some internal issues.

In OEM 13c, a couple of OEM agents reported some internal errors  like :

Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620

Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:621

From: oracle 
Sent: Monday, 13 February 2017 10:41 AM
To: 
Subject: EM Event: Critical:ractest:3872 - Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620.

Host=ractest
Target type=Agent 
Target name=ractest:3872 
Categories=Diagnostics 
Message=Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620. 
Severity=Critical 
...
..
.
Update Details:
Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620.

Extracted from gcagent.log :

2017-02-13 10:10:00,329 [256:7761FFD4] WARN - IntervalSchedule: Skip schedule [rac_database:RACTEST:observer_11g] - skipping due to execution time exceeding interval
2017-02-13 10:10:00,329 [257:B85D2018:GC.Executor.6 (oracle_database:RACTEST:Response)] WARN - IntervalSchedule: Skip schedule [oracle_database:RACTEST:Response] - skipping due to execution time exceeding interval
2017-02-13 10:10:00,329 [258:5A4306F7:GC.Executor.7 (rac_database:RACTEST:observer_11g)] WARN - Action result processing failure rac_database.RACTEST::observer_11g
java.lang.InterruptedException 
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)
...
..
.
oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:620)
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:643)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at oracle.sysman.gcagent.task.TaskFutureImpl.run1(TaskFutureImpl.java:380)
...
..
. 

 at java.lang.Thread.run(Thread.java:662)
2017-02-13 10:10:00,331 [257:GC.Executor.6] ERROR - Critical error:
oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:620)
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:643)

WORKAROUND

Subscribe to get access

Read more of this content when you subscribe today.

ME ( Metric Extension ) Metric Is Not Collected For Some Databases

By default, ME metric will not be collected if associated target has a severity on response:state metric. So override this default by explicitly choosing Advanced option while creating ME.

For some databases, ME ( Metric Extension ) is not collecting data. But for the rest databases, it is working fine.

me_not_collecting

Checking the ME ( Metric Extension ) definition and found an advanced option is not selected.

selection

After creating a new version of this ME ( Metric Extension ) by checking this mentioned option, it works fine.

The reason is  this database has frequent Warning/Critical response metric alert. like  :

…..
….

Event Type=Metric Alert
Event name=Server_Adaptive_Threshold_Metric:instance_efficiency__response_time_pt
Metric Group=Server_Adaptive_Threshold_Metric
Metric=Response Time Per Txn
Metric value=258.497253691275
Key Value=SYSTEM
Rule Name=DEFAULT_RULESET_FOR_ALL_TARGETS,METRIC_ALERT_INCIDENT_CREATION
Rule Owner=
Update Details:
Metrics “Response Time Per Txn” is at 258.497
Incident created by rule (Name = Incident management rule set for all targets, Create incident for critical metric alerts [System generated rule]).

CRS-4535: Cannot communicate with Cluster Ready Services

This error is caused by crsd not running, while both cssd and evmd processes are running. Also all the database instances are running.

Environment is 12.1.0.2 GI and 11.2.0.4 databases.

$ crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services  
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
$ps -eaf|grep crsd 

No process found

Try to start up CRS:

# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.

Subscribe to get access

Read more of this content when you subscribe today.

Oracle Proxy User Configurations

Oracle Proxy User allows to access a schema via a different username/password combination, which is done by using the GRANT CONNECT THROUGH clause on the destination user.

A proxy user is a user who is able to connect as another user without password required. In this example, user “TESTUSER” needs connect as user “APPDBA” without knowing the password of user “APPDBA”.

  • Create destination user with appropriate role(s).
SQL> create user appdba identified by Password;
User created.

SQL> grant connect,resource,dba to appdba;
Grant succeeded.
  • Create the proxy user account.
SQL> create user testuser identified by PasswordProxy;
User created.
  • Alter the destinatiom user ( APPDBA) to connect through the proxy user (TESTUSER).
SQL> alter user appdba grant connect through testuser;
User altered.
  • Test the connection for proxy user.
SQL> select count(*) from dba_users;

 COUNT(*)
----------
 31


SQL> connect testuser[appdba]/PasswordProxy
Connected.
SQL> show user
USER is "APPDBA"


SQL> select * from proxy_users;

PROXY      CLIENT       AUT FLAGS
---------- ------------ --- -----------------------------------
TESTUSER   APPDBA       NO  PROXY MAY ACTIVATE ALL CLIENT ROLES


SQL> select sys_context('userenv','session_user') session_user,
            sys_context('userenv','session_schema') session_schema,
            sys_context('userenv','current_schema') current_schema,
            sys_context('userenv','proxy_user') proxy_user
    from dual;

SESSION_USER    SESSION_SCHEMA   CURRENT_SCHEMA PROXY_USER
--------------- --------------- --------------- ---------------
APPDBA          APPDBA          APPDBA          TESTUSER


SQL>select USERNAME,ACTION_NAME,COMMENT_TEXT,PROXY_SESSIONID 
      from dba_audit_trail 
      where PROXY_SESSIONID is not null

USERNAME ACTION_NAM COMMENT_TEXT               PROXY_SESSIONID
-------- ---------- -------------------------- ---------------
APPDBA   LOGON      Authenticated by: PROXY;    222208723
                    EXTERNAL NAME: oracle      


SQL> select USERNAME,ACTION_NAME,COMMENT_TEXT ,PROXY_SESSIONID 
     from dba_audit_trail 
     where SESSIONID=222208723;

USERNAME  ACTION_NAME               COMMENT_TEXT     PROXY_SESSIONID
--------  ------------------------  ---------------- ---------------
TESTUSER  PROXY AUTHENTICATION ONLY Authenticated by: 
                                    DATABASE

  • If you don’t want the proxy user (testuser ) has all the privilege of user APPDBA, alter the proxy user by giving the certain roles ( CONNECT, RESOURCE ). In this case, DBA_USERS is not accessible to user TESTUSER.
SQL> alter user appdba grant connect through testuser with role connect, resource;

User altered.

SQL> connect testuser[appdba]/PasswordProxy
Connected.

SQL> select count(*) from dba_users;
select count(*) from dba_users
 *
ERROR at line 1:
ORA-00942: table or view does not exist
  • The proxy authentication can be revoked using the following command.
ALTER USER appdba REVOKE CONNECT THROUGH testuser;