OEM Agent Throws “Internal error detected: java.lang.IllegalStateException:oracle.sysman.gcagent.target.interaction.execution.ConfigStateMgr:827”

There are some issues for agent to run host_storage collection every 24 hours.

Some of the OEM agents throw the internal errors in OEM alerts:

From: oracle 
Sent: Tuesday, 14 February 2017 6:08 PM
To: 
Subject: EM Event: Critical:racnode1:3872 - Internal error detected: java.lang.IllegalStateException:oracle.sysman.gcagent.target.interaction.execution.ConfigStateMgr:827.
...
..
.

Error messages from agent log file:

2017-02-14 18:07:32,288 [265:GC.Executor.6 (host:racmode1:host_storage) (host:racnode1:host_storage:storage_reporting_data)] ERROR - null
javax.xml.bind.UnmarshalException
 - with linked exception:
[org.xml.sax.SAXParseException: <Line 1, Column 9207>: XML-20221: (Fatal Error) Invalid char in text.]

Workaround:

1) Remove <agent_inst>/sysman/emd/state/configstate/host/racnode1/storage_*.xml files.

2) Restart the agent.

This will restart the collection of the storage data. As these data are collected every 24 hours, please wait for at least 24 hours to check the report.

You can also run the collection manually:

$ emctl status agent scheduler | grep host | grep -i storage
2017-02-16 18:25:52.117 : host:racnode1:host_storage
2017-02-16 18:34:37.318 : host:racnode1:HostStorageSupport

$ emctl control agent runCollection racnode1:host host_storage
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully


$ emctl control agent runCollection racnode1:host HostStorageSupport
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD runCollection completed successfully

OEM Agent Throws Error “Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620”

Apply some hidden parameters into emd.properties file, then bounce the agent. It might fix some internal issues.

In OEM 13c, a couple of OEM agents reported some internal errors  like :

Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620

Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:621

From: oracle 
Sent: Monday, 13 February 2017 10:41 AM
To: 
Subject: EM Event: Critical:ractest:3872 - Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620.

Host=ractest
Target type=Agent 
Target name=ractest:3872 
Categories=Diagnostics 
Message=Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620. 
Severity=Critical 
...
..
.
Update Details:
Internal error detected: oracle.sysman.gcagent.task.TaskZombieException:oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask:620.

Extracted from gcagent.log :

2017-02-13 10:10:00,329 [256:7761FFD4] WARN - IntervalSchedule: Skip schedule [rac_database:RACTEST:observer_11g] - skipping due to execution time exceeding interval
2017-02-13 10:10:00,329 [257:B85D2018:GC.Executor.6 (oracle_database:RACTEST:Response)] WARN - IntervalSchedule: Skip schedule [oracle_database:RACTEST:Response] - skipping due to execution time exceeding interval
2017-02-13 10:10:00,329 [258:5A4306F7:GC.Executor.7 (rac_database:RACTEST:observer_11g)] WARN - Action result processing failure rac_database.RACTEST::observer_11g
java.lang.InterruptedException 
 at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1302)
...
..
.
oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:620)
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:643)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at oracle.sysman.gcagent.task.TaskFutureImpl.run1(TaskFutureImpl.java:380)
...
..
. 

 at java.lang.Thread.run(Thread.java:662)
2017-02-13 10:10:00,331 [257:GC.Executor.6] ERROR - Critical error:
oracle.sysman.gcagent.task.TaskZombieException: task declared as a zombie
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.accountedCall(TaskFutureImpl.java:620)
 at oracle.sysman.gcagent.task.TaskFutureImpl$WrappedTask.call(TaskFutureImpl.java:643)

WORKAROUND

Subscribe to get access

Read more of this content when you subscribe today.

ME ( Metric Extension ) Metric Is Not Collected For Some Databases

By default, ME metric will not be collected if associated target has a severity on response:state metric. So override this default by explicitly choosing Advanced option while creating ME.

For some databases, ME ( Metric Extension ) is not collecting data. But for the rest databases, it is working fine.

me_not_collecting

Checking the ME ( Metric Extension ) definition and found an advanced option is not selected.

selection

After creating a new version of this ME ( Metric Extension ) by checking this mentioned option, it works fine.

The reason is  this database has frequent Warning/Critical response metric alert. like  :

…..
….

Event Type=Metric Alert
Event name=Server_Adaptive_Threshold_Metric:instance_efficiency__response_time_pt
Metric Group=Server_Adaptive_Threshold_Metric
Metric=Response Time Per Txn
Metric value=258.497253691275
Key Value=SYSTEM
Rule Name=DEFAULT_RULESET_FOR_ALL_TARGETS,METRIC_ALERT_INCIDENT_CREATION
Rule Owner=
Update Details:
Metrics “Response Time Per Txn” is at 258.497
Incident created by rule (Name = Incident management rule set for all targets, Create incident for critical metric alerts [System generated rule]).