SITUATION
The following alerts are received from racnode1 -“The agent is overloaded [current requests: 128]”
From: oracle Sent: Friday, 4 August 2017 7:07 PM Cc: Subject: EM Event: Warning: racnode1 - Agent Unreachable (REASON = The agent is overloaded [current requests: 128]). Host is reachable. ... .. . Categories=Availability Message=Agent Unreachable (REASON = The agent is overloaded [current requests: 128]). Host is reachable. Severity=Warning Event reported time=Aug 4, 2017 7:06:27 PM AEST ... .. .
INVESTIGATING
1) Check agent status
Agent is running
Agent upload is not working
Agent reload is not working
OMS heartbeat is not working
$ emctl status agent Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 12.1.0.5.0 OMS Version : 13.2.0.0.0 Protocol Version : 12.1.0.1.0 .. . Last Reload : 2017-08-04 11:28:59 Last successful upload : 2017-08-04 14:51:03 <--- 5 hours ago Last attempted upload : 2017-08-04 14:51:03 .. . Last attempted heartbeat to OMS : 2017-08-04 14:50:23 Last successful heartbeat to OMS : 2017-08-04 14:50:23 Next scheduled heartbeat to OMS : 2017-08-04 14:51:23
2) Upload agent
$ emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error:The agent is overloaded [current requests: 128]
3) Reload agent
$ emctl reload agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD reload error:The agent is overloaded [current requests: 128]
4) “emagent_perl.trc” file has no information updated since agent restarted
5) Check “gcagent.log”
Java layer deadlock —“Dead Lock detected!!”
2017-08-04 19:28:59,071 [43:GCThread-13] ERROR -
Dead Lock detected!!
Participating threads:Thread Info Dump:
=================
"HTTP Listener-3592 - /emd/main/ (~Task-free~ OMS.pbs@16398@omsnode=>[150183756670001])" tid=3592 WAITING
> Accumulated wait time (msec): 1372208 (1 times)
"HTTP Listener-2141 - /emd/main/ (~Task-free~ OMS.pbs@13103@omsnode=>[150182243190001])" tid=2141 BLOCKED
> Accumulated wait time (msec): 11036289 (76 times)
> Accumulated blocked time (msec): 16506994 (4 times)
"oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Fri Aug 04 14:51:06 EST 2017)" tid=3088 BLOCKED
> Accumulated blocked time (msec): 16672145 (7 times)
"HTTP Listener-1022 - /emd/main/ (~Task-free~ OMS.pbs@16398@omsnode=>[150181021899001])" tid=1022 WAITING
> Accumulated wait time (msec): 28746227 (37 times)
> Accumulated blocked time (msec): 133 (12 times)
"HTTP Listener-1078 - /emd/main/ (DispatchRequests OMS.console@16398@omsnode=>[150181015881006])" tid=1078 WAITING
> Accumulated wait time (msec): 28719225 (44 times)
=================
Thread Info Dump:
=================
"HTTP Listener-3592 - /emd/main/ (~Task-free~ OMS.pbs@16398@omsnode=>[150183756670001])" tid=3592 WAITING
sun.misc.Unsafe.park(Native Method)
- waiting on <0x149717ec> (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is owned by "HTTP Listener-2141 - /emd/main/ (~Task-free~ OMS.pbs@13103@omsnode=>[150182243190001])" (tid=2141)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
...
..
.
SOLUTION
1) Stop agent
$ emctl stop agent Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Stopping agent ... stopped.
2) Start agent
$ emctl start agent Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Starting agent ............................................ started.
3) Upload agent successfully
$ emctl upload agent Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD upload completed successfully
4) Reload agent successfully
$ emctl reload agent Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD reload completed successfully
5)Check agent status successfully
$ emctl status agent ... .. Last attempted heartbeat to OMS : 2017-08-04 19:53:31 Last successful heartbeat to OMS : 2017-08-04 19:53:31 Next scheduled heartbeat to OMS : 2017-08-04 19:54:32 --------------------------------------------------------------- Agent is Running and Ready
Make DBA Life Easy 