One of client’s OEM agent is working fine, but OEM console show “Agent Unreachable’ for this agent and all the other targets monitored by this agent.
Read more of this content when you subscribe today.
This post demonstrates the installation of Oracle Enterprise Manager Cloud Control 13c Release 3 (13.3.0.0) on Oracle Linux 7 (x86_64).
Read more of this content when you subscribe today.
URL : https://oemnode1:7803/em
username: sysman
Password: you specified during your installation
[oracle@oemnode1 scripts]$ . oraenv ORACLE_SID = [OEMREP] ? AGENT [oracle@oemnode1 scripts]$ emctl status agent Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 13.3.0.0.0 OMS Version : 13.3.0.0.0 Protocol Version : 12.1.0.1.0 Agent Home : /u01/app/oracle/Product/Agent/agent_inst Agent Log Directory : /u01/app/oracle/Product/Agent/agent_inst/sysman/log Agent Binaries : /u01/app/oracle/Product/Agent/agent_13.3.0.0.0 Core JAR Location : /u01/app/oracle/Product/Agent/agent_13.3.0.0.0/jlib Agent Process ID : 6463 Parent Process ID : 6403 Agent URL : https://oemnode1:3872/emd/main/ Local Agent URL in NAT : https://oemnode1:3872/emd/main/ Repository URL : https://oemnode1:4903/empbs/upload Started at : 2020-03-27 23:43:12 Started by user : oracle Operating System : Linux version 4.14.35-1902.10.7.el7uek.x86_64 (amd64) Number of Targets : 33 Last Reload : (none) Last successful upload : 2020-03-28 00:14:07 Last attempted upload : 2020-03-28 00:14:07 Total Megabytes of XML files uploaded so far : 1.72 Number of XML files pending upload : 0 Size of XML files pending upload(MB) : 0 Available disk space on upload filesystem : 13.15% Collection Status : Collections enabled Heartbeat Status : Ok Last attempted heartbeat to OMS : 2020-03-28 00:13:52 Last successful heartbeat to OMS : 2020-03-28 00:13:52 Next scheduled heartbeat to OMS : 2020-03-28 00:14:52 Agent is Running and Ready
[oracle@oemnode1 ~]$ . oraenv ORACLE_SID = [OMS] ? OMS [oracle@oemnode1 ~]$ emctl status oms -bip_only Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. BI Publisher Server is Up [oracle@oemnode1 ~]$ emctl status oms -details Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host : oemnode1 HTTP Console Port : 7788 HTTPS Console Port : 7803 HTTP Upload Port : 4889 HTTPS Upload Port : 4903 EM Instance Home : /u01/app/oracle/Product/gc_inst/em/EMGC_OMS1 OMS Log Directory Location : /u01/app/oracle/Product/gc_inst/em/EMGC_OMS1/sysman/log OMS is not configured with SLB or virtual hostname Agent Upload is locked. OMS Console is locked. Active CA ID: 1 Console URL: https://oemnode1:7803/em Upload URL: https://oemnode1:4903/empbs/upload WLS Domain Information Domain Name : GCDomain Admin Server Host : oemnode1 Admin Server HTTPS Port: 7102 Admin Server is RUNNING Oracle Management Server Information Managed Server Instance Name: EMGC_OMS1 Oracle Management Server Instance Host: oemnode1 WebTier is Up Oracle Management Server is Up JVMD Engine is Up BI Publisher Server Information BI Publisher Managed Server Name: BIP BI Publisher Server is Up BI Publisher HTTP Managed Server Port : 9701 BI Publisher HTTPS Managed Server Port : 9803 BI Publisher HTTP OHS Port : 9788 BI Publisher HTTPS OHS Port : 9851 BI Publisher is locked. BI Publisher Server named 'BIP' running at URL: https://oemnode1:9851/xmlpserver/servlet/home BI Publisher Server Logs: /u01/app/oracle/Product/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/ BI Publisher Log : /u01/app/oracle/Product/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/bipublisher/bipublisher.log [oracle@oemnode1 ~]$
[oracle@oemnode1 ~]$ emctl stop oms Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Stopping Oracle Management Server... Oracle Management Server Successfully Stopped Oracle Management Server is Down JVMD Engine is Down [oracle@oemnode1 ~]$ emctl status oms Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. WebTier is Up Oracle Management Server is Down JVMD Engine is Down BI Publisher Server is Up [oracle@oemnode1 ~]$
[oracle@oemnode1 ~]$ emctl stop oms -all Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Stopping Oracle Management Server... WebTier Successfully Stopped Oracle Management Server Successfully Stopped Oracle Management Server is Down JVMD Engine is Down Stopping BI Publisher Server... BI Publisher Server Successfully Stopped AdminServer Successfully Stopped BI Publisher Server is Down [oracle@oemnode1 ~]$
[oracle@oemnode1 ~]$ emctl status oms Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. WebTier is Up Oracle Management Server is Up JVMD Engine is Up BI Publisher Server is Up [oracle@oemnode1 ~]$
[oracle@oemnode1 ~]$ emctl status oms -details Oracle Enterprise Manager Cloud Control 13c Release 3 Copyright (c) 1996, 2018 Oracle Corporation. All rights reserved. Enter Enterprise Manager Root (SYSMAN) Password : Console Server Host : oemnode1 HTTP Console Port : 7788 HTTPS Console Port : 7803 HTTP Upload Port : 4889 HTTPS Upload Port : 4903 EM Instance Home : /u01/app/oracle/Product/gc_inst/em/EMGC_OMS1 OMS Log Directory Location : /u01/app/oracle/Product/gc_inst/em/EMGC_OMS1/sysman/log OMS is not configured with SLB or virtual hostname Agent Upload is locked. OMS Console is locked. Active CA ID: 1 Console URL: https://oemnode1:7803/em Upload URL: https://oemnode1:4903/empbs/upload WLS Domain Information Domain Name : GCDomain Admin Server Host : oemnode1 Admin Server HTTPS Port: 7102 Admin Server is RUNNING Oracle Management Server Information Managed Server Instance Name: EMGC_OMS1 Oracle Management Server Instance Host: oemnode1 WebTier is Up Oracle Management Server is Up JVMD Engine is Up BI Publisher Server Information BI Publisher Managed Server Name: BIP BI Publisher Server is Up BI Publisher HTTP Managed Server Port : 9701 BI Publisher HTTPS Managed Server Port : 9803 BI Publisher HTTP OHS Port : 9788 BI Publisher HTTPS OHS Port : 9851 BI Publisher is locked. BI Publisher Server named 'BIP' running at URL: https://oemnode1:9851/xmlpserver/servlet/home BI Publisher Server Logs: /u01/app/oracle/Product/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/ BI Publisher Log : /u01/app/oracle/Product/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/bipublisher/bipublisher.log
[oracle@oemnode1 ~]$ emctl stop agent
[oracle@oemnode1 ~]$ emctl start agent
[oracle@oemnode1 ~]$ emctl status agent
Java layer deadlock —“Dead Lock detected!!”, bounce the agent, and then everything is working fine.
The following alerts are received from racnode1 -“The agent is overloaded [current requests: 128]”
From: oracle Sent: Friday, 4 August 2017 7:07 PM Cc: Subject: EM Event: Warning: racnode1 - Agent Unreachable (REASON = The agent is overloaded [current requests: 128]). Host is reachable. ... .. . Categories=Availability Message=Agent Unreachable (REASON = The agent is overloaded [current requests: 128]). Host is reachable. Severity=Warning Event reported time=Aug 4, 2017 7:06:27 PM AEST ... .. .
Agent is running
Agent upload is not working
Agent reload is not working
OMS heartbeat is not working
$ emctl status agent Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 12.1.0.5.0 OMS Version : 13.2.0.0.0 Protocol Version : 12.1.0.1.0 .. . Last Reload : 2017-08-04 11:28:59 Last successful upload : 2017-08-04 14:51:03 <--- 5 hours ago Last attempted upload : 2017-08-04 14:51:03 .. . Last attempted heartbeat to OMS : 2017-08-04 14:50:23 Last successful heartbeat to OMS : 2017-08-04 14:50:23 Next scheduled heartbeat to OMS : 2017-08-04 14:51:23
$ emctl upload agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD upload error:The agent is overloaded [current requests: 128]
$ emctl reload agent
Oracle Enterprise Manager Cloud Control 12c Release 5
Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
EMD reload error:The agent is overloaded [current requests: 128]
Java layer deadlock —“Dead Lock detected!!”
2017-08-04 19:28:59,071 [43:GCThread-13] ERROR -
Dead Lock detected!!
Participating threads:Thread Info Dump:
=================
"HTTP Listener-3592 - /emd/main/ (~Task-free~ OMS.pbs@16398@omsnode=>[150183756670001])" tid=3592 WAITING
> Accumulated wait time (msec): 1372208 (1 times)
"HTTP Listener-2141 - /emd/main/ (~Task-free~ OMS.pbs@13103@omsnode=>[150182243190001])" tid=2141 BLOCKED
> Accumulated wait time (msec): 11036289 (76 times)
> Accumulated blocked time (msec): 16506994 (4 times)
"oracle.dfw.impl.incident.DiagnosticsDataExtractorImpl - Incident Dump Executor (created: Fri Aug 04 14:51:06 EST 2017)" tid=3088 BLOCKED
> Accumulated blocked time (msec): 16672145 (7 times)
"HTTP Listener-1022 - /emd/main/ (~Task-free~ OMS.pbs@16398@omsnode=>[150181021899001])" tid=1022 WAITING
> Accumulated wait time (msec): 28746227 (37 times)
> Accumulated blocked time (msec): 133 (12 times)
"HTTP Listener-1078 - /emd/main/ (DispatchRequests OMS.console@16398@omsnode=>[150181015881006])" tid=1078 WAITING
> Accumulated wait time (msec): 28719225 (44 times)
=================
Thread Info Dump:
=================
"HTTP Listener-3592 - /emd/main/ (~Task-free~ OMS.pbs@16398@omsnode=>[150183756670001])" tid=3592 WAITING
sun.misc.Unsafe.park(Native Method)
- waiting on <0x149717ec> (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is owned by "HTTP Listener-2141 - /emd/main/ (~Task-free~ OMS.pbs@13103@omsnode=>[150182243190001])" (tid=2141)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
...
..
.
$ emctl stop agent Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Stopping agent ... stopped.
$ emctl start agent Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. Starting agent ............................................ started.
$ emctl upload agent Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD upload completed successfully
$ emctl reload agent Oracle Enterprise Manager Cloud Control 12c Release 5 Copyright (c) 1996, 2015 Oracle Corporation. All rights reserved. --------------------------------------------------------------- EMD reload completed successfully
$ emctl status agent ... .. Last attempted heartbeat to OMS : 2017-08-04 19:53:31 Last successful heartbeat to OMS : 2017-08-04 19:53:31 Next scheduled heartbeat to OMS : 2017-08-04 19:54:32 --------------------------------------------------------------- Agent is Running and Ready
Make DBA Life Easy 