How to Tune OEM 13c Agent Parameters when Agent is Monitoring Big Amount of Targets

In Exadata environment, the OEM agent is monitoring up to one hundred of targets. The agent always stopped or bounced by itself with the following different errors. This post demonstrates how to tune the agent parameters to improve the agent satiability and performance.

Subscribe to get access

Read more of this content when you subscribe today.

EMAGENT Stopped With “OutOfMemoryError”

Java paramemetes like ” -Xmx” and “-XX:MaxPermSize” might cause OEM agent up/down periodically.

SYMPTOMS

The agent fails to start, or keeps restarting  itself,  or becomes unresponsive and the following messages are seen in logs.

$AGENT_INST/bin/emctl start agent reports following error :

......
......
......
Starting agent .........................................
Consult emctl.log and emagent.nohup in: /db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log

$AGENT_INST/sysman/log/emagent.nohup reports following error :

......
......
2016-03-06 20:03:08,691 [1:main] WARN - Missing filename for log handler 'opsscfg
Agent is going down due to an OutOfMemoryError
----- Sun Mar 6 20:03:28 2016::23451::Checking status of EMAgent : 30249 -----
----- Sun Mar 6 20:03:28 2016::23451::EMAgent exited at Sun Mar 6 20:03:28 2016 with return value 57. -----
----- Sun Mar 6 20:03:28 2016::23451::EMAgent will be restarted because of an Out of Memory Exception. 
......
......

$AGENT_INST/sysman/log/gcagent.log reports following error :

......
......
2016-03-06 20:02:52,549 [1:main] FATAL - Fatal error: Target Interaction Manager failed at Startup
java.lang.OutOfMemoryError: Java heap space
......
......
......
2016-03-06 20:02:52,552 [1:main] ERROR - agent main threw an error
java.lang.OutOfMemoryError: Java heap space

$GENT_INST/bin/emctl start agent reports following error :

Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting agent ............................................failed.
Fatal agent error: Target Interaction Manager failed at Startup
Fatal agent error: Target Interaction Manager failed at Startup
Fatal agent error: Target Interaction Manager failed at Startup
EMAgent is Thrashing. Exiting watchdog

$AGENT_INST/sysman/log/emagent.nohup reports following error :

......
......
2016-03-06 20:03:08,691 [1:main] WARN - Missing filename for log handler 'opsscfg
Agent is going down due to an OutOfMemoryError

$AGENT_INST/sysman/log/gcagent.log reports following error :

2016-03-06 20:02:52,549[1:3305B9] INFO - Invoking STARTUP_P1 (1) on Target Interaction Manager
2016-03-06 20:02:52,549[1:main] FATAL - Fatal error: Target Interaction Manager failed at Startup
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)

OR

$AGENT_INST/sysman/log/gcagent.log reports following errors:

......
......
......
java.lang.OutOfMemoryError: PermGen space
......
......
......
java.lang.OutOfMemoryError: PermGen space
2016-03-06 20:02:52,660 [64:GC.Executor.8 (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response) (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response:Response)] ERROR - oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response:Response
java.lang.OutOfMemoryError: PermGen space

SOLUTION

1. Stop the agent first.

$ $AGENT_INST/bin/emctl stop agent

If agent does not shutdown gracefully, then kill all agent background processes by first grepping for agent perl and java processes only.

$ps -eaf | grep java | grep '/db/oracle/product/agent12c/12.1.0.4/agent_inst'

$ps -eaf | grep perl
$kill -9 <Process id>

2. Take backup and edit $AGENT_INST/sysman/config/emd.properties file.

Note : Tune -Xmx and -XX:MaxPermSize parameters , in the below example heap value is increased to 512M from 128M .

The default current value:

agentJavaDefines=-Xmx128M -XX:MaxPermSize=96M

The new value:

agentJavaDefines=-Xmx512M -XX:MaxPermSize=128M

3. Restart agent successfully.

$ $AGENT_INST/bin/emctl start agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting agent ............................................... started.

$AGENT_INST/bin/emctl status agent
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 12.1.0.4.0
OMS Version : 12.1.0.4.0
Protocol Version : 12.1.0.1.0
Agent Home : /db/oracle/product/agent12c/12.1.0.4/agent_inst
Agent Log Directory : /db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log
Agent Binaries : /db/oracle/product/agent12c/12.1.0.4/core/12.1.0.4.0
Agent Process ID : 6532
Parent Process ID : 6208
......
......
......
Operating System : Linux version 2.6.32-358.6.1.el6.x86_64 (amd64)
Last Reload : (none)
Last successful upload : 2016-03-06 20:19:15
Last attempted upload : 2016-03-06 20:19:15
Total Megabytes of XML files uploaded so far : 0.03
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0
Available disk space on upload filesystem : 41.27%
Collection Status : Collections enabled
Heartbeat Status : Ok
Last attempted heartbeat to OMS : 2016-03-06 20:19:14
Last successful heartbeat to OMS : 2016-03-06 20:19:14
Next scheduled heartbeat to OMS : 2016-03-06 20:20:14

---------------------------------------------------------------
Agent is Running and Ready