SYMPTOMS
The agent fails to start, or keeps restarting itself, or becomes unresponsive and the following messages are seen in logs.
$AGENT_INST/bin/emctl start agent reports following error :
...... ...... ...... Starting agent ......................................... Consult emctl.log and emagent.nohup in: /db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log
$AGENT_INST/sysman/log/emagent.nohup reports following error :
......
......
2016-03-06 20:03:08,691 [1:main] WARN - Missing filename for log handler 'opsscfg
Agent is going down due to an OutOfMemoryError
----- Sun Mar 6 20:03:28 2016::23451::Checking status of EMAgent : 30249 -----
----- Sun Mar 6 20:03:28 2016::23451::EMAgent exited at Sun Mar 6 20:03:28 2016 with return value 57. -----
----- Sun Mar 6 20:03:28 2016::23451::EMAgent will be restarted because of an Out of Memory Exception.
......
......
$AGENT_INST/sysman/log/gcagent.log reports following error :
...... ...... 2016-03-06 20:02:52,549 [1:main] FATAL - Fatal error: Target Interaction Manager failed at Startup java.lang.OutOfMemoryError: Java heap space ...... ...... ...... 2016-03-06 20:02:52,552 [1:main] ERROR - agent main threw an error java.lang.OutOfMemoryError: Java heap space
$GENT_INST/bin/emctl start agent reports following error :
Oracle Enterprise Manager Cloud Control 12c Release 4
Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved.
Starting agent ............................................failed.
Fatal agent error: Target Interaction Manager failed at Startup
Fatal agent error: Target Interaction Manager failed at Startup
Fatal agent error: Target Interaction Manager failed at Startup
EMAgent is Thrashing. Exiting watchdog
$AGENT_INST/sysman/log/emagent.nohup reports following error :
......
......
2016-03-06 20:03:08,691 [1:main] WARN - Missing filename for log handler 'opsscfg
Agent is going down due to an OutOfMemoryError
$AGENT_INST/sysman/log/gcagent.log reports following error :
2016-03-06 20:02:52,549[1:3305B9] INFO - Invoking STARTUP_P1 (1) on Target Interaction Manager
2016-03-06 20:02:52,549[1:main] FATAL - Fatal error: Target Interaction Manager failed at Startup
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
OR
$AGENT_INST/sysman/log/gcagent.log reports following errors:
...... ...... ...... java.lang.OutOfMemoryError: PermGen space ...... ...... ...... java.lang.OutOfMemoryError: PermGen space 2016-03-06 20:02:52,660 [64:GC.Executor.8 (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response) (oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response:Response)] ERROR - oracle_camm_manager:ADPManager_EMGC_ADPMANAGER2:Response:Response java.lang.OutOfMemoryError: PermGen space
SOLUTION
1. Stop the agent first.
$ $AGENT_INST/bin/emctl stop agent
If agent does not shutdown gracefully, then kill all agent background processes by first grepping for agent perl and java processes only.
$ps -eaf | grep java | grep '/db/oracle/product/agent12c/12.1.0.4/agent_inst' $ps -eaf | grep perl $kill -9 <Process id>
2. Take backup and edit $AGENT_INST/sysman/config/emd.properties file.
Note : Tune -Xmx and -XX:MaxPermSize parameters , in the below example heap value is increased to 512M from 128M .
The default current value:
agentJavaDefines=-Xmx128M -XX:MaxPermSize=96M
The new value:
agentJavaDefines=-Xmx512M -XX:MaxPermSize=128M
3. Restart agent successfully.
$ $AGENT_INST/bin/emctl start agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. Starting agent ............................................... started. $AGENT_INST/bin/emctl status agent Oracle Enterprise Manager Cloud Control 12c Release 4 Copyright (c) 1996, 2014 Oracle Corporation. All rights reserved. --------------------------------------------------------------- Agent Version : 12.1.0.4.0 OMS Version : 12.1.0.4.0 Protocol Version : 12.1.0.1.0 Agent Home : /db/oracle/product/agent12c/12.1.0.4/agent_inst Agent Log Directory : /db/oracle/product/agent12c/12.1.0.4/agent_inst/sysman/log Agent Binaries : /db/oracle/product/agent12c/12.1.0.4/core/12.1.0.4.0 Agent Process ID : 6532 Parent Process ID : 6208 ...... ...... ...... Operating System : Linux version 2.6.32-358.6.1.el6.x86_64 (amd64) Last Reload : (none) Last successful upload : 2016-03-06 20:19:15 Last attempted upload : 2016-03-06 20:19:15 Total Megabytes of XML files uploaded so far : 0.03 Number of XML files pending upload : 0 Size of XML files pending upload(MB) : 0 Available disk space on upload filesystem : 41.27% Collection Status : Collections enabled Heartbeat Status : Ok Last attempted heartbeat to OMS : 2016-03-06 20:19:14 Last successful heartbeat to OMS : 2016-03-06 20:19:14 Next scheduled heartbeat to OMS : 2016-03-06 20:20:14 --------------------------------------------------------------- Agent is Running and Ready
Make DBA Life Easy 