DOYENSYS Knowledge Portal




We Welcome you to visit our DOYENSYS KNOWLEDGE PORTAL : Doyensys Knowledge Portal




Monday, November 3, 2014

Enterprise Manager Cloud Control 12c Agent Keeps Going Down With java.lang.OutOfMemoryError: unable to create new native thread

If your are getting the below error while starting the 12c OEM agent,

 java.lang.OutOfMemoryError: unable to create new native thread

Then check for the below logfiles of 12c OEM agent,

$AGENT_HOME/sysman/log/emagent.nohup

emagent.nohup:


Agent is going down due to an OutOfMemoryError

----- Sat Sep  6 01:28:02 2014::31358::Checking status of EMAgent : 1685 -----
----- Sat Sep  6 01:28:02 2014::31358::EMAgent exited at Sat Sep  6 01:28:02 2014 with return value 57. -----
----- Sat Sep  6 01:28:02 2014::31358::EMAgent will be restarted because of an Out of Memory Exception. -----
----- Sat Sep  6 01:28:02 2014::31358::EMAgent is Thrashing. Exiting loop. -----
----- Sat Sep  6 01:28:02 2014::31358::Commiting Process death. -----
----- Sat Sep  6 01:28:02 2014::31358::writeAbnormalExitTimestampToAgntStmp: exitCause=OOM : restartRequired=0 -----
----- Sat Sep  6 01:28:02 2014::31358::Exited due to Thrash. -----


gcagent.log


2014-09-06 01:27:55,220 [57:GC.Executor.0 (oracle_database:abc.oracle.com7:problemTbsp_10i_Dct)] ERROR - Critical error:
java.lang.OutOfMemoryError: unable to create new native thread
        at java.lang.Thread.start0(Native Method)
        at java.lang.Thread.start(Thread.java:640)
        at java.lang.UNIXProcess$1.run(UNIXProcess.java:141)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:103)
        at java.lang.ProcessImpl.start(ProcessImpl.java:65)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
        at oracle.sysman.gcagent.common.SafeProcessBuilder.start(SafeProcessBuilder.java:120)
       at oracle.sysman.gcagent.util.channel.OSChannel.runStreamingCommand(OSChannel.java:414)
        at oracle.sysman.gcagent.util.channel.OSChannel.runCommand(OSChannel.java:449)

We are getting the above error because,

The kernel nproc limit is set too low and needs to be increased.
The more targets the agent is monitoring, the higher the nproc limit must be. We have more than 20 instance running in the server. So we have increased the nproc limit to unlimited as below and started the agent without any issues,

From:

[oracle@abc] ulimit -Su
1024

[oracle@abc] ulimit -Hu
16384


To:

[oracle@abc] ulimit -Su
unlimited

[oracle@abc] ulimit -Hu
unlimited

2 comments:

java said...

Hiring the right people is what gives you the competitive advantage you are looking for. Thus, carrying out a hiring program that brings to your fingertips the right talent to create a framework that lets you serve your clients in the most empowered manner is critical.

Sridevi K said...

Regards
Sridevi Koduru (Senior Oracle Apps Trainer Oracleappstechnical.com)
LinkedIn profile - https://in.linkedin.com/in/sridevi-koduru-9b876a8b
Please Contact for One to One Online Training on Oracle Apps Technical, Financials, SCM, SQL, PL/SQL, D2K at training@oracleappstechnical.com | +91 - 9581017828.