Category: OEM( Oracle Enterprise Manager)

Oracle Enterprise Manager 12c,13c and more

Failed to connect to ASM instance. The connection is closed: The connection is closed

One client OEM reports the following incidents intermittently from OEM 13c.

Host=racnode1.virtuallab
Target type=Automatic Storage Management
Target name=+ASM1_racnode1.virtuallab
Categories=Availability
Message=Failed to connect to ASM instance. The connection is closed: The connection is closed
Severity=Fatal
Event reported time=20/12/2020 09:19:01AM EST
Target Lifecycle Status=Test
Operating System=LINUX
Platform=X86_64
Associated Incident Id=12345
Associated Incident Status=New
Associated Incident Owner=oracle
Associated Incident Acknowledged By Owner=No
Associated Incident Priority=High
Associated Incident Escalation Level=0
Event Type=Target Availability
Event name=Status
Availability status=Down

Let’s do some investigations, and provide the easy workaround if patches are unavailable to some clients.

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe
Log in

ORA-24327: need explicit attach before authenticating a user (DBD ERROR: OCISessionBegin)

One of the OEM agent log has the following error, and some of the jobs fail due to the database connection issues:

$ grep ORA-24327 *
gcagent.log.16:em_error=Could not connect to dbsnmp/(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=racnode1.virtuallab)(Port=1521))(CONNECT_DATA=(SID=RACTESTDB1))): ORA-24327: need explicit attach before authenticating a user (DBD ERROR: OCISessionBegin)

The output from CA( Corrective Action) job:

Connector descriptor is (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=racnode1.virtuallab)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=RACTESTDB)(SERVER=DEDICATED)))
Could not connect to SYSTEM/(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=racnode1.virtuallab)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=RACTESTDB)(SERVER=DEDICATED))):ORA-24327: need explicit attach before authenticating a user (DBD ERROR: OCISessionBegin)

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe
Log in

oracle.sysman.gcagent.task.TaskTimeoutException: task timeout: 120000 MILLISECONDS for oracle_database: ORCL:Response

There following alert has been received with timeout error message “…oracle.sysman.gcagent.task.TaskTimeoutException: task timeout: 120000 MILLISECONDS…” 

 Host=oemnode1 
 Target type=Database Instance 
 Target name=OEMPROD
 Incident creation time=13/11/2020 10:09:39 PM AEDT 
 Last updated time=13/11/2020 10:09:39 PM AEDT 
 Message=Metric evaluation error start - oracle.sysman.emSDK.agent.fetchlet.exception.FetchletException: Timeout during collection: oracle.sysman.gcagent.task.TaskTimeoutException: task timeout: 120000 MILLISECONDS for oracle_database:OEMPROD:Response 
 Severity=Critical 
 Incident ID=463520 
 Event count=1 
 Incident Status=New 
 Escalated=No 
 Priority=High 
 Incident owner=SYSMAN
 Incident Acknowledged By Owner=No 
 Categories=Availability 
 ...
..
.

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe

An internal error has occurred at the agent: “java.lang.IllegalStateException: getAgentSubstitutionMap(“cluster.ClusterName”) == null”

While testing a named credential in OEM 13c by following command with errors:

Here “CLU-RACTEST” is the GI cluster name.

$emcli test_named_credential -cred_names="SEC_RACTEST" -target_name="CLU-RACTEST" -target_type="cluster"

An internal error has occurred at the agent: "java.lang.IllegalStateException: getAgentSubstitutionMap("cluster.CLU-RACTEST") == null". Check the agent's error logs for details.

Run “emctl config agent listtargets”, it shows there is no ‘cluster’ target type listed for any of the monitoring agents.

The <AGENT_INST>/sysman/log/gcagent.log showed the below errors.

2020-09-11 14:20:12,572 [631842:5A0FE62C] WARN - target cluster.CLU-RACTEST does not exist
...
..
.
[159979801250001])] ERROR - remote target "cluster.CLU-RACTEST" has NO substitution properties; check that it is properly configured
2020-09-11 14:20:12,573 [631842:HTTP Listener-631842 - /emd/main/ (DispatchRequests OMS.console@17163@oemnode1=>[159979801250001])] ERROR - Critical error:
java.lang.IllegalStateException: getAgentSubstitutionMap("cluster.CLU-RACTEST") == null
at oracle.sysman.gcagent.jobs.JobRequestDriver.substitute(JobRequestDriver.java:2162)
at oracle.sysman.gcagent.jobs.JobRequestDriver.(JobRequestDriver.java:1500)
at oracle.sysman.gcagent.jobs.JobRequestDriver.(JobRequestDriver.java:1529)
at oracle.sysman.gcagent.dispatch.cxl.PerformOperationAction.satisfyRequest(PerformOperationAction.java:130)
...
..
.

SOLUTION

From the OEM Console, go to Cluster homepage -> Target Setup -> Monitoring Configuration.
Click on the Update button (no need to make any other changes).

This will push the cluster target details to the agent side.

Then run “emcli test_named_credential” successfully.

$emcli test_named_credential -cred_names="SEC_RACTEST" -target_name="CLU-RACTEST" -target_type="cluster"

Credentials "SEC_RACTEST:SYSMAN" tested successfully

How to Run ADR Command Interpreter(adrci) from OMS and Agent Oracle Home

Run adrci command from OEM 13c OMS or OEM 13c Agent ORACLE_HOME, and get “No ADR base is set” error.

$ adrci
ADRCI: Release 11.2.0.3.0 - Production on Tue Sep 8 20:57:51 2020
Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.
No ADR base is set
adrci>

Here are the steps of how to run adrci properly from OMS or Agent ORACLE_HOME.

Subscribe to get access

Read more of this content when you subscribe today.

Subscribe