An internal error has occurred at the agent: “java.lang.IllegalStateException: getAgentSubstitutionMap(“cluster.ClusterName”) == null”

While testing a named credential in OEM 13c by following command with errors:

Here “CLU-RACTEST” is the GI cluster name.

$emcli test_named_credential -cred_names="SEC_RACTEST" -target_name="CLU-RACTEST" -target_type="cluster"

An internal error has occurred at the agent: "java.lang.IllegalStateException: getAgentSubstitutionMap("cluster.CLU-RACTEST") == null". Check the agent's error logs for details.

Run “emctl config agent listtargets”, it shows there is no ‘cluster’ target type listed for any of the monitoring agents.

The <AGENT_INST>/sysman/log/gcagent.log showed the below errors.

2020-09-11 14:20:12,572 [631842:5A0FE62C] WARN - target cluster.CLU-RACTEST does not exist
...
..
.
[159979801250001])] ERROR - remote target "cluster.CLU-RACTEST" has NO substitution properties; check that it is properly configured
2020-09-11 14:20:12,573 [631842:HTTP Listener-631842 - /emd/main/ (DispatchRequests OMS.console@17163@oemnode1=>[159979801250001])] ERROR - Critical error:
java.lang.IllegalStateException: getAgentSubstitutionMap("cluster.CLU-RACTEST") == null
at oracle.sysman.gcagent.jobs.JobRequestDriver.substitute(JobRequestDriver.java:2162)
at oracle.sysman.gcagent.jobs.JobRequestDriver.(JobRequestDriver.java:1500)
at oracle.sysman.gcagent.jobs.JobRequestDriver.(JobRequestDriver.java:1529)
at oracle.sysman.gcagent.dispatch.cxl.PerformOperationAction.satisfyRequest(PerformOperationAction.java:130)
...
..
.

SOLUTION

From the OEM Console, go to Cluster homepage -> Target Setup -> Monitoring Configuration.
Click on the Update button (no need to make any other changes).

This will push the cluster target details to the agent side.

Then run “emcli test_named_credential” successfully.

$emcli test_named_credential -cred_names="SEC_RACTEST" -target_name="CLU-RACTEST" -target_type="cluster"

Credentials "SEC_RACTEST:SYSMAN" tested successfully

Create Named Credentials by Using emcli

In order to create a named credential, you need to know:

  • Target name (unless you create a global credential)
  • Target type
  •  Credential type associated with the target type

Now we start to create a named credential for cluster database ( RACTEST ) user ( JAMES).

1) To get TARGET_TYPE from TARGET_NAME. Normally we know a couple of target_type from daily emcli practice.

$ emcli get_targets -targets="RACTEST:%"

Status ID Status Target Type  Target Name
1         Up     rac_database  RACTETS

2) To get the credential types (and their attributes) associated with “rac_database”.

$ emcli show_credential_type_info -target_type=rac_database
Target Type  Cred Type Name      Cred Type Column Name Key Column
rac_database DBCreds             DBPassword            No
                                 DBRole                No
                                 DBUserName            Yes
             DBHostCreds         HostPassword          No
                                 HostUserName          Yes
             DBKerberosCreds     DBKerberosPassword    No
                                 DBKerberosUserName    Yes
             DBPkiCreds          DBPkiUserWallet       Yes
                                 DBPkiUserWalletPassword No
                                 DBPkiUserWalletType   No
             HostSSHCreds        SSH_PUB_KEY           No
                                 SSH_PVT_KEY           No
                                 USERNAME              Yes

We can see target type rac_database has four credential types. we will use credential type  DBCreds in the following steps.

3) Create named credential for JAMES user of the RAC database.

$emcli create_named_credential -auth_target_type=rac_database \
 -cred_scope=Instance -target_type=rac_database \
-target_name=RACTEST -cred_type=DBCreds -cred_name=NC_RACTEST_JAMES \
-attributes="DBUserName:JAMES;DBPassword:yourpasswd" -test

Credential NC_RACTEST_JAMES created.

Here “-test” to test the credential before saving.

If you want to create a global named credential, just remove “cred_scope, target_type, target_name” parameters.