ExaCC: How to Run ExaCLI

ExaCLI offers a subset of the commands found in the on-premises Exadata command line utility. The utility runs on the database compute nodes in the Exadata Cloud Service.

Find the Name of the VM Cluster

Run the following command as GI user ( grid ).

[grid@racnode1 ~]$ crsctl get cluster name
CRS-6724: Current cluster name is 'usa2598clu01cb1'

Find Password

The password for cloud_user_clustername is initially set to a random value, which you can view by running the following command as as opc user on any cluster node.

[opc@racnode1 ~]$ /opt/exacloud/get_cs_data.py -h
usage: get_cs_data.py [-h] [--dataonly]
Obtain secrets stored during provisioning
optional arguments:
  -h, --help  show this help message and exit
  --dataonly  data only output

[opc@racnode1 ~]$ /opt/exacloud/get_cs_data.py --dataonly
5k3ew9qYTnKb#LD

As per Oracle doc, the following command should be run but not working, it seems due to ‘get_cs_data.py’ version difference.

[opc@racnode1 ~]$ /opt/exacloud/get_cs_data.py --data_file /opt/exacloud/cs_data.enc
usage: get_cs_data.py [-h] [--dataonly]
get_cs_data.py: error: unrecognized arguments: --data_file /opt/exacloud/cs_data.enc

Connect to a Storage Server with ExaCLI

Find Target Storage Server’s IP Address

[root@racnode1 ~]#cat /etc/oracle/cell/network-config/cellip.oracle
cell="192.168.136.5;cell="192.168.136.6"
cell="192.168.136.7;cell="192.168.136.8"
cell="192.168.136.9;cell="192.168.136.10"
...
..
.

Run ExaCLI Command

#exacli -c cloud_user_usa2598clu01cb1@192.168.136.6 -e LIST IORMPLAN DETAIL

Password: ***************
         name:                   usa2598clu01cb1_IORMPLAN
         catPlan:
         dbPlan:
         clusterPlan:
         objective:              auto
         status:                 active

If you are connecting to a storage cell for the first time using ExaCLI, you may be prompted to accept an SSL certificate. The ExaCLI output in this case will look like the following:

#exacli -c cloud_user_usa2598clu01cb1@192.168.136.6 -e LIST IORMPLAN DETAIL
EXA-30016: This connection is not secure. You have asked ExaCLI to connect to cell 192.168.136.6 securely. The identity of 192.168.136.6 cannot be verified.
Got certificate from server:
C=US,ST=California,L=Redwood City,O=Oracle Corporation,OU=Oracle Exadata,CN=localhost
Do you want to accept and store this certificate? (Press y/n)

Accept the self-signed Oracle certificate by pressing “y” to continue using ExaCLI.

How to upgrade DBAAS Cloud Tooling using dbaascli

Logon as root user

$ sudo -s
#

 Check your current version of cloud tooling

#  rpm -qa|grep -i dbaastools
dbaastools_exa-1.0-1+22.2.1.1.0_220713.1149.x86_64

Check whether any cloud tooling updates are available

# sudo dbaascli admin showLatestStackVersion
DBAAS CLI version 22.2.1.1.0
Executing command admin showLatestStackVersion
Job id: 7cf1c287-3617-4d9b-990b-7b2147d1e164
{
  "description" : "DBAAS Tools for Oracle Public Cloud",
  "version" : "22.3.1.0.1_220831.2106",
  "baseVersion" : "22.3.1.0.1"
}
dbaascli execution completed

# dbaascli patch tools list
DBAAS CLI version 22.2.1.1.0
Executing command patch tools list
Checking Current tools on all nodes

exaccnode1: Patchid : 22.2.1.1.0_220713.1149

Available Patches
Patchid : 22.3.1.0.1_220831.2106(LATEST)

Install the latest version of dbaastools by using the command: dbaascli admin updateStack

exaccnode2: Patchid : 22.2.1.1.0_220713.1149

Available Patches
Patchid : 22.3.1.0.1_220831.2106(LATEST)

Install the latest version of dbaastools by using the command: dbaascli admin updateStack

exaccnode3: Patchid : 22.2.1.1.0_220713.1149

Available Patches
Patchid : 22.3.1.0.1_220831.2106(LATEST)

Install the latest version of dbaastools by using the command: dbaascli admin updateStack

exaccnode4: Patchid : 22.2.1.1.0_220713.1149

Available Patches
Patchid : 22.3.1.0.1_220831.2106(LATEST)

Install the latest version of dbaastools by using the command: dbaascli admin updateStack

All Nodes have the same tools version

Upgrading Prechecks

# dbaascli admin updateStack --prechecksOnly --version LATEST
DBAAS CLI version 22.2.1.1.0
Executing command admin updateStack --prechecksOnly --version LATEST
INFO : Review log file => /var/opt/oracle/log/tooling/Update/Update_yyyy-mm-dd_hh24:mi:ss.nnnnnnnnnnnn.log 
============ Starting RPM update operation =========== 
Loading PILOT...
Session ID of the current execution is: 274
Log file location: /var/opt/oracle/log/tooling/Update/pilot_mm-dd_hh24:mi:ss-PM
-----------------
Running Plugin_initialization job
Completed Plugin_initialization job
-----------------
Running Default_value_initialization job
Completed Default_value_initialization job
-----------------
Running Rpm_version_validation job
Completed Rpm_version_validation job
-----------------
Running Rpm_source_validation job
Completed Rpm_source_validation job
-----------------
Running Disk_space_download_validation job
Completed Disk_space_download_validation job
-----------------
Running Rpm_download job
Skipping. Job is detected as not applicable.
-----------------
Running Rpm_validation job
Skipping. Job is detected as not applicable.
-----------------
Running Rpm_disk_space_validation job
Skipping. Job is detected as not applicable.
-----------------
Running Rpm_local_installation job
Skipping. Job is detected as not applicable.
-----------------
Running Rpm_remote_installation job
Skipping. Job is detected as not applicable.
-----------------
Running Installed_rpm_backup job
Skipping. Job is detected as not applicable.
-----------------
Running Cleanup_downloaded_rpm job
Skipping. Job is detected as not applicable.
-----------------
Running Cleanup_legacy_rpms job
Skipping. Job is detected as not applicable

Upgrade Cloud Tooling

# dbaascli admin updateStack --version LATEST
...
..
.

Check your current version of cloud tooling

# rpm -qa --last|egrep -i "dbaastools_exa|dbcs"
dbaastools_exa-1.0-1+22.3.1.0.1_220831.2106.x86_64 Fri 16 Sep 2022 02:14:38 PM AEST
dbcs-agent-update-exacc-22.2.1.1.0-220713.1149.x86_64 Thu 28 Jul 2022 01:25:22 PM AEST
dbcs-agent-exacc-2.8OL7-21.1.1.0.2_210408.1900.x86_64 Wed 07 Jul 2021 08:07:29 PM AEST

Check Available Updates

# dbaascli admin showLatestStackVersion
DBAAS CLI version 22.3.1.0.1
Executing command admin showLatestStackVersion
Job id: 886e630e-f911-425d-9219-936d62e5ab5e
{
  "description" : "DBAAS Tools for Oracle Public Cloud",
  "version" : "22.3.1.0.1_220831.2106",
  "baseVersion" : "22.3.1.0.1"
}
dbaascli execution completed


# dbaascli patch tools list
DBAAS CLI version 22.3.1.0.1
Executing command patch tools list
Checking Current tools on all nodes

exaccnode1: Patchid : 22.3.1.0.1_220831.2106
No applicable tools patches are available

exaccnode2: Patchid : 22.3.1.0.1_220831.2106
No applicable tools patches are available

exaccnode3: Patchid : 22.3.1.0.1_220831.2106
No applicable tools patches are available

exaccnode4: Patchid : 22.3.1.0.1_220831.2106
No applicable tools patches are available

All Nodes have the same tools version

The Cluster Health Advisor (CHA) detected an unexpected increase in CPU utilization by databases or applications on this node. Identify CPU intensive processes and databases by reviewing Cluster Health Monitoring (CHM) data. Relocate databases to less busy machines, or limit the number of connections to databases on this node. Add nodes if more resources are required

A client keeps receiving the following alerts from OEM:

Host=racnode1.ractest
Target type=Cluster 
Target name=RACTEST-CLUSTER
Incident creation time=15/12/2021 2:11:20 PM AEDT 
Last updated time=15/12/2021 2:11:20 PM AEDT 
Message=Host CPU Utilization on Host racnode1 Database/Cluster RACTEST-CLUSTER Instance . The Cluster Health Advisor (CHA) detected an unexpected increase in CPU utilization by databases or applications on this node. Identify CPU intensive processes and databases by reviewing Cluster Health Monitoring (CHM) data. Relocate databases to less busy machines, or limit the number of connections to databases on this node. Add nodes if more resources are required. 
Severity=Warning 
Incident ID=501920 
Event count=1 
Incident Status=New 
Escalated=No 
Priority=High 
Incident owner=SYSMAN
Incident Acknowledged By Owner=No 
Categories= 
Rule Name=RuleSet - SYSMAN,Email SYSMAN About the Incidents 
Rule Owner=SYSMAN
...
..
.

Now let’s look into further where the alerts come from, the logs, GI processes and related metrics configurations on OEM.

Subscribe to get access

Read more of this content when you subscribe today.

CRS-2878: Failed to restart resource ‘ora.net1.network’ CRS-2769: Unable to failover resource ‘ora.net1.network’

The following messages appear in alert.log:

...
..
.
2021-10-26 6:34:11.701 [CRSD(56455)]CRS-2878: Failed to restart resource 'ora.net1.network'
2021-10-26 6:34:11.705 [CRSD(56455)]CRS-2769: Unable to failover resource 'ora.net1.network'.
2021-10-26 6:34:11.807 [CRSD(56455)]CRS-2878: Failed to restart resource 'ora.net1.network'
2021-10-26 6:34:11.880 [CRSD(56455)]CRS-2769: Unable to failover resource 'ora.net1.network'.
...
..
.

In the mean time, there are other resources failure from scan listeners, VIPs and local listeners, etc.

Subscribe to get access

Read more of this content when you subscribe today.

AFD-0654: AFD is not supported on Exadata systems

The following messages appear in CRS alert.log of ExaCC or Exadata:

2021-10-24 05:10:52.584 [CLSECHO(189590)]AFD-0654: AFD is not supported on Exadata systems
2021-10-24 11:10:53.734 [CLSECHO(288932)]AFD-0654: AFD is not supported on Exadata systems
2021-10-24 17:10:56.450 [CLSECHO(398661)]AFD-0654: AFD is not supported on Exadata systems
2021-10-24 23:10:55.077 [CLSECHO(92820)]AFD-0654: AFD is not supported on Exadata systems
2021-10-25 05:10:54.274 [CLSECHO(195018)]AFD-0654: AFD is not supported on Exadata systems
2021-10-25 11:10:53.935 [CLSECHO(268638)]AFD-0654: AFD is not supported on Exadata systems
2021-10-25 17:10:52.051 [CLSECHO(361889)]AFD-0654: AFD is not supported on Exadata systems
2021-10-25 23:10:52.858 [CLSECHO(37352)]AFD-0654: AFD is not supported on Exadata systems
2021-10-26 05:10:56.178 [CLSECHO(135982)]AFD-0654: AFD is not supported on Exadata systems

AFD protects from unauthorized processes over write to the asm claimed disks
in raw disk environment which is not applicable to exadata disks (non-oracle processes could not recognize / write to exadata disks).

so the message “FD-0654: AFD is not supported on Exadata systems” in alert.log can be ignored.