[Placeholder for all my notes about the RAC srvctl and crs_* commands]

* In 10g, if sqlplus can start/stop instance but srvctl cannot, check 
$ORACLE_HOME/log/`hostname -s`/racg/imon<DBname>.log for any message. 11g doesn't have this file. 
See https://support.oracle.com/CSP/main/article?cmd=show&type=NOT&id=844272.1


* The best way to turn on srvctl trace is
SRVM_TRACE=1 srvctl ...
That is, if using bash (or any Bourne-derived shell), precede srvctl with `SRVM_TRACE=<anything>'.
This is better than `export SRVM_TRACE=<anything>' followed by srvctl because turning off trace
is `unset SRVM_TRACE', not `export SRVM_TRACE=false', which still turns on trace. Read srvctl 
script to know why.


* srvctl should always run as the database software owner, most likely oracle, not root. If you 
accidentally run it as root, e.g. `srvctl add database...', the `crs_getperm ora.<DB>.db' shows 
the owner to be root. Correct it with `crs_setperm ora.<DB>.db -o oracle'. Also check file and 
directory ownership from $ORACLE_HOME down, particularly $ORACLE_HOME/log:
find $ORACLE_HOME -user root -ls
Correct them as needed:
find $ORACLE_HOME -user root -exec chown oracle:oinstall {} \;
(A few files such as oradism are better owned by root so you can change them back to root, if desired.)


* In 11g before 11.2.0.2, due to Bug 9251136 "INSTANCE WILL NOT USE HUGEPAGE IF STARTED BY SRVCTL". The 
root cause is process ulimit settings (as in /etc/security/limits.conf) are not used. To confirm, check 
/proc/<pmon pid>/limits on Linux or plimit <pid> on Solaris. Add
ulimit -l <locked memory upper limit in KB>
to $CRS_HOME/bin/ohasd or /etc/init.d/ohasd, but not /etc/profile or /etc/init.d/init.ohasd to 
work around the bug. The bug is fixed in 11.2.0.2.


* Sometimes a resource cannot be removed ("does not exist") and cannot be added ("already exists"). 
Try `srvctl remove ... -f' to force remove first. If that does not work, use crs_unregister command.


* When a resource's state is UNKNOWN, try `srvctl stop ...'. The state should change to OFFLINE. 
Then you can `srvctl start ...'.


* VIP should depend on listener, and instance on ASM. To add dependency of, e.g. instance, on ASM:
srvctl modify instance -d <DB> -i <INST> -s +ASM<number>

In 10g, there should not be any other dependency, 
such as ASM on VIP. If yes, remove it per Note:4865736.8: 
ASM resource name is in the form of ora.<nodename>.<ASM instance name>.asm. ASM instance name does not include "+" sign
  VIP resource name is in the form of ora.<nodename>.vip
  - Run "crs_stat -p <ASM resource name> > /tmp/<ASM resource name>.cap". Note file name must be <ASM resource name>.cap
  - Edit /tmp/<ASM resource name>.cap to remove VIP resource name from the REQUIRED_RESOURCES attribute. I.e. leave the 
    right part of the equal sign empty.
  - Run "crs_register -u <ASM resource name> -dir /tmp" to actually remove the dependency info in OCR
  - Use "crs_stat -p <ASM resource name> | grep REQUIRED" to verify REQUIRED_RESOURCE attribute is updated (i.e. emptied).
If you don't remove it, stopping 
nodeapps will crash ASM and then instance.


* Use these commands to check all dependencies:
10g:
crs_stat -p | grep -v =$ | egrep '^NAME|^REQUIRED_RESOURCES'
or with indentation for easy reading:
crs_stat -p | grep -v =$ | perl -nle 'if(/^NAME/){print}elsif(/^REQUIRED_RESOURCES/){print "\t$_"}'
11g:
crsctl stat res -p | grep -v =$ | egrep '^NAME|DEPENDENCIES'
or with indentation for easy reading:
crsctl stat res -p | grep -v =$ | perl -nle 'if(/^NAME/){print}elsif(/DEPENDENCIES/){print "\t$_"}'