Kill Session Test in RAC

On Oracle RAC, if an instance is shutdown, a user session connecting to a database service capable 
of failover should be able to be re-connected to another instance. But to test the failover by 
shutting down an instance takes too long. It would be nice to check session migration by simply 
killing the server process of the session. On Linux, there're 64 signals. The following is a test 
of each and their result. Oracle server is 11.2.0.3 on OS (shown below) and client is 12.1.0.1.0 
on Windows7.

$ uname -a
Linux <hsotname> 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Dec 13 06:58:20 EST 2013 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.5 (Santiago)
$ kill -l
 1) SIGHUP       2) SIGINT       3) SIGQUIT      4) SIGILL       5) SIGTRAP
 6) SIGABRT      7) SIGBUS       8) SIGFPE       9) SIGKILL     10) SIGUSR1
11) SIGSEGV     12) SIGUSR2     13) SIGPIPE     14) SIGALRM     15) SIGTERM
16) SIGSTKFLT   17) SIGCHLD     18) SIGCONT     19) SIGSTOP     20) SIGTSTP
21) SIGTTIN     22) SIGTTOU     23) SIGURG      24) SIGXCPU     25) SIGXFSZ
26) SIGVTALRM   27) SIGPROF     28) SIGWINCH    29) SIGIO       30) SIGPWR
31) SIGSYS      34) SIGRTMIN    35) SIGRTMIN+1  36) SIGRTMIN+2  37) SIGRTMIN+3
38) SIGRTMIN+4  39) SIGRTMIN+5  40) SIGRTMIN+6  41) SIGRTMIN+7  42) SIGRTMIN+8
43) SIGRTMIN+9  44) SIGRTMIN+10 45) SIGRTMIN+11 46) SIGRTMIN+12 47) SIGRTMIN+13
48) SIGRTMIN+14 49) SIGRTMIN+15 50) SIGRTMAX-14 51) SIGRTMAX-13 52) SIGRTMAX-12
53) SIGRTMAX-11 54) SIGRTMAX-10 55) SIGRTMAX-9  56) SIGRTMAX-8  57) SIGRTMAX-7
58) SIGRTMAX-6  59) SIGRTMAX-5  60) SIGRTMAX-4  61) SIGRTMAX-3  62) SIGRTMAX-2
63) SIGRTMAX-1  64) SIGRTMAX

In sqlplus:

SQL> col inst for a10
SQL> select sys_context('userenv','instance') inst, sid, s.serial#, spid from v$session s, v$process p where s.paddr=p.addr and sid=(select sid from v$mystat where rownum=1);

INST              SID    SERIAL# SPID
---------- ---------- ---------- ------------------------
2                 781        103 29120

At OS level of node 2, kill -1 29120. Back to sqlplus. Run the query again. It returns the same 
result. Record that as "no effect". Run kill -2 <pid>. Check in sqlplus again, and so on...



kill -4 <pid>

Next execution of the query in sqlplus shows:

*
ERROR at line 1:
ORA-25408: can not safely replay call

But most of the time, it shows:

*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 12703	<-- OS level pid of *new* session, not the killed one
Session ID: 777 Serial number: 8405	<-- SID and serial# of the *new* session

ORA-25408 is the same error I got when I shutdown the instance.


Most of the signals cause the next SQL execution in sqlplus to throw ORA-3113. The signal numbers 
are in parentheses below.

kill -(4|5-9|11|16|24|25|29-31|34-64) <pid>

*
ERROR at line 1:
ORA-03113: end-of-file on communication channel
Process ID: 12703
Session ID: 777 Serial number: 8405


Some signals have no effect.

kill -(1-3|10|12-15|17|18|20-23|26-28|32|33) <pid>: no effect (for signal 18, see below)


Signal 19 is STOP. If that's sent, signal 18 (CONT) will make it continue.

kill -19 <pid>: stops/hangs process; signal 18 resumes it



Typically, the signal status for the process is as follows (read the binary bits from right to left):

$ grep ^Sig /proc/7932/status
SigQ:   1/62787
SigPnd: 0000000000000000 <-- no signal pending
SigBlk: 0000000000010000 <-- binary 10000000000000000: signal 17 blocked
SigIgn: 0000000006005207 <-- binary 110000000000101001000000111: signals 1-3,10,13,15,26,27 ignored
SigCgt: 00000001c9802cf8 <-- binary 111001001100000000010110011111000: signals 4-8,11,12,14,24,25,28,31,32,33 caught (will be handled)