Extensive search on Metalink for "waiting for logins to complete" shows that the shutdown hanging could be caused by 1. OEM or dbsnmp running against the database. dbsnmp has been known to hang database shutdown even with immediate option. But it's possible this bug was finally fixed in latest versions (9i or maybe later versions of 8.1.7? Can you confirm?). Look at these articles on Metalink: 7.3.3: metalink.oracle.com/metalink/plsql/ml2_documents.showFOR?p_id=36912.996 8.0.5: metalink.oracle.com/metalink/plsql/ml2_documents.showFOR?p_id=49100.999 8.0.6: metalink.oracle.com/metalink/plsql/ml2_documents.showFOR?p_id=254554.999 8.1.7.0: metalink.oracle.com/metalink/plsql/ml2_documents.showFOR?p_id=375535.999 2. bdump directory not accessible 3. Dangling distributed transaction has some problems (metalink.oracle.com/metalink/plsql/ml2_documents.showDocument?p_database_id=FOR&p_id=281316.999) and committing or rolling back session force the session in dba_2pc_pending should help. 4. A bug related to auditing. 5. A bug related to MTS. Note:118228.1 (Hang During Startup/Shutdown on Unix When System Uptime > 248 Days) Regardless the cause, shutdown abort is the way to go. Our cold backup starts up the database restrict and then shuts down normal anyway. Yong P.S. marty.grinstead AT oracle.com emailed me saying dbsnmp causes shutdown immediate to hang only if incorrectly configured. -----Original Message----- From: Huang, Yong Sent: Monday, December 31, 2001 11:41 AM Subject: RE: Problem: llnk database or listener llnk shutdown at 20:10pm Sat. hung till this morning Gerald came in and aborted it. The error is Sat Dec 29 20:15:52 2001 SHUTDOWN: waiting for logins to complete. Mon Dec 31 00:48:56 2001 Shutting down instance (abort) License high water mark = 3 I don't know what happened. I'll look it up on Metalink. Yong The case for port on 20020202: oracle AT eris:port:/app/oracle/admin/port/bdump>tail alert_port.log Sat Feb 2 20:46:29 2002 ARC0: Completed archiving log# 3 seq# 3531 Sat Feb 2 20:53:58 2002 Restarting dead background process EMN0 EMN0 started with pid=27 Sat Feb 2 20:54:00 2002 Shutting down instance (immediate) License high water mark = 175 Sat Feb 2 20:59:10 2002 SHUTDOWN: waiting for active calls to complete. oracle AT eris:port:/app/oracle/admin/port/udump>ps -ef | grep port oracle 16699 16698 0 20:53:58 ? 0:01 oracleport (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq))) livelink 16629 16625 0 20:53:18 ? 0:00 otsearch -otstarts -server -port 8500 -mem 8 -inifile /app/livelink/config/open oracle 10117 1 0 Jan 26 ? 0:09 ora_ckpt_port livelink 16636 16625 0 20:53:21 ? 0:00 ipupdate -port 8501 -hostname eris -readipool file:///data/livelink/index/enter oracle 10115 1 0 Jan 26 ? 9:05 ora_lgwr_port oracle 10111 1 0 Jan 26 ? 0:01 ora_pmon_port livelink 16633 16630 0 20:53:19 ? 0:00 mltcon -port 8501 -updateport 8502 -inifile /app/livelink/config/opentext.ini oracle 10131 1 0 Jan 26 ? 1:05 ora_arc0_port oracle 16701 1 0 20:53:58 ? 0:00 ora_emn0_port oracle 1216 1 0 Jan 20 ? 0:00 f60srvm port=9001 pool=1 mode=socket oracle 25324 1 0 19:30:47 ? 0:15 oracleport (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) oracle 10113 1 0 Jan 26 ? 0:28 ora_dbw0_port oracle 16656 16654 0 20:53:49 ? 0:00 /bin/ksh /app/oracle/local/scripts/blu_backup/coldbackup.sh port oracle 29077 27316 0 22:03:26 pts/0 0:00 grep port oracle 16654 2892 0 20:53:49 ? 0:00 /bin/ksh /app/oracle/local/scripts/blu_backup/coldbackup_port.sh oracle 10121 1 0 Jan 26 ? 0:04 ora_reco_port oracle 10119 1 0 Jan 26 ? 0:05 ora_smon_port oracle 29075 29060 4 22:03:19 ? 0:06 /u76/gnu/SunOS5.6_sun4/bin/gzip --fast /f04/archives/port/arch_1_3531.arc oracle 26931 1 0 Jan 31 ? 0:01 oracleport (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) oracle 16660 16656 0 20:53:49 ? 0:00 /bin/sh /app/oracle/local/scripts/stop_db port oracle AT eris:port:/app/oracle/admin/port/udump>truss -aflp 26931 26931/1: psargs: oracleport (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 26931/1: read(15, 0x01A0DC3E, 2064) (sleeping...) oracle AT eris:port:/app/oracle/admin/port/udump>truss -aflp 25324 25324/1: psargs: oracleport (DESCRIPTION=(LOCAL=no)(ADDRESS=(PROTOCOL=BEQ))) 25324/1: poll(0xEFFF5498, 1, -1) (sleeping...) oracle AT eris:port:/app/oracle/admin/port/udump>truss -aflp 16699 16699/1: psargs: oracleport (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq))) 16699/1: Received signal #14, SIGALRM, in semop() [caught] 16699/1: semop(1245185, 0xEFFFC544, 1) Err#91 ERESTART 16699/1: sigprocmask(SIG_BLOCK, 0xEFFFC070, 0x00000000) = 0 16699/1: sigprocmask(SIG_UNBLOCK, 0xEFFFC070, 0x00000000) = 0 16699/1: getcontext(0xEFFFBE38) 16699/1: setcontext(0xEFFFBE38) 16699/1: sigprocmask(SIG_BLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: setitimer(ITIMER_REAL, 0xEFFFC2D4, 0x00000000) = 0 16699/1: sigprocmask(SIG_UNBLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: getcontext(0xEFFFC240) 16699/1: sigprocmask(SIG_BLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: setitimer(ITIMER_REAL, 0xEFFFC2D4, 0x00000000) = 0 16699/1: sigprocmask(SIG_UNBLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: Received signal #14, SIGALRM, in semop() [caught] 16699/1: semop(1245185, 0xEFFFC544, 1) Err#91 ERESTART 16699/1: sigprocmask(SIG_BLOCK, 0xEFFFC070, 0x00000000) = 0 16699/1: sigprocmask(SIG_UNBLOCK, 0xEFFFC070, 0x00000000) = 0 16699/1: getcontext(0xEFFFBE38) 16699/1: setcontext(0xEFFFBE38) 16699/1: sigprocmask(SIG_BLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: setitimer(ITIMER_REAL, 0xEFFFC2D4, 0x00000000) = 0 16699/1: sigprocmask(SIG_UNBLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: getcontext(0xEFFFC240) 16699/1: sigprocmask(SIG_BLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: setitimer(ITIMER_REAL, 0xEFFFC2D4, 0x00000000) = 0 16699/1: sigprocmask(SIG_UNBLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: Received signal #14, SIGALRM, in semop() [caught] 16699/1: semop(1245185, 0xEFFFC544, 1) Err#91 ERESTART 16699/1: sigprocmask(SIG_BLOCK, 0xEFFFC070, 0x00000000) = 0 16699/1: sigprocmask(SIG_UNBLOCK, 0xEFFFC070, 0x00000000) = 0 16699/1: getcontext(0xEFFFBE38) 16699/1: setcontext(0xEFFFBE38) 16699/1: sigprocmask(SIG_BLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: setitimer(ITIMER_REAL, 0xEFFFC2D4, 0x00000000) = 0 16699/1: sigprocmask(SIG_UNBLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: getcontext(0xEFFFC240) 16699/1: sigprocmask(SIG_BLOCK, 0xEFFFC34C, 0x00000000) = 0 16699/1: setitimer(ITIMER_REAL, 0xEFFFC2D4, 0x00000000) = 0 16699/1: sigprocmask(SIG_UNBLOCK, 0xEFFFC34C, 0x00000000) = 0 oracle AT eris:port:/app/oracle/admin/port/udump>truss -aflp 16701 16701/1: psargs: ora_emn0_port 16701/1: semop(1245185, 0xEFFFE0BC, 1) (sleeping...) 16701/1: Received signal #14, SIGALRM, in semop() [caught] 16701/1: semop(1245185, 0xEFFFE0BC, 1) Err#91 ERESTART 16701/1: sigprocmask(SIG_BLOCK, 0xEFFFDBE8, 0x00000000) = 0 16701/1: sigprocmask(SIG_UNBLOCK, 0xEFFFDBE8, 0x00000000) = 0 16701/1: getcontext(0xEFFFD9B0) 16701/1: setcontext(0xEFFFD9B0) 16701/1: sigprocmask(SIG_BLOCK, 0xEFFFDEC4, 0x00000000) = 0 16701/1: setitimer(ITIMER_REAL, 0xEFFFDE4C, 0x00000000) = 0 16701/1: sigprocmask(SIG_UNBLOCK, 0xEFFFDEC4, 0x00000000) = 0 16701/1: getcontext(0xEFFFDDB8) 16701/1: sigprocmask(SIG_BLOCK, 0xEFFFDEC4, 0x00000000) = 0 16701/1: setitimer(ITIMER_REAL, 0xEFFFDE4C, 0x00000000) = 0 16701/1: sigprocmask(SIG_UNBLOCK, 0xEFFFDEC4, 0x00000000) = 0 16701/1: Received signal #14, SIGALRM, in semop() [caught] 16701/1: semop(1245185, 0xEFFFE0BC, 1) Err#91 ERESTART 16701/1: sigprocmask(SIG_BLOCK, 0xEFFFDBE8, 0x00000000) = 0 16701/1: sigprocmask(SIG_UNBLOCK, 0xEFFFDBE8, 0x00000000) = 0 16701/1: getcontext(0xEFFFD9B0) 16701/1: setcontext(0xEFFFD9B0) 16701/1: sigprocmask(SIG_BLOCK, 0xEFFFDEC4, 0x00000000) = 0 16701/1: setitimer(ITIMER_REAL, 0xEFFFDE4C, 0x00000000) = 0 16701/1: sigprocmask(SIG_UNBLOCK, 0xEFFFDEC4, 0x00000000) = 0 16701/1: getcontext(0xEFFFDDB8) 16701/1: sigprocmask(SIG_BLOCK, 0xEFFFDEC4, 0x00000000) = 0 16701/1: setitimer(ITIMER_REAL, 0xEFFFDE4C, 0x00000000) = 0 16701/1: sigprocmask(SIG_UNBLOCK, 0xEFFFDEC4, 0x00000000) = 0 ^Coracle AT eris:port:/app/oracle/admin/port/udump>ps -ef | grep [s]nmp root 204 1 0 Jan 20 ? 0:00 /etc/opt/SUNWconn/atm/bin/atmsnmpd -n root 1311 1 0 Jan 20 ? 0:00 /usr/lib/snmp/snmpdx -y -c /etc/snmp/conf root 1321 1 0 Jan 20 ? 0:00 /usr/lib/dmi/snmpXdmid -s eris I noticed that even after killing -9 the 3 processes, port database was still hanging (with two lines of "Starting ORACLE instance (normal)"). So I went to internal and shutdown abort. Now the database is down.