* Free space viewed from v$asm_diskgroup SQL> select type, total_mb, free_mb, cold_used_mb, required_mirror_free_mb, usable_file_mb from v$asm_diskgroup where name='DATA'; TYPE TOTAL_MB FREE_MB COLD_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB ------ ---------- ---------- ------------ ----------------------- -------------- NORMAL 4607928 968998 3638930 255996 356501 SQL> select (356501+255996/2)/968998 from dual; (356501+255996/2)/968998 ------------------------ .5 This means that in a 2-way mirrored normal redundancy diskgroup, the total free space (968998 MB) is split into two, each able to accomodate usable file space (356501 MB) plus half of required mirror free space (255996/2 MB) as overhead. Note that the first four numbers in the SQL are for the whole diskgroup, not a mirror in it (total_mb, free_mb, cold_used_mb, required_mirror_free_mb), while usable_file_mb takes into account of the mirrors. If you set db_recovery_file_dest_size, it may be a good idea to set it to (total_mb-required_mirror_free_mb)/2 for the diskgroup pointed to by db_recovery_file_dest, if the diskgroup is exclusively used by this database. Change 2 to 3 if you have 3 mirrors in the diskgroup as in case of high redundancy. * More logging for ASMLib By default, only ASMLib errors and notices are logged in /var/log/messages, which is controlled by: $ cat /proc/fs/oracleasm/log_mask ENTRY deny EXIT deny DISK off REQUEST off BIO off IOC off ABI off ERROR allow NOTICE allow One Oracle note talks about it: FAQ ASMLIB CONFIGURE,VERIFY, TROUBLESHOOT (Doc ID 359266.1) ASMLIB provides additional logging for the following areas: ENTRY /* func call entry */ EXIT /* func call exit */ DISK /* Disk information */ REQUEST /* I/O requests */ BIO /* bios backing I/O */ IOC /* asm_iocs */ ABI /* ABI entry points */ There are three possible values: deny off allow But the example in that note is not very clear. According to Bug 6041447 : DATABASE CRASHES WHEN ONE OF THE PATHS ARE OFFLINED ON A MULTIPATH STORAGE all you need to do is e.g. echo "BIO REQUEST ENTRY EXIT allow" > /proc/fs/oracleasm/log_mask if you want to log BIOS-backed I/O's, I/O requests, etc. Exactly what is logged for each message type can be found in the source code available from https://oss.oracle.com/projects/oracleasm/files/sources/ Check kernel/oracleasm.c for strings like "mlog(ML_NOTICE". * Using ASMLib, disk file names must match disk names inside disks for i in /dev/oracleasm/disks/*; do dskname=$(kfed read $i | awk '$1~/^kfdhdb.dskname/{print $2}') echo "Checking $dskname ..." if [[ $dskname != $(basename $i) ]]; then echo "$i disk label mismatch!" fi done You can manually run kfed on one disk as follows: kfed read /dev/oracleasm/disks/ASM_BACKUP01_DCG | awk '$1~/^kfdhdb.dskname/{print $2}' The output should be ASM_BACKUP01_DCG * 12c AFD (ASM Filter Driver) is selected by default. To deconfigure AFD on standalone DB server: https://docs.oracle.com/database/121/OSTMG/GUID-6AABD466-CC4D-4D17-92DB-42D65FB713F2.htm # . oraenv +ASM # crsctl stop has # acfsload stop # asmcmd afd_deconfigure ASMCMD-9520: AFD is not 'Supported' $ asmcmd afd_state ASMCMD-9545: The installed AFD drivers are not correct for this operating system. Alternatively, completely cleanup Oracle software by rm -rf on all Oracle related files and directories in each of the file systems. To unload the oracleafd module, manually remove the link /etc/rc.d/rc3.d/s96afd and reboot. Then remove the .ko files. Ref: https://community.oracle.com/thread/4027233 * 12c ASM password file Check password: sqlplus sys/password as sysasm sqlplus sys@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=dbhost)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=+ASM))) as sysasm or you may have to use SID and add UR=A: sqlplus sys@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=dbhost)(PORT=1521))(CONNECT_DATA=(SERVER=DEDICATED)(SID=+ASM)(UR=A))) as sysasm Offload password to file system: orapwd input_file=+DATA/orapwasm file=/tmp/PasswordFileExtractedFromASM That file can serve as backup or be used for standby. Load into ASM from file system: orapwd input_file=PasswordFileExtractedFromASM file=+DATA/orapwASM asm=y force=y Newer versions have pw* commands in asmcmd. To find the password location in ASM: pwget --asm To backup to file system: pwcopy /tmp/orapwASM.bak To change sys password, just use SQL command: alter user sys identified by ... In case of RAC, if the password file is in ASM, that' all you need to do. If the password file is on the file system ($GIHOME/dbs/orapw+ASM), and if ASM instance is running on the other node(s), the password file on remote node(s) will be updated. But if ASM is not up on remote node(s), you have to scp the newly updated password file from this node to the other node(s). Ref: How To Recreate The ASM Password File And Change ASM Users Password On 11.2 RAC. (Doc ID 1638586.1) How to recreate shared ASM password file in 12c GI cluster (Doc ID 1929673.1) How to recreate shared ASM password file in 19c Grid Infrastructure (GI) (Doc ID 2717306.1) How To Change ASM SYS PASSWORD ? (Doc ID 452076.1) How to connect to ASM instance from a remote client (Oracle Net) (Doc ID 340277.1) * If you use ASMLIB, after Linux kernel update, oracleasm module must be upgraded. Download the new version from http://www.oracle.com/technetwork/topics/linux/index-101839.html#oracleasm_2_0 On the actual download page (e.g. page for Red Hat Enterprise Linux 5 AS), find in page (^F in browser) whatever `uname -r` says. Usually there're 2 drivers, one for 64-bit, the other for 32-bit. Also make sure you're looking at the correct architecture. Install the rpm package: rpm -ivh ... Optionally, uninstall the old package: rpm -e ... Then as root modprobe oracleasm #install the module into kernel lsmod | grep oracleasm #check to see it's installed modinfo oracleasm #verify module version Enable Oracle to use the new module: /etc/init.d/oracleasm enable #this will also scandisks /etc/init.d/oracleasm listdisks #verify * Unless you use OEL (Oracle's Linux), use udev instead of ASMLIB. Consider adding /bin/chown oracle:dba /dev/mapper/%cp1; /bin/chmod 660 /dev/mapper/%cp1 into the /bin/bash -c '...' string of the RUN+= action for the /sbin/dmsetup info line. Be aware that you may need to add it again if udev is upgraded or patched. So it may be better to create a separate rules file, with a number smaller than the standard, say, 39-my_multipath.rules. Use /dev/mapper/* as asm_diskstring. If voting file is not in ASM, you can choose to postpone changing ownership/permissions by adding the commands to /etc/rc.local. * If you want to create ASMLib on Red Hat 6 yourself: Download the latest snapshot of asmlib from https://oss.oracle.com/git/gitweb.cgi?p=oracleasm.git;a=summary run autogen.sh, configure, make, make install. Ref: http://www.freelists.org/post/oracle-l/udev-VS-oracleasm-disk-aliasing-preferences,9 http://www.itpub.net/thread-1556963-2-1.html [Update 2013-07] Oracle offers ASMLib for Red Hat 6: http://www.oracle.com/technetwork/server-storage/linux/asmlib/rhel6-1940776.html * oracleasm createdisk fails with no other message /etc/init.d/oracleasm createdisk is notorious for not giving enough info for debugging. You could use asmtool directly to force createdisk (Ref: 469163.1, 782872.1, etc). But a more systematic troubleshooting method is still highly preferred. One way to achieve this is to read the write() calls: # strace -s 100 -f -e write /etc/init.d/oracleasm createdisk VOL1 /dev/mapper/asmvol1p1 or # strace -s 100 -f /etc/init.d/oracleasm createdisk VOL1 /dev/mapper/asmvol1p1 2>&1 | grep write ... [pid 22707] write(1, "Device \"/dev/mapper/VG0-LV8\" is already labeled for ASM disk \"VOL1\"\n", 68) = 68 It's interesting that writing to file descriptor 1 still does not show the error on screen, which only has Marking disk "VOL1" as an ASM disk: [FAILED] Anyway, with the specific error shown in the write() call, you know you can wipe out the header and createdisk again: dd if=/dev/mapper/asmvol1p1 count=10 | strings #optional, check to see if the ASM header is readlly there dd if=/dev/zero of=/dev/mapper/asmvol1p1 count=100 /etc/init.d/oracleasm createdisk ... * `dd if= count=10 | strings' is a crude way to see if the disk (partition) has an ASM header. You can also use `kfed read '. If it's not ASM disk, you'll get kfbh.type: 0 ; 0x002: KFBTYP_INVALID ... ...[some binary dump]... KFED-00322: file not found; arguments: [kfbtTraverseBlock] [Invalid OSM block type] [] [0] For more about kfed, see entries for kfed and kfod at ../computer/oraclebin.html. Interesting message: http://www.freelists.org/post/oracle-l/Setting-up-storage-Array-for-ASM,6 -------------------- Old Note -------------------- Q: cssd is not started on reboot (no process name matches cssd.bin although there're ocssd), and so ASM and database instances are not started. A: /dev/raw/raw* may be owned by root after reboot. Run chown oracle:dba /dev/raw/raw* /etc/init.d/init.cssd start sleep 90 export ORACLE_SID=+ASM1 sqlplus / as sysdba startup export ORACLE_SID=RACDB1 sqlplus / as sysdba startup Add the above chown line to the end of /etc/rc.d/rc.local so the ownership persists after reboot. Q: sqlplus / as sysdba throws ORA-1031 (insufficient privileges) on Linux/UNIX. But sqlplus sys/oracle as sysdba works. A: Make sure $ORACLE_HOME/network/admin/sqlnet.ora does not have SQLNET.AUTHENTICATION_SERVICES=(NTS) Q: ASM instance is up and v$asm_disgroup shows MY_DG2 is mounted. This command srvctl modify instance -d racdb -i racdb1 -s +ASM1 per Note:276208.1 is run. But startup the database instance throws ORA-205 (error in identifying control file): Wed Jan 18 14:42:07 2006 ORA-00202: control file: '+MY_DG2/racdb/controlfile/current.260.562151423' ORA-17503: ksfdopn:2 Failed to open file +MY_DG2/racdb/controlfile/current.260.562151423 ORA-15001: diskgroup "MY_DG2" does not exist or is not mounted ORA-15055: unable to connect to ASM instance ORA-01031: insufficient privileges Wed Jan 18 14:42:07 2006 ORA-205 signalled during: ALTER DATABASE MOUNT... Shutting down instance: further logons disabled A: SQLNET.AUTHENTICATION_SERVICES should not be set to NTS on Linux/UNIX and must be set to NTS on Windows. Database instance startup must login without specifying password (orapwd file must be used so make sure that's set correctly; if necessary, recreate orapwd). http://download-west.oracle.com/docs/cd/B13789_01/server.101/b10739/storeman.htm says [On ASM, startup] MOUNT Mounts the disk groups specified in the ASM_DISKGROUPS initialization parameter NOMOUNT Starts up the ASM instance without mounting any disk groups Here's my +ASM1: SQL> select * from v$database; select * from v$database * ERROR at line 1: ORA-01507: database not mounted SQL> startup mount ORA-01081: cannot start already-running ORACLE - shut it down first SQL> select instance_name, status, parallel, logins, database_status, instance_role from v$instance; INSTANCE_NAME STATUS PAR LOGINS DATABASE_STATUS INSTANCE_ROLE ---------------- ------------ --- ---------- ----------------- ------------------ +ASM1 STARTED YES ALLOWED ACTIVE UNKNOWN 'STARTED' status means startup nomount. So does 'UNKNOWN' instance_role. -------------------------------------------------------------------------------- column parameter format a37 column description format a30 word_wrapped column "Session Value" format a10 column "Instance Value" format a10 select a.ksppinm "Parameter", a.ksppdesc "Description", b.ksppstvl "Session Value", c.ksppstvl "Instance Value" from x$ksppi a, x$ksppcv b, x$ksppsv c where a.indx = b.indx and a.indx = c.indx and a.ksppinm like '%asm%' escape '\' order by 1 / Parameter Description Session Va Instance V ------------------------------------- ------------------------------ ---------- ---------- _asm_acd_chunks initial ACD chunks created 1 1 _asm_allow_only_raw_disks Discovery only raw devices TRUE TRUE _asm_allow_resilver_corruption Enable disk resilvering for FALSE FALSE external redundancy _asm_ausize allocation unit size 1048576 1048576 _asm_blksize metadata block size 4096 4096 _asm_disk_repair_time seconds to wait before 14400 14400 dropping a failing disk _asm_droptimeout timeout before offlined disks 60 60 get dropped (in 3s ticks) _asm_emulmax max number of concurrent disks 10000 10000 to emulate I/O errors _asm_emultimeout timeout before emulation 0 0 begins (in 3s ticks) _asm_kfdpevent KFDP event 0 0 _asm_libraries library search order for ufs ufs discovery _asm_maxio Maximum size of individual I/O 1048576 1048576 request _asm_stripesize ASM file stripe size 131072 131072 _asm_stripewidth ASM file stripe width 8 8 _asm_wait_time Max/imum time to wait before 18 18 asmb exits _asmlib_test Osmlib test event 0 0 _asmsid ASM instance id asm asm asm_diskgroups disk groups to mount MY_DG1, MY MY_DG1, MY automatically _DG2 _DG2 asm_diskstring disk set locations for discovery asm_power_limit number of processes for disk 1 1 rebalancing 20 rows selected. -------------------------------------------------------------------------------- http://download-west.oracle.com/docs/cd/B13789_01/server.101/b10739/storeman.htm says ASM shutdown NORMAL means "ASM waits for the connected ASM instances (and other ASM SQL sessions) to exit before shutting down the instance." It probably means "connected database instances (and other database SQL sessions)". SQL> shutdown ORA-15097: cannot SHUTDOWN ASM instance with connected RDBMS instance SQL> shutdown normal ORA-15097: cannot SHUTDOWN ASM instance with connected RDBMS instance SQL> select instance_name, db_name, status from v$asm_client; INSTANCE_NAME DB_NAME STATUS ---------------------------------------------------------------- -------- ------------ RACDB1 RACDB CONNECTED -------------------------------------------------------------------------------- CREATE TABLESPACE TEST db_create_file_dest is set to '+MY_DG2'. create tablespace testts; dba_tablespaces shows: initial_extent=65536, LMT, sysetm allocation, ASSM dba_data_files shows: filename='+MY_DG2/racdb/datafile/testts.273.579021687', bytes=100mb, blocks=12800, autoextensible='YES', maxbytes=3.4360E+10, maxblocks=4194302, increment_by=12800, user_blocks=12792 (8 blocks smaller than total blocks) create tablespace testts datafile '+MY_DG2' size 1m; dba_tablspaces shows the same dba_data_files shows: filename='+MY_DG2/racdb/datafile/testts.273.579021797', bytes=1048576, blocks=128, auto='NO', maxbytes=0, increment=0, user_blocks=120 create tablespace testts datafile '+MY_DG2' size 1m extent management local uniform size 128k; dba_tablespaces shows: initial/next extent=131072, min_extents=1, max=2147483645, allocation='UNIFORM' dba_data_files shows: blocks=128, user_blocks=112 (16 blocks smaller) -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- --------------------------------------------------------------------------------