> We have 2-node Oracle RAC database setup on 2 linux boxes sharing OCFS file > system. OCFS has several partitions, some of which are shown as 100% full ( > using "df -k" or "du -ks"). We have autoextensible datafiles residing in those > partitions, so theoretically these files should not be able to extend, but so > far we do not have any errors. Does this mean that "df/du" is not the right > tools to monitor OCFS utilization du/df should still work. Try creating another datafile on your 100% full filesystem. If you can create it with a very small size, resize it larger. You'll get Linux Error: 28: No space left on device. My test on Linux RAC shows that when df -k says I have 4096K left on 1.2GB /ocfs and 100% Used, I can still increase filesize a little at a time until only 1792K is left. One anomaly, though, is that sometimes I can't make a big jump to a large size but I *can* if I do it with multiple small jumps: SQL> alter database datafile '/ocfs/O10G/ts.dbf' resize 85m; alter database datafile '/ocfs/O10G/ts.dbf' resize 85m * ERROR at line 1: ORA-01237: cannot extend datafile 8 ORA-01110: data file 8: '/ocfs/O10G/ts.dbf' ORA-27059: could not reduce file size Linux Error: 28: No space left on device Additional information: 2 SQL> alter database datafile '/ocfs/O10G/ts.dbf' resize 83m; Database altered. SQL> alter database datafile '/ocfs/O10G/ts.dbf' resize 84m; Database altered. SQL> alter database datafile '/ocfs/O10G/ts.dbf' resize 85m; Database altered. Isn't that interesting? My DB version is 10.1.0.2.0. db_block_size 2048. OCFS is # rpm -qa | grep ocfs ocfs-support-1.0.10-1 ocfs-tools-1.0.10-1 ocfs-2.4.21-EL-1.0.12-1 Or check this way: # cat /proc/ocfs/version 1.0.12-PROD1 Fri May 28 14:58:57 PDT 2004 (build b69b4447c830d8da62c911c84601a08a) uname -r: 2.4.21-15.EL. Red Hat Enterprise Linux AS release 3 (Taroon Update 2) -------------------------------------------------------------------------------- Access denied caused by O_DIRECT? You can't view the database files on an OCFS filesystem when they're opened by an Oracle instance even if you're the owner of the files or you're root; strings or od (or cat) throws access denied error. But a datafile for an off-lined tablespace can be viewed. You can also create a regular file (by using vi e.g.) on OCFS and the file is readable. Note:252331.1 explains that "When the database opens a file on ocfs, it opens it with an extra flag 'O_DIRECT'. once a file on any node is opened with O_DIRECT, regular file access is not allowed". Using Oracle's customized dd utility can access it. Actually, I don't buy that explanation. Sure, the first part of the statement is a correct observation. But it's more of a side effect than the cause of inaccessibility. The reason is that on Linux ext2 (or ext3), you can set filesystemio_options to directio in Oracle and the files will also be opened with O_DIRECT flag and, no surprise, those files are still accessible by regular commands! Since Linux doesn't have pfiles command as on Solaris, the way to look at the file open flags is to trace the opening action (strace -e trace=open -p . You can offline and online a tablespace to watch open() calls): open("/home/oracle/ts.dbf", O_RDWR|O_SYNC|O_DIRECT|O_LARGEFILE) = 16 and that file, on ext3 filesystem, can be accessed by regular commands such as `strings`, even though O_DIRECT was set (it was set because filesystemio_options parameter is set to directio). However, if filesystemio_options is none (which is default), only OCFS datafiles have O_DIRECT flag set: open("/ocfs/O10G/ts.dbf", O_RDWR|O_SYNC|O_DIRECT|O_LARGEFILE) = 28 Now if a datafile is created on a non-OCFS filesystem such as my /home/oracle, you'll see O_RDWR|O_SYNC|O_LARGEFILE but not O_DIRECT. Incidentally, redo logfiles are also opened with O_DIRECT flag. If I alter database add logfile thread 1 group 5 '/ocfs/O10G/redo05.log' size 10486272 in sqlplus and strace -e trace=open -p , I see open("/ocfs/O10G/redo05.log", O_RDONLY|O_DIRECT|O_LARGEFILE) = 30 open("/ocfs/O10G/redo05.log", O_RDWR|O_SYNC|O_DIRECT|O_LARGEFILE) = 30 But strings /ocfs/O10G/redo05.log throws "Permission denied" error. -------------------------------------------------------------------------------- If you want to umount (as you do before you run resizeocfs), you get "device busy". But fuser /ocfs doesn't show any process. (fuser -v /ocfs says kernel has mount on it.) That's very strange. But you can use lsof | grep /ocfs to find all those processes. -------------------------------------------------------------------------------- debugocfs shows wrong mount point: # debugocfs -h /ocfs | grep mount_point mount_point = /oracle I don't have /oracle directory. But it matches this: # cat /proc/ocfs/0/mountpoint /oracle -------------------------------------------------------------------------------- Pass "correct" file and directory path to debugocfs (... means omitted to save space): # mount | grep ocfs /dev/sdd1 on /ocfs type ocfs (rw,_netdev,reclaimid) # ls -l /ocfs/ocr_file -rw-r----- 1 root dba 2666496 Jul 9 17:02 /ocfs/ocr_file # debugocfs -f /ocfs/ocr_file /dev/sdd1 #no fileinfo is shown debugocfs 1.0.10-PROD1 Fri Mar 5 14:42:39 PST 2004 (build fcb0206676afe0fcac47a99c90de0e7b) fileinfo: # debugocfs -f /ocr_file /dev/sdd1 #now it does debugocfs 1.0.10-PROD1 Fri Mar 5 14:42:39 PST 2004 (build fcb0206676afe0fcac47a99c90de0e7b) fileinfo: Name = /ocr_file curr_master = 0 file_lock = OCFS_DLM_ENABLE_CACHE_LOCK ... # debugocfs -d /ocfs /dev/sdd1 debugocfs 1.0.10-PROD1 Fri Mar 5 14:42:39 PST 2004 (build fcb0206676afe0fcac47a99c90de0e7b) dirinfo: # debugocfs -d / /dev/sdd1 debugocfs 1.0.10-PROD1 Fri Mar 5 14:42:39 PST 2004 (build fcb0206676afe0fcac47a99c90de0e7b) dirinfo: Name = / curr_master = 0 file_lock = OCFS_DLM_NO_LOCK ... -------------------------------------------------------------------------------- Is OCFS automatically mounted with noatime option? Commands show inconsistent results. I'll test to make sure atime is not updated: # mount -v | grep ocfs /dev/sdd1 on /ocfs type ocfs (rw,_netdev,reclaimid) # grep ocfs /etc/fstab /dev/sdd1 /ocfs ocfs _netdev,reclaimid 0 0 /dev/sdd1 # grep ocfs /proc/mounts #Where does this noatime come from? (And _netdev,reclaimid is gone) /dev/sdd1 /ocfs ocfs rw,noatime 0 0 -------------------------------------------------------------------------------- $ORACLE_HOME/bin/dbfsize can only run against a datafile when the tablespace is offline. It reports file type as "file system" instead of raw: $ dbfsize ts.dbf Database file: ts.dbf Database file type: file system Database file size: 64 2048 byte blocks In contrast, Veritas QIO-based Oracle datafiles are reported to be "raw device" in file type. -------------------------------------------------------------------------------- I want to find out the actual driver transfer size for the OCFS filesystem. mount | grep ocfs says it's on /dev/sdd1. I believe this is dev8-3 in sar -d output, which has sect/s=127.96 and tps=10.03. So the transfer size is about 12.8 sectors or 6532 bytes. I have to verify.