[Note: see bottom for update to 11g or 12c] Order KST trw files in a cdmp directory [oracle@dbhost cdmp_20120815041643]$ pwd /u01/app/oracle/product/10.2.0/asm/admin/+ASM/bdump/cdmp_20120815041643 [oracle@dbhost cdmp_20120815041643]$ ls +asm1_ckpt_24272.trw +asm1_gmon_24278.trw +asm1_lmd0_24260.trw +asm1_mman_24266.trw +asm1_ora_17587.trw +asm1_ora_24485.trw +asm1_ora_362.trw +asm1_ora_9082.trw +asm1_rbal_24276.trw +asm1_dbw0_24268.trw +asm1_lck0_24290.trw +asm1_lmon_24258.trw +asm1_ora_10300.trw +asm1_ora_17741.trw +asm1_ora_25084.trw +asm1_ora_4733.trw +asm1_pmon_24252.trw +asm1_smon_24274.trw +asm1_diag_24254.trw +asm1_lgwr_24270.trw +asm1_lms0_24262.trw +asm1_ora_16371.trw +asm1_ora_24245.trw +asm1_ora_31884.trw +asm1_ora_7941.trw +asm1_psp0_24256.trw order.sh [oracle@dbhost cdmp_20120815041643]$ cat order.sh cat -- *.trw > tmp grep '^........:' tmp > tmp2 sort -t: -k2 tmp2 > tmp rm tmp2 [oracle@dbhost cdmp_20120815041643]$ sh order.sh [oracle@dbhost cdmp_20120815041643]$ grep 08-15 tmp #all files are created on 08/15 57A55077:11738A5A 19 32 10280 2 kst: trace buffer wrapped on 08-15 00:14:51.703 50663987:11A83422 16 35 10280 2 kst: trace buffer wrapped on 08-15 03:52:41.368 9E1E3663:11AEC7D0 21 27 10280 2 kst: trace buffer wrapped on 08-15 04:14:56.565 A0A8970B:11AEE708 20 0 10280 1 kst: process state object created on 08-15 04:15:40.211 A14C6584:11AEEEF2 2 49 10280 2 kst: trace buffer wrapped on 08-15 04:15:51.204 A16209BD:11AEEFB7 12 39 10280 2 kst: trace buffer wrapped on 08-15 04:15:52.656 A1B2A99B:11AEF360 13 38 10280 2 kst: trace buffer wrapped on 08-15 04:15:58.067 A1EFBF1B:11AEF612 8 43 10280 2 kst: trace buffer wrapped on 08-15 04:16:02.166 A2AE7F04:11AEFE12 4 47 10280 2 kst: trace buffer wrapped on 08-15 04:16:14.967 A37E8874:11AF06D8 11 40 10280 2 kst: trace buffer wrapped on 08-15 04:16:28.929 A3A886DB:11AF0880 10 41 10280 2 kst: trace buffer wrapped on 08-15 04:16:31.747 A3BF4001:11AF0991 15 36 10280 2 kst: trace buffer wrapped on 08-15 04:16:33.272 A406D5FB:11AF0C37 7 44 10280 2 kst: trace buffer wrapped on 08-15 04:16:37.663 A4175722:11AF0D4B 9 42 10280 2 kst: trace buffer wrapped on 08-15 04:16:39.183 A4191CAC:11AF0D83 17 0 10280 1 kst: process state object created on 08-15 04:16:39.302 A43172C2:11AF0EB9 14 37 10280 2 kst: trace buffer wrapped on 08-15 04:16:40.935 A457383D:11AF0FFE 5 46 10280 2 kst: trace buffer wrapped on 08-15 04:16:43.077 A463F326:11AF10B0 6 45 10280 2 kst: trace buffer wrapped on 08-15 04:16:43.912 A468DEFB:11AF1147 17 30 10280 2 kst: trace buffer wrapped on 08-15 04:16:44.655 A46CB345:11AF116D 18 34 10280 2 kst: trace buffer wrapped on 08-15 04:16:44.912 [oracle@dbhost cdmp_20120815041643]$ vi tmp 6E5C7A4E:00000017 3 0 10280 1 kst: process state object created on 08-02 20:22:28.854 ^sequence ^ ^event ^message maybe v$process.pid? A .trc file looks like this: Trace Bucket Dump Begin: default bucket for process 182 (osid: 7072) TIME(*=approx):SEQ:COMPONENT:FILE@LINE:FUNCTION:SECT/DUMP: [EVENT#:PID:SID] DATA ------------------------------------------------------------------------------- 2012-09-02 00:04:48.677991 :90942FDF:db_trace:ktsmg.c@3571:ktsmgCursorScan(): [10449:182:615] hash: 3830645107 elapsed: 114207 fetchtm: 1346448081 curtm: 1346562288 sttm: 1346448081 QRY_EXPRN_TIME(secs): 1800 ^time ^seq ^component ^function ^sect/dump ^data ^file ^line ^event:pid:sid [Note: 11g or 12c] My preliminary study of KST trace files is not very useful. I put it here just so I may resume it in the future from where I left off. In 11g or 12c, trw files don't exist. There're trm files which are read by ADRCI; they're trace metadata files (see Doc 750982.1). The trc files are like the old trw files. To sort all of them by sequence: cat -- *.trc > tmp grep '^201.-..-.. ..:..:..' tmp > tmp2 #can be more restrictive, may need manual check sort -t: -k4 tmp2 > tmp rm tmp2 vi tmp 2014-09-03 13:59:13.888906 :000002CC:db_trace:ksl2.c@14619:ksliwat(): [10005:499:628] KSL POST RCVD poster=25 num=1 loc='kso2.h LINE:473 ID:ksoreq_reply' id1=0 id2=0 name= type=0 fac#=3 posted=0x3 may_be_posted=1 2014-09-03 13:59:13.889003 :000002CD:db_trace:ksb.c@2563:ksbs1p_real(): [10254:499:628] KSBS1P: spawn succeeded ospid = 9751 ... ^time ^seq ^componte^file ^line^function ^event^pid^sid ^data