12c rac 实例无法启动之磁盘组空间耗尽

1、实例启动失败 SYS@orcl1> startup ORACLE instance started. Total System Global Area 1241513984 bytes Fixed Size 2923872 bytes Variable Size 620757664 bytes Database Buffers 603979776 bytes Redo Buffers 13852672 bytes Database mounted. ORA-03113: end-of-file on communication channel Process ID: 8482 Session ID: 1 Serial number: 48907

2、查看日志部分省略............... Starting background process ARC2 Starting background process ARC3 ARC2 started with pid=44, OS id=8737 Unable to create archive log file '+DATADG' Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_8659.trc: ORA-19504: failed to create file "+DATADG" ORA-17502: ksfdcre:4 Failed to create file +DATADG ORA-15041: diskgroup "DATADG" space exhausted ARCH: Error 19504 Creating archive log file to '+DATADG' Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_8659.trc: ORA-16038: log 2 sequence# 108 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 2 thread 1: '+DATADG/ORCL/ONLINELOG/group_2.263.954539539' ARC3 started with pid=45, OS id=8739 System state dump requested by (instance=1, osid=8659), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_8552_20171231165944.trc USER (ospid: 8659): terminating the instance due to error 16038 Dumping diagnostic data in directory=[cdmp_20171231165944], requested by (instance=1, osid=8659), summary=[abnormal insta nce termination]. Instance terminated by USER, pid = 8659

3、确认磁盘组空间大小 <rac02:+ASM2:/home/grid>$asmcmd -p ASMCMD [+] > lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 4096 1048576 10240 5763 0 5763 0 Y CRSDG/ MOUNTED EXTERN N 512 4096 1048576 6144 13 0 13 0 N DATADG/ ASMCMD [+] >

5、注意需要使用rman进行crosscheck ,然后再删除归档日志在控制文件中的记录。 RMAN> crosscheck archivelog all; using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=66 instance=orcl2 device type=DISK validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 RECID=11 STAMP=962120097 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 RECID=13 STAMP=962074554 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 RECID=14 STAMP=962074555 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_53.288.962121385 RECID=19 STAMP=962121385 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_54.289.962184681 RECID=20 STAMP=962184682 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_55.291.962185743 RECID=22 STAMP=962185742 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_56.292.962188105 RECID=23 STAMP=962188106 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_57.295.962188239 RECID=26 STAMP=962188238 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_58.297.962188923 RECID=28 STAMP=962188924 部分省略.................................. validation succeeded for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_31/thread_2_seq_109.288.964199163 RECID=132 STAMP=964199165 validation succeeded for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_31/thread_2_seq_110.325.964199595 RECID=135 STAMP=964199601 Crosschecked 124 objects RMAN> RMAN> delete expired archivelog all; released channel: ORA_DISK_1 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=66 instance=orcl2 device type=DISK List of Archived Log Copies for database with db_unique_name ORCL ===================================================================== Key Thrd Seq S Low Time ------- ---- ------- - -------------------- 11 1 50 X 07-DEC-2017 15:24:55 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 13 1 51 X 07-DEC-2017 15:34:57 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 14 1 52 X 07-DEC-2017 02:55:53 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 19 1 53 X 07-DEC-2017 15:56:20 部分省略 ........................................... 51 2 66 X 08-DEC-2017 07:41:03 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_66.319.962225883 53 2 67 X 08-DEC-2017 20:57:57 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_67.321.962225885 57 2 68 X 09-DEC-2017 10:16:05 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_68.325.962273785 59 2 69 X 09-DEC-2017 10:16:23 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_69.327.962274133 Do you really want to delete the above objects (enter YES or NO)? yes deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 RECID=11 STAMP=962120097 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 RECID=13 STAMP=962074554 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 RECID=14 STAMP=962074555 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_53.288.962121385 RECID=19 STAMP=962121385 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_54.289.962184681 RECID=20 STAMP=962184682 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_55.291.962185743 RECID=22 STAMP=962185742 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_56.292.962188105 RECID=23 STAMP=962188106 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_57.295.962188239 RECID=26 STAMP=962188238 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_58.297.962188923 RECID=28 STAMP=962188924 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_59.298.962206427 RECID=29 STAMP=962206427 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_60.300.962208543 RECID=31 STAMP=962208544 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_61.302.962223761 RECID=48 STAMP=962177843 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_62.305.962223817 RECID=36 STAMP=962223817 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_63.306.962223899 RECID=37 STAMP=962223899 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_64.309.962224067 RECID=40 STAMP=962224066 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_65.310.962224095 RECID=41 STAMP=962224094 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_66.311.962224101 RECID=42 STAMP=962224100 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_67.314.962224111 RECID=45 STAMP=962224111 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_68.316.962224463 RECID=47 STAMP=962224462 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_69.317.962224863 RECID=49 STAMP=962224862 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_70.320.962225885 RECID=52 STAMP=962225884 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_71.322.962273773 RECID=54 STAMP=962273772 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_72.323.962273775 RECID=55 STAMP=962273775 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_73.324.962273779 RECID=56 STAMP=962273783 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_74.326.962273785 RECID=58 STAMP=962273785 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_47.281.962120099 RECID=12 STAMP=962073300 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_48.284.962121379 RECID=15 STAMP=962121380 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_49.285.962121381 RECID=16 STAMP=962121380 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_50.286.962121383 RECID=17 STAMP=962121382 部分省略 .................... deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_66.319.962225883 RECID=51 STAMP=962225883 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_67.321.962225885 RECID=53 STAMP=962225884 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_68.325.962273785 RECID=57 STAMP=962273784 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_69.327.962274133 RECID=59 STAMP=962274134 Deleted 48 EXPIRED objects

相关阅读