Oracle DG备库发现Oracle Home目录使用空间巨增的问题处理

同事巡检查发现Oracle RAC DG备库的Oracle Home目录使用空间巨增,执行命令发现oracle目录占用了87G,这肯定不正常。

[root@dgdb1 app]# du -sh *
11G     11.2.0
8.0K    asmpfile.ora
3.1G    grid
87G     oracle
4.0M    oraInventory

进一步检查发现ORACLE_HOME目录下的dbs目录占用了61G。

[root@dgdb1 db_1]# du -sh *
177M    apex
303M    assistants
459M    bin
58M     ccr
8.0K    cdata
2.9M    cfgtoollogs
68K     clone
4.0K    config
5.9M    crs
24K     csmig
236K    css
296M    ctx
3.3M    cv
61G     dbs
12K     dc_ocm
396K    deinstall
1.1M    demo
16K     diagnostics
6.9M    dv
40K     emcli
8.0K    EMStage
1.9M    has
40K     hs
11M     ide
7.4M    install
1.5M    instantclient
215M    inventory
60M     j2ee
118M    javavm
26M     jdbc
520K    jdev
186M    jdk
43M     jlib
2.7M    ldap
768M    lib
64K     log
33M     md
96K     mesg
908K    mgw
8.3M    network
33M     nls
326M    oc4j
1.7M    odbc
13M     olap
5.5M    OPatch
528K    opmn
40M     oracore
4.0K    oraInst.loc
7.4M    ord
29M     oui
545M    owb
1.7M    owm
60M     perl
1.6M    plsql
5.1M    precomp
316K    racg
127M    rdbms
124K    relnotes
4.0K    root.sh
12K     scheduler
20K     slax
82M     sqldeveloper
4.8M    sqlj
484K    sqlplus
9.6M    srvm
21M     suptools
220M    sysman
112K    timingframework
512K    ucp
3.7M    uix
800K    usm
8.0K    utl
556K    wwg
19M     xdk

检查dbs目录发现多了很多broken*这样的文件,并且在是三月2号8:36分到9:49分之间生成的

[root@dgdb1 dbs]# ls -lrt
total 63341528
-rw-r--r-- 1 oracle oinstall       2851 May 15  2009 init.ora
-rw-rw---- 1 oracle asmadmin       1544 Nov  9  2016 hc_rlzy1.dat
-rw-r----- 1 oracle oinstall       1536 Nov 11  2016 orapwRLZY1
-rw-r--r-- 1 oracle oinstall         60 Nov 20  2016 initRLZY1.ora
-rw-r--r-- 1 oracle oinstall         60 Nov 24  2016 initRLZY1.ora.bk
-rw-r--r-- 1 oracle oinstall       1922 Nov 24  2016 initCAIWU1.ora.bk
-rw-r--r-- 1 oracle oinstall         60 Nov 24  2016 initCAIWU1.ora
-rw-r----- 1 oracle oinstall       1536 Nov 24  2016 orapwCAIWU1
-rw-rw---- 1 oracle asmadmin       1544 Nov 24  2016 hc_chdyldg.dat
-rw-r--r-- 1 oracle oinstall       1979 Nov 24  2016 initchdyl1.ora.bk
-rw-r--r-- 1 oracle oinstall         61 Nov 24  2016 initchdyl1.ora
-rw-r----- 1 oracle oinstall       1536 Nov 24  2016 orapwchdyl1
-rw-r--r-- 1 oracle oinstall       2019 Nov 28  2016 initsjjh1.ora.bk
-rw-r--r-- 1 oracle oinstall         60 Nov 28  2016 initsjjh1.ora.bak.dgdb1
-rw-r----- 1 oracle oinstall       1536 Nov 28  2016 orapwsjjh1.bk
-rw-r--r-- 1 oracle oinstall         83 Nov 29  2016 initsjjh1.ora
-rw-rw---- 1 oracle asmadmin       1544 Apr  1  2017 hc_dbdb1.dat
-rw-r--r-- 1 oracle oinstall       1783 Apr  1  2017 initdadb1.ora.bk
-rw-r--r-- 1 oracle oinstall         53 Apr  1  2017 initdadb1.ora.bak.dgdb1
-rw-r----- 1 oracle oinstall       1536 Apr  1  2017 orapwdadb1
-rw-r--r-- 1 oracle oinstall         76 Apr  5  2017 initdadb1.ora
-rw-rw---- 1 oracle asmadmin       1544 Apr 22  2019 hc_chdyl1.dat
-rw-rw---- 1 oracle asmadmin       1544 Apr 22  2019 hc_dadb1.dat
-rw-rw---- 1 oracle asmadmin       1544 Apr 22  2019 hc_CAIWU1.dat
-rw-rw---- 1 oracle asmadmin       1544 Jul 29  2019 hc_RLZY1.dat
-rw-r----- 1 oracle oinstall       1536 Aug  6  2019 orapwsjjh
-rw-rw---- 1 oracle asmadmin       1544 Aug  6  2019 hc_sjjh.dat
-rw-r----- 1 oracle asmadmin   10797056 Oct 14  2019 snapcf_sjjh1.f
-rw-r----- 1 oracle oinstall       1536 Mar  1 18:31 orapwsjjh1
-rw-r----- 1 oracle asmadmin 1262486016 Mar  2 08:36 broken0
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:36 broken1
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:36 broken2
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:36 broken3
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:36 broken4
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:36 broken5
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:37 broken6
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:37 broken7
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:37 broken8
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:37 broken9
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:37 broken10
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:37 broken11
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:38 broken12
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:38 broken13
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:38 broken14
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:38 broken15
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:38 broken16
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:38 broken17
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:38 broken18
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:39 broken19
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:39 broken20
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:39 broken21
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:39 broken22
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:39 broken23
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:39 broken24
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:40 broken25
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:40 broken26
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:40 broken27
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:40 broken28
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:40 broken29
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:48 broken59
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:48 broken33
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:48 broken35
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:48 broken34
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:48 broken36
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:49 broken37
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:49 broken39
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:49 broken38
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:49 broken40
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:49 broken41
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:49 broken42
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:49 broken43
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:50 broken44
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:50 broken45
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:50 broken46
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:50 broken47
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:50 broken48
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:50 broken49
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:51 broken51
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:51 broken50
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:51 broken52
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:51 broken53
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:51 broken54
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:51 broken55
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:52 broken57
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:52 broken56
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 08:52 broken58
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 09:17 broken32
-rw-r----- 1 oracle asmadmin 1073742336 Mar  2 09:33 broken31
-rw-r----- 1 oracle asmadmin 1262486016 Mar  2 09:49 broken30
-rw-rw---- 1 oracle asmadmin       1544 Mar  2 10:02 hc_sjjh1.dat
-rw-r----- 1 oracle asmadmin   48316416 Jul 21 03:00 snapcf_RLZY1.f

查看文件broken0发现是sjjh数据库与日志文件相关的错误

查看实例的alert.log文件的内容,果然是在写日志文件时出现了错误:

Successful mount of redo thread 1, with mount id 4246975374
Physical Standby Database mounted.
Lost write protection disabled
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken0'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken0'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken1'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 2 of thread 1
ORA-00312: online log 2 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken1'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken2'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 3 of thread 1
ORA-00312: online log 3 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken2'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 4 of thread 1
ORA-00312: online log 4 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken3'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 4 of thread 1
ORA-00312: online log 4 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken3'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 5 of thread 1
ORA-00312: online log 5 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken4'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 5 of thread 1
ORA-00312: online log 5 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken4'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 6 of thread 1
ORA-00312: online log 6 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken5'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 6 of thread 1
ORA-00312: online log 6 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken5'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 7 of thread 1
ORA-00312: online log 7 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken6'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 7 of thread 1
ORA-00312: online log 7 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken6'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 8 of thread 1
ORA-00312: online log 8 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken7'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 8 of thread 1
ORA-00312: online log 8 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken7'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 9 of thread 1
ORA-00312: online log 9 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken8'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 9 of thread 1
ORA-00312: online log 9 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken8'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 10 of thread 1
ORA-00312: online log 10 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken9'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 10 of thread 1
ORA-00312: online log 10 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken9'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 11 of thread 1
ORA-00312: online log 11 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken10'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 11 of thread 1
ORA-00312: online log 11 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken10'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 12 of thread 1
ORA-00312: online log 12 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken11'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 12 of thread 1
ORA-00312: online log 12 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken11'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 13 of thread 1
ORA-00312: online log 13 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken12'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 13 of thread 1
ORA-00312: online log 13 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken12'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 14 of thread 1
ORA-00312: online log 14 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken13'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 14 of thread 1
ORA-00312: online log 14 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken13'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 15 of thread 1
ORA-00312: online log 15 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken14'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 15 of thread 1
ORA-00312: online log 15 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken14'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 16 of thread 1
ORA-00312: online log 16 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken15'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 16 of thread 1
ORA-00312: online log 16 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken15'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 17 of thread 1
ORA-00312: online log 17 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken16'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 17 of thread 1
ORA-00312: online log 17 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken16'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 18 of thread 1
ORA-00312: online log 18 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken17'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 18 of thread 1
ORA-00312: online log 18 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken17'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 19 of thread 1
ORA-00312: online log 19 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken18'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 19 of thread 1
ORA-00312: online log 19 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken18'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 20 of thread 1
ORA-00312: online log 20 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken19'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 20 of thread 1
ORA-00312: online log 20 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken19'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 21 of thread 1
ORA-00312: online log 21 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken20'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 21 of thread 1
ORA-00312: online log 21 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken20'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 22 of thread 1
ORA-00312: online log 22 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken21'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 22 of thread 1
ORA-00312: online log 22 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken21'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 23 of thread 1
ORA-00312: online log 23 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken22'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 23 of thread 1
ORA-00312: online log 23 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken22'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 24 of thread 1
ORA-00312: online log 24 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken23'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 24 of thread 1
ORA-00312: online log 24 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken23'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 25 of thread 1
ORA-00312: online log 25 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken24'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 25 of thread 1
ORA-00312: online log 25 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken24'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 26 of thread 1
ORA-00312: online log 26 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken25'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 26 of thread 1
ORA-00312: online log 26 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken25'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 27 of thread 1
ORA-00312: online log 27 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken26'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 27 of thread 1
ORA-00312: online log 27 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken26'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 28 of thread 1
ORA-00312: online log 28 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken27'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 28 of thread 1
ORA-00312: online log 28 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken27'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 29 of thread 1
ORA-00312: online log 29 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken28'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 29 of thread 1
ORA-00312: online log 29 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken28'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 30 of thread 1
ORA-00312: online log 30 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken29'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 30 of thread 1
ORA-00312: online log 30 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken29'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 31 of thread 1
ORA-00312: online log 31 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken30'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 31 of thread 1
ORA-00312: online log 31 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken30'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 32 of thread 1
ORA-00312: online log 32 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken31'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 32 of thread 1
ORA-00312: online log 32 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken31'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 33 of thread 1
ORA-00312: online log 33 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken32'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 33 of thread 1
ORA-00312: online log 33 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken32'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 34 of thread 1
ORA-00312: online log 34 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken33'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 34 of thread 1
ORA-00312: online log 34 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken33'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 35 of thread 1
ORA-00312: online log 35 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken34'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 35 of thread 1
ORA-00312: online log 35 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken34'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 36 of thread 1
ORA-00312: online log 36 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken35'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 36 of thread 1
ORA-00312: online log 36 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken35'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 37 of thread 1
ORA-00312: online log 37 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken36'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 37 of thread 1
ORA-00312: online log 37 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken36'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 38 of thread 1
ORA-00312: online log 38 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken37'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 38 of thread 1
ORA-00312: online log 38 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken37'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 39 of thread 1
ORA-00312: online log 39 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken38'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 39 of thread 1
ORA-00312: online log 39 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken38'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 40 of thread 1
ORA-00312: online log 40 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken39'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 40 of thread 1
ORA-00312: online log 40 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken39'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 41 of thread 1
ORA-00312: online log 41 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken40'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 41 of thread 1
ORA-00312: online log 41 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken40'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 42 of thread 1
ORA-00312: online log 42 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken41'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 42 of thread 1
ORA-00312: online log 42 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken41'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 43 of thread 1
ORA-00312: online log 43 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken42'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 43 of thread 1
ORA-00312: online log 43 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken42'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 44 of thread 1
ORA-00312: online log 44 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken43'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 44 of thread 1
ORA-00312: online log 44 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken43'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 45 of thread 1
ORA-00312: online log 45 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken44'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 45 of thread 1
ORA-00312: online log 45 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken44'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 46 of thread 1
ORA-00312: online log 46 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken45'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 46 of thread 1
ORA-00312: online log 46 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken45'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 47 of thread 1
ORA-00312: online log 47 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken46'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 47 of thread 1
ORA-00312: online log 47 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken46'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 48 of thread 1
ORA-00312: online log 48 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken47'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 48 of thread 1
ORA-00312: online log 48 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken47'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 49 of thread 1
ORA-00312: online log 49 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken48'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 49 of thread 1
ORA-00312: online log 49 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken48'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 50 of thread 1
ORA-00312: online log 50 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken49'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 50 of thread 1
ORA-00312: online log 50 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken49'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 51 of thread 1
ORA-00312: online log 51 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken50'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 51 of thread 1
ORA-00312: online log 51 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken50'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 52 of thread 1
ORA-00312: online log 52 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken51'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 52 of thread 1
ORA-00312: online log 52 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken51'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 53 of thread 1
ORA-00312: online log 53 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken52'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 53 of thread 1
ORA-00312: online log 53 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken52'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 54 of thread 1
ORA-00312: online log 54 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken53'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 54 of thread 1
ORA-00312: online log 54 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken53'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 55 of thread 1
ORA-00312: online log 55 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken54'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 55 of thread 1
ORA-00312: online log 55 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken54'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 56 of thread 1
ORA-00312: online log 56 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken55'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 56 of thread 1
ORA-00312: online log 56 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken55'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 57 of thread 1
ORA-00312: online log 57 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken56'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 57 of thread 1
ORA-00312: online log 57 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken56'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 58 of thread 1
ORA-00312: online log 58 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken57'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 58 of thread 1
ORA-00312: online log 58 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken57'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 59 of thread 1
ORA-00312: online log 59 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken58'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 59 of thread 1
ORA-00312: online log 59 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken58'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 60 of thread 1
ORA-00312: online log 60 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken59'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Errors in file /u01/app/oracle/diag/rdbms/sjjhdg/sjjh1/trace/sjjh1_lgwr_54343.trc:
ORA-00313: open failed for members of log group 60 of thread 1
ORA-00312: online log 60 thread 1: '/u01/app/oracle/product/11.2.0/db_1/dbs/broken59'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3
Completed: ALTER DATABASE MOUNT
ARC3: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
Mon Mar 02 08:35:50 2020
alter database open
AUDIT_TRAIL initialization parameter is changed to OS, as DB is NOT compatible for database opened with read-only access
查看主库的alert.log文件发现主库不能将日志同步到备库,原因是因为主备库之间网络故障
Mon Mar 02 01:17:22 2020
Thread 1 cannot allocate new log, sequence 3357
Private strand flush not complete
Current log# 11 seq# 3356 mem# 0: /oradata/sjjh/redo11.log
LGWR: Standby redo logfile selected for thread 1 sequence 3357 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3357 (LGWR switch)
Current log# 12 seq# 3357 mem# 0: /oradata/sjjh/redo12.log
Mon Mar 02 01:17:29 2020
Archived Log entry 6342 added for thread 1 sequence 3356 ID 0xfc687662 dest 1:
Mon Mar 02 02:00:00 2020
Closing scheduler window
Closing Resource Manager plan via scheduler window
Clearing Resource Manager plan via parameter
Mon Mar 02 02:33:30 2020
Thread 1 cannot allocate new log, sequence 3358
Private strand flush not complete
Current log# 12 seq# 3357 mem# 0: /oradata/sjjh/redo12.log
LGWR: Standby redo logfile selected for thread 1 sequence 3358 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3358 (LGWR switch)
Current log# 13 seq# 3358 mem# 0: /oradata/sjjh/redo13.log
Mon Mar 02 02:33:38 2020
Archived Log entry 6344 added for thread 1 sequence 3357 ID 0xfc687662 dest 1:
Mon Mar 02 03:45:12 2020
Thread 1 cannot allocate new log, sequence 3359
Private strand flush not complete
Current log# 13 seq# 3358 mem# 0: /oradata/sjjh/redo13.log
LGWR: Standby redo logfile selected for thread 1 sequence 3359 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3359 (LGWR switch)
Current log# 14 seq# 3359 mem# 0: /oradata/sjjh/redo14.log
Mon Mar 02 03:45:19 2020
Archived Log entry 6346 added for thread 1 sequence 3358 ID 0xfc687662 dest 1:
Mon Mar 02 08:24:54 2020
LGWR: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (3113)
LGWR: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Error 3113 for archive log file 14 to 'jhk_st'
LGWR: Error 1041 disconnecting from destination LOG_ARCHIVE_DEST_2 standby host 'jhk_st'
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
Thread 1 cannot allocate new log, sequence 3360
Private strand flush not complete
Current log# 14 seq# 3359 mem# 0: /oradata/sjjh/redo14.log
LGWR: Failed to archive log 14 thread 1 sequence 3359 (3113)
Thread 1 advanced to log sequence 3360 (LGWR switch)
Current log# 15 seq# 3360 mem# 0: /oradata/sjjh/redo15.log
Mon Mar 02 08:25:01 2020
Archived Log entry 6347 added for thread 1 sequence 3359 ID 0xfc687662 dest 1:
Mon Mar 02 08:30:31 2020
Thread 1 cannot allocate new log, sequence 3361
Private strand flush not complete
Current log# 15 seq# 3360 mem# 0: /oradata/sjjh/redo15.log
Mon Mar 02 08:30:52 2020
LGWR: Standby redo logfile selected for thread 1 sequence 3361 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3361 (LGWR switch)
Current log# 16 seq# 3361 mem# 0: /oradata/sjjh/redo16.log
Mon Mar 02 08:30:52 2020
Archived Log entry 6349 added for thread 1 sequence 3360 ID 0xfc687662 dest 1:
Mon Mar 02 08:30:52 2020
ARC0: Standby redo logfile selected for thread 1 sequence 3360 for destination LOG_ARCHIVE_DEST_2
Mon Mar 02 08:54:23 2020
LGWR: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (272)
LGWR: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Error 272 for archive log file 16 to 'jhk_st'
Mon Mar 02 08:55:04 2020
Thread 1 cannot allocate new log, sequence 3362
Private strand flush not complete
Current log# 16 seq# 3361 mem# 0: /oradata/sjjh/redo16.log
LGWR: Failed to archive log 16 thread 1 sequence 3361 (272)
Thread 1 advanced to log sequence 3362 (LGWR switch)
Current log# 17 seq# 3362 mem# 0: /oradata/sjjh/redo17.log
Mon Mar 02 08:55:07 2020
Archived Log entry 6351 added for thread 1 sequence 3361 ID 0xfc687662 dest 1:
Mon Mar 02 09:00:04 2020
ARC3: Standby redo logfile selected for thread 1 sequence 3361 for destination LOG_ARCHIVE_DEST_2
Mon Mar 02 09:00:04 2020
Thread 1 cannot allocate new log, sequence 3363
Private strand flush not complete
Current log# 17 seq# 3362 mem# 0: /oradata/sjjh/redo17.log
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED
LGWR: Standby redo logfile selected for thread 1 sequence 3363 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3363 (LGWR switch)
Current log# 18 seq# 3363 mem# 0: /oradata/sjjh/redo18.log
Mon Mar 02 09:00:08 2020
Archived Log entry 6353 added for thread 1 sequence 3362 ID 0xfc687662 dest 1:
ARC3: Standby redo logfile selected for thread 1 sequence 3362 for destination LOG_ARCHIVE_DEST_2
Mon Mar 02 09:13:31 2020
LGWR: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (3113)
LGWR: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Error 3113 for archive log file 18 to 'jhk_st'
LGWR: Error 1041 disconnecting from destination LOG_ARCHIVE_DEST_2 standby host 'jhk_st'
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
Thread 1 cannot allocate new log, sequence 3364
Private strand flush not complete
Current log# 18 seq# 3363 mem# 0: /oradata/sjjh/redo18.log
LGWR: Failed to archive log 18 thread 1 sequence 3363 (3113)
Thread 1 advanced to log sequence 3364 (LGWR switch)
Current log# 19 seq# 3364 mem# 0: /oradata/sjjh/redo19.log
Mon Mar 02 09:13:36 2020
Archived Log entry 6355 added for thread 1 sequence 3363 ID 0xfc687662 dest 1:
Mon Mar 02 09:14:42 2020
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY SID='*';
Mon Mar 02 09:14:43 2020
ARC3: Standby redo logfile selected for thread 1 sequence 3363 for destination LOG_ARCHIVE_DEST_2
Mon Mar 02 09:14:45 2020
Thread 1 cannot allocate new log, sequence 3365
Private strand flush not complete
Current log# 19 seq# 3364 mem# 0: /oradata/sjjh/redo19.log
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED
LGWR: Standby redo logfile selected for thread 1 sequence 3365 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3365 (LGWR switch)
Current log# 20 seq# 3365 mem# 0: /oradata/sjjh/redo20.log
Mon Mar 02 09:14:48 2020
Archived Log entry 6357 added for thread 1 sequence 3364 ID 0xfc687662 dest 1:
Mon Mar 02 09:16:14 2020
LGWR: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (272)
LGWR: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Error 272 for archive log file 20 to 'jhk_st'
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
Thread 1 cannot allocate new log, sequence 3366
Private strand flush not complete
Current log# 20 seq# 3365 mem# 0: /oradata/sjjh/redo20.log
LGWR: Failed to archive log 20 thread 1 sequence 3365 (272)
Thread 1 advanced to log sequence 3366 (LGWR switch)
Current log# 21 seq# 3366 mem# 0: /oradata/sjjh/redo21.log
Mon Mar 02 09:16:19 2020
Archived Log entry 6359 added for thread 1 sequence 3365 ID 0xfc687662 dest 1:
Mon Mar 02 09:22:07 2020
Thread 1 cannot allocate new log, sequence 3367
Private strand flush not complete
Current log# 21 seq# 3366 mem# 0: /oradata/sjjh/redo21.log
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED
LGWR: Standby redo logfile selected for thread 1 sequence 3367 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3367 (LGWR switch)
Current log# 22 seq# 3367 mem# 0: /oradata/sjjh/redo22.log
Mon Mar 02 09:22:10 2020
Archived Log entry 6361 added for thread 1 sequence 3366 ID 0xfc687662 dest 1:
Mon Mar 02 09:29:22 2020
LGWR: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (3113)
LGWR: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned
Error 3113 for archive log file 22 to 'jhk_st'
LGWR: Error 1041 disconnecting from destination LOG_ARCHIVE_DEST_2 standby host 'jhk_st'
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
Thread 1 cannot allocate new log, sequence 3368
Private strand flush not complete
Current log# 22 seq# 3367 mem# 0: /oradata/sjjh/redo22.log
LGWR: Failed to archive log 22 thread 1 sequence 3367 (3113)
Thread 1 advanced to log sequence 3368 (LGWR switch)
Current log# 23 seq# 3368 mem# 0: /oradata/sjjh/redo23.log
Mon Mar 02 09:29:26 2020
Archived Log entry 6363 added for thread 1 sequence 3367 ID 0xfc687662 dest 1:
Mon Mar 02 09:30:04 2020
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY SID='*';
Mon Mar 02 09:30:05 2020
ARC3: Standby redo logfile selected for thread 1 sequence 3367 for destination LOG_ARCHIVE_DEST_2
Mon Mar 02 09:30:05 2020
Thread 1 cannot allocate new log, sequence 3369
Private strand flush not complete
Current log# 23 seq# 3368 mem# 0: /oradata/sjjh/redo23.log
Suppressing further error logging of LOG_ARCHIVE_DEST_2.
FAL[server, ARC3]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance sjjh - Archival Error. Archiver continuing.
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED
LGWR: Standby redo logfile selected for thread 1 sequence 3369 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3369 (LGWR switch)
Current log# 24 seq# 3369 mem# 0: /oradata/sjjh/redo24.log
Mon Mar 02 09:30:08 2020
Archived Log entry 6365 added for thread 1 sequence 3368 ID 0xfc687662 dest 1:
ARC3: Standby redo logfile selected for thread 1 sequence 3368 for destination LOG_ARCHIVE_DEST_2
Mon Mar 02 09:31:27 2020
Suppressing further error logging of LOG_ARCHIVE_DEST_2.
Suppressing further error logging of LOG_ARCHIVE_DEST_2.
Suppressing further error logging of LOG_ARCHIVE_DEST_2.
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
Thread 1 cannot allocate new log, sequence 3370
Private strand flush not complete
Current log# 24 seq# 3369 mem# 0: /oradata/sjjh/redo24.log
LGWR: Failed to archive log 24 thread 1 sequence 3369 (272)
Thread 1 advanced to log sequence 3370 (LGWR switch)
Current log# 25 seq# 3370 mem# 0: /oradata/sjjh/redo25.log
Mon Mar 02 09:31:33 2020
Archived Log entry 6367 added for thread 1 sequence 3369 ID 0xfc687662 dest 1:
Mon Mar 02 09:37:09 2020
Thread 1 cannot allocate new log, sequence 3371
Private strand flush not complete
Current log# 25 seq# 3370 mem# 0: /oradata/sjjh/redo25.log
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED
LGWR: Standby redo logfile selected for thread 1 sequence 3371 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3371 (LGWR switch)
Current log# 26 seq# 3371 mem# 0: /oradata/sjjh/redo26.log
Mon Mar 02 09:37:12 2020
Archived Log entry 6369 added for thread 1 sequence 3370 ID 0xfc687662 dest 1:
Mon Mar 02 09:37:12 2020
ARC3: Standby redo logfile selected for thread 1 sequence 3370 for destination LOG_ARCHIVE_DEST_2
Mon Mar 02 09:42:26 2020
Destination LOG_ARCHIVE_DEST_2 is UNSYNCHRONIZED
Thread 1 cannot allocate new log, sequence 3372
Private strand flush not complete
Current log# 26 seq# 3371 mem# 0: /oradata/sjjh/redo26.log
LGWR: Failed to archive log 26 thread 1 sequence 3371 (3113)
Thread 1 advanced to log sequence 3372 (LGWR switch)
Current log# 27 seq# 3372 mem# 0: /oradata/sjjh/redo27.log
Mon Mar 02 09:42:31 2020
Archived Log entry 6371 added for thread 1 sequence 3371 ID 0xfc687662 dest 1:
Mon Mar 02 09:44:23 2020
ALTER SYSTEM SET log_archive_dest_state_2='ENABLE' SCOPE=MEMORY SID='*';
Mon Mar 02 09:44:23 2020
ARC0: Standby redo logfile selected for thread 1 sequence 3371 for destination LOG_ARCHIVE_DEST_2
Mon Mar 02 09:44:25 2020
Thread 1 cannot allocate new log, sequence 3373
Private strand flush not complete
Current log# 27 seq# 3372 mem# 0: /oradata/sjjh/redo27.log
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED
LGWR: Standby redo logfile selected for thread 1 sequence 3373 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3373 (LGWR switch)
Current log# 28 seq# 3373 mem# 0: /oradata/sjjh/redo28.log
Mon Mar 02 09:44:28 2020
Archived Log entry 6373 added for thread 1 sequence 3372 ID 0xfc687662 dest 1:
Mon Mar 02 09:44:28 2020
ARC3: Standby redo logfile selected for thread 1 sequence 3372 for destination LOG_ARCHIVE_DEST_2
Mon Mar 02 09:45:34 2020
Thread 1 cannot allocate new log, sequence 3374
Private strand flush not complete
Current log# 28 seq# 3373 mem# 0: /oradata/sjjh/redo28.log
LGWR: Standby redo logfile selected for thread 1 sequence 3374 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3374 (LGWR switch)
Current log# 29 seq# 3374 mem# 0: /oradata/sjjh/redo29.log
Mon Mar 02 09:45:35 2020
Archived Log entry 6376 added for thread 1 sequence 3373 ID 0xfc687662 dest 1:
Mon Mar 02 13:45:51 2020
LGWR: Standby redo logfile selected for thread 1 sequence 3375 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3375 (LGWR switch)
Current log# 30 seq# 3375 mem# 0: /oradata/sjjh/redo30.log
Mon Mar 02 13:45:55 2020
Archived Log entry 6378 added for thread 1 sequence 3374 ID 0xfc687662 dest 1:
Mon Mar 02 13:47:12 2020
Thread 1 cannot allocate new log, sequence 3376
Private strand flush not complete
Current log# 30 seq# 3375 mem# 0: /oradata/sjjh/redo30.log
Destination LOG_ARCHIVE_DEST_2 no longer supports SYNCHRONIZATION
Thread 1 advanced to log sequence 3376 (LGWR switch)
Current log# 1 seq# 3376 mem# 0: /oradata/sjjh/redo01.log
Mon Mar 02 13:47:19 2020
Archived Log entry 6380 added for thread 1 sequence 3375 ID 0xfc687662 dest 1:
Mon Mar 02 22:00:00 2020
Setting Resource Manager plan SCHEDULER[0x32D9]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Mon Mar 02 22:00:00 2020
Starting background process VKRM
Mon Mar 02 22:00:00 2020
VKRM started with pid=47, OS id=81983
Mon Mar 02 22:00:02 2020
Begin automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"
Mon Mar 02 22:01:00 2020
End automatic SQL Tuning Advisor run for special tuning task "SYS_AUTO_SQL_TUNING_TASK"
Tue Mar 03 00:38:04 2020
Thread 1 cannot allocate new log, sequence 3377
Private strand flush not complete
Current log# 1 seq# 3376 mem# 0: /oradata/sjjh/redo01.log
Destination LOG_ARCHIVE_DEST_2 is SYNCHRONIZED
LGWR: Standby redo logfile selected for thread 1 sequence 3377 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3377 (LGWR switch)
Current log# 2 seq# 3377 mem# 0: /oradata/sjjh/redo02.log
Tue Mar 03 00:38:12 2020
Archived Log entry 6382 added for thread 1 sequence 3376 ID 0xfc687662 dest 1:
Tue Mar 03 00:40:10 2020
LGWR: Standby redo logfile selected for thread 1 sequence 3378 for destination LOG_ARCHIVE_DEST_2
Thread 1 advanced to log sequence 3378 (LGWR switch)
Current log# 3 seq# 3378 mem# 0: /oradata/sjjh/redo03.log

明白原因之后现在可以删除dbs目录中的broker*文件来释放空间

私有IP丢失造成Oracle 12C RAC集群节点不能启动

某生产环境Oracle Linux 7.1,Oracle 12C RAC显示节点1的集群资源没有启动,信息如下:

[grid@cs2 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       cs2                      STABLE
ora.CRS.dg
               ONLINE  ONLINE       cs2                      STABLE
ora.DATA.dg
               ONLINE  ONLINE       cs2                      STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       cs2                      STABLE
ora.chad
               ONLINE  OFFLINE      cs2                      STABLE
ora.net1.network
               ONLINE  ONLINE       cs2                      STABLE
ora.ons
               ONLINE  OFFLINE      cs2                      STABLE
ora.proxy_advm
               OFFLINE OFFLINE      cs2                      STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       cs2                      STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       cs2                      STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       cs2                      STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       cs2                      169.254.255.92 88.88
                                                             .88.191,STABLE
ora.asm
      1        ONLINE  OFFLINE                               STABLE
      2        ONLINE  ONLINE       cs2                      Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cs.db
      1        ONLINE  OFFLINE                               STABLE
      2        ONLINE  OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.cs1.vip
      1        ONLINE  INTERMEDIATE cs2                      FAILED OVER,STABLE
ora.cs2.vip
      1        ONLINE  ONLINE       cs2                      STABLE
ora.cvu
      1        ONLINE  ONLINE       cs2                      STABLE
ora.gns
      1        ONLINE  OFFLINE                               STABLE
ora.gns.vip
      1        ONLINE  OFFLINE                               STABLE
ora.mgmtdb
      1        ONLINE  OFFLINE                               Instance Shutdown,ST
                                                             ABLE
ora.qosmserver
      1        ONLINE  ONLINE       cs2                      STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       cs2                      STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       cs2                      STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       cs2                      STABLE
--------------------------------------------------------------------------------

如果查看节点1上的alert.log文件,信息如下:

[root@cs1 soft]# tail -100 /u01/app/grid/diag/crs/cs1/crs/trace/alert.log

2020-01-14 20:45:50.226 [CVUD(2366CRS-10051: CVU found following errors with Clusterware setup : PRVE-3191 :
(DESCRIPTION = (LOAD_BALANCE=on)  (ADDRESS = (PROTOCOL = TCP)(HOST = cs-cluster-scan.cs-cluster.jy.net)(PORT = 1521)) (CONNECT_DATA =(SERVER = DEDICATED)(SERVICE_NAME = cs)))
Listener refused the connection with the following error:
ORA-12514, TNS:listener does not currently know of service requested in connect descriptor
 

2020-02-17 14:49:28.563 [ORAROOTAGENT(15500)]CRS-5822: Agent '/u01/app/product/12.2.0/crs/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:5:63832} in /u01/app/grid/diag/crs/cs1/crs/trace/crsd_orarootagent_root.trc.
2020-02-17 14:49:28.584 [ORAAGENT(15466)]CRS-5822: Agent '/u01/app/product/12.2.0/crs/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:1:36} in /u01/app/grid/diag/crs/cs1/crs/trace/crsd_oraagent_grid.trc.
2020-02-17 14:49:29.607 [GPNPD(3293)]CRS-2329: GPNPD on node cs1 shut down. 
2020-02-17 14:49:30.123 [OSYSMOND(6077)]CRS-8504: Oracle Clusterware OSYSMOND process with operating system process ID 6077 is exiting
2020-02-17 14:49:31.377 [MDNSD(3254)]CRS-5602: mDNS service stopping by request.
2020-02-17 14:49:31.747 [MDNSD(3254)]CRS-8504: Oracle Clusterware MDNSD process with operating system process ID 3254 is exiting
2020-02-17 14:49:46.650 [OCTSSD(5883)]CRS-2405: The Cluster Time Synchronization Service on host cs1 is shutdown by user
2020-02-17 14:49:46.651 [OCTSSD(5883)]CRS-8504: Oracle Clusterware OCTSSD process with operating system process ID 5883 is exiting
2020-02-17 14:49:47.651 [OCSSD(3616)]CRS-1603: CSSD on node cs1 has been shut down.
2020-02-17 14:49:47.958 [OCSSD(3616)]CRS-1660: The CSS daemon shutdown has completed
2020-02-17 14:49:47.959 [OCSSD(3616)]CRS-8504: Oracle Clusterware OCSSD process with operating system process ID 3616 is exiting
2020-02-17 14:49:49.726 [ORAAGENT(3235)]CRS-5822: Agent '/u01/app/product/12.2.0/crs/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:9:818} in /u01/app/grid/diag/crs/cs1/crs/trace/ohasd_oraagent_grid.trc.
2020-02-17 14:49:49.727 [ORAROOTAGENT(3029)]CRS-5822: Agent '/u01/app/product/12.2.0/crs/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:1:21} in /u01/app/grid/diag/crs/cs1/crs/trace/ohasd_orarootagent_root.trc.

根据上面的错误信息提示查看crsd_oraagent_grid.trc跟踪文件:

[root@cs1 ~]# more /u01/app/grid/diag/crs/cs1/crs/trace/crsd_oraagent_grid.trc
Trace file /u01/app/grid/diag/crs/cs1/crs/trace/crsd_oraagent_grid.trc
Oracle Database 12c Clusterware Release 12.2.0.1.0 - Production Copyright 1996, 2016 Oracle. All rights reserved.

*** TRACE CONTINUED FROM FILE /u01/app/grid/diag/crs/cs1/crs/trace/crsd_oraagent_grid_910.trc ***

2020-02-17 14:39:22.129 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] LsnrAgent::regEndpoint 110 reset regEndPointDone:0
2020-02-17 14:39:22.129 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] generateEndPointStrings:generate endpoints
2020-02-17 14:39:22.129 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] VendorType=0
2020-02-17 14:39:22.129 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] VendorType=0
2020-02-17 14:39:22.129 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] VendorType=0
2020-02-17 14:39:22.129 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] VendorType=0
2020-02-17 14:39:22.129 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] generateEndPointStrings:No IP matching the subnet 88.88.88.0
2020-02-17 14:39:22.130 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] Ocr Context init default level 1814629536
2020-02-17 14:39:22.133 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] Agent::getDeploymentPlatformId return 
2020-02-17 14:39:22.133 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] LsnrAgent::generateEndPoints 000 entry {nonPriv
2020-02-17 14:39:22.133 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] LsnrAgent::generateEndPoints 000 entry { lsnrResName: endpAttr:TCP:1526 type:4
2020-02-17 14:39:22.133 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] LsnrAgent::generateEndPoints 040 Listener ResName:
2020-02-17 14:39:22.133 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] LsnrAgent::removeDuplicates 
2020-02-17 14:39:22.133 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] LsnrAgent::generateEndPoints 999 exit nonPriv}
2020-02-17 14:39:22.133 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] LsnrAgent::regEndpoint 200 endpStrings empty
2020-02-17 14:39:22.134 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] AsmLsnrAgent::init
2020-02-17 14:39:22.154 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] VendorType=0
2020-02-17 14:39:22.154 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] VendorType=0
2020-02-17 14:39:22.154 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] VendorType=0
2020-02-17 14:39:22.154 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] VendorType=0
2020-02-17 14:39:22.154 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] Cannot find an IP address matching the subnet 88.88.88.0
2020-02-17 14:39:22.154 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] Ocr Context init default level 1814629536
2020-02-17 14:39:22.157 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] Agent::getDeploymentPlatformId return 
2020-02-17 14:39:22.158 : CLSCEVT:2462013184: (:CLSCE0028:)clsce_unsubscribe 0x7f725c004510 successfully unsubscribed : 974833
2020-02-17 14:39:22.158 : USRTHRD:2462013184: {0:1:18} WorkerThread::removeWorker total workers: 2
2020-02-17 14:39:22.158 : USRTHRD:2462013184: {0:1:18} Worker::schedule adding worker 'ReconnSub-LsnrAgentSub-ASMNET1LSNR_ASM' to thread 'ReconnectingSubscribers'
2020-02-17 14:39:22.158 : USRTHRD:2462013184: {0:1:18} WorkerThread::addWorker total workers: 3
2020-02-17 14:39:22.158 :CLSDYNAM:2462013184: [ora.ASMNET1LSNR_ASM.lsnr]{0:1:18} [check] Skipping CSS Initialization for rebootless recovery
2020-02-17 14:39:22.159 : CLSCEVT:2478823168: clsce_subscribe 0x7f72600d06b0 filter='(^CRS_RESOURCE_PROFILE_CHANGE.*RESOURCE_CLASS='(scan_vip|vip)')|(^CRS_RESOURCE_PROFILE_CHANGE.*TYPE='ora\.network\.type')', flags=1, handler=0xc6ad0c, 
arg=0x7f726016de10

从上面的错误信息中可以找到Cannot find an IP address matching the subnet 88.88.88.0,这说明私有IP丢失了。

如是查看网络IP地址
节点2私有IP正常

[grid@cs2 ~]$ ifconfig -a
ens160: flags=4163  mtu 1500
        inet 10.13.13.191  netmask 255.255.255.0  broadcast 10.13.13.255
        inet6 fe80::250:56ff:fea0:92af  prefixlen 64  scopeid 0x20
        ether 00:50:56:a0:92:af  txqueuelen 1000  (Ethernet)
        RX packets 127560233  bytes 10186683662 (9.4 GiB)
        RX errors 0  dropped 353  overruns 0  frame 0
        TX packets 18954431  bytes 1554204413 (1.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens160:1: flags=4163  mtu 1500
        inet 10.13.13.130  netmask 255.255.255.0  broadcast 10.13.13.255
        ether 00:50:56:a0:92:af  txqueuelen 1000  (Ethernet)

ens160:2: flags=4163  mtu 1500
        inet 10.13.13.147  netmask 255.255.255.0  broadcast 10.13.13.255
        ether 00:50:56:a0:92:af  txqueuelen 1000  (Ethernet)

ens160:3: flags=4163  mtu 1500
        inet 10.13.13.141  netmask 255.255.255.0  broadcast 10.13.13.255
        ether 00:50:56:a0:92:af  txqueuelen 1000  (Ethernet)

ens160:4: flags=4163  mtu 1500
        inet 10.13.13.138  netmask 255.255.255.0  broadcast 10.13.13.255
        ether 00:50:56:a0:92:af  txqueuelen 1000  (Ethernet)

ens160:5: flags=4163  mtu 1500
        inet 10.13.13.137  netmask 255.255.255.0  broadcast 10.13.13.255
        ether 00:50:56:a0:92:af  txqueuelen 1000  (Ethernet)

ens160:6: flags=4163  mtu 1500
        inet 10.13.13.136  netmask 255.255.255.0  broadcast 10.13.13.255
        ether 00:50:56:a0:92:af  txqueuelen 1000  (Ethernet)

ens192: flags=4163  mtu 1500
        inet 88.88.88.191  netmask 255.255.255.0  broadcast 88.88.88.255
        inet6 fe80::250:56ff:fea0:a05c  prefixlen 64  scopeid 0x20
        ether 00:50:56:a0:a0:5c  txqueuelen 1000  (Ethernet)
        RX packets 294716840  bytes 215309349272 (200.5 GiB)
        RX errors 0  dropped 355  overruns 0  frame 0
        TX packets 63703465  bytes 363393728693 (338.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens192:1: flags=4163  mtu 1500
        inet 169.254.255.92  netmask 255.255.0.0  broadcast 169.254.255.255
        ether 00:50:56:a0:a0:5c  txqueuelen 1000  (Ethernet)

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 90029207  bytes 223833448058 (208.4 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 90029207  bytes 223833448058 (208.4 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:d0:6e:0b  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0-nic: flags=4098  mtu 1500
        ether 52:54:00:d0:6e:0b  txqueuelen 500  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


[grid@cs2 ~]$ cat /etc/hosts

127.0.0.1      localhost
10.13.13.190  cs1
10.13.13.191  cs2

#10.13.13.192 cs1-vip.jy.net cs1-vip
#10.13.13.193 cs2-vip.jy.net cs2-vip

88.88.88.190   cs1-priv.jy.net cs1-priv
88.88.88.191   cs2-priv.jy.net cs2-priv


10.13.13.141 cs-cluster-scan 
10.13.13.142 cs-cluster-scan 
10.13.13.143 cs-cluster-scan 

节点1的私有IP丢失

[root@cs1 ~]# ifconfig -a
ens160: flags=4163  mtu 1500
        inet 10.13.13.190  netmask 255.255.255.0  broadcast 10.13.13.255
        inet6 fe80::250:56ff:fea0:4e69  prefixlen 64  scopeid 0x20
        inet6 2018::2  prefixlen 64  scopeid 0x0
        ether 00:50:56:a0:4e:69  txqueuelen 1000  (Ethernet)
        RX packets 53266410  bytes 3937179306 (3.6 GiB)
        RX errors 0  dropped 143  overruns 0  frame 0
        TX packets 2988621  bytes 284679938 (271.4 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens192: flags=4163  mtu 1500
        inet6 fe80::250:56ff:fea0:17dc  prefixlen 64  scopeid 0x20
        ether 00:50:56:a0:17:dc  txqueuelen 1000  (Ethernet)
        RX packets 70361859  bytes 5602168856 (5.2 GiB)
        RX errors 0  dropped 141  overruns 0  frame 0
        TX packets 24902418  bytes 195063342601 (181.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 30235303  bytes 10958756943 (10.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 30235303  bytes 10958756943 (10.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:d0:6e:0b  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0-nic: flags=4098  mtu 1500
        ether 52:54:00:d0:6e:0b  txqueuelen 500  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0



设置私有IP

[root@cs1 network-scripts]# cat ifcfg-Ethernet_connection_ens192
TYPE=Ethernet
BOOTPROTO=none
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
NAME=ens192
UUID=8ec45d90-c1fb-4924-84a0-72f58fc9b82f
DEVICE=ens192
ONBOOT=yes
IPADDR=88.88.88.190
PREFIX=24
NETMASK=255.255.255.0

[root@cs1 network-scripts]# ifconfig -a
ens160: flags=4163  mtu 1500
        inet 10.138.130.190  netmask 255.255.255.0  broadcast 10.138.130.255
        inet6 fe80::250:56ff:fea0:4e69  prefixlen 64  scopeid 0x20
        inet6 2018::2  prefixlen 64  scopeid 0x0
        ether 00:50:56:a0:4e:69  txqueuelen 1000  (Ethernet)
        RX packets 53277081  bytes 3937945677 (3.6 GiB)
        RX errors 0  dropped 143  overruns 0  frame 0
        TX packets 2989099  bytes 284806540 (271.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens192: flags=4163  mtu 1500
        inet 88.88.88.190  netmask 255.255.255.0  broadcast 88.88.88.255
        inet6 fe80::250:56ff:fea0:17dc  prefixlen 64  scopeid 0x20
        ether 00:50:56:a0:17:dc  txqueuelen 1000  (Ethernet)
        RX packets 70371906  bytes 5602881459 (5.2 GiB)
        RX errors 0  dropped 141  overruns 0  frame 0
        TX packets 24902428  bytes 195063343317 (181.6 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 30236949  bytes 10959017396 (10.2 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 30236949  bytes 10959017396 (10.2 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0: flags=4099  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:d0:6e:0b  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

virbr0-nic: flags=4098  mtu 1500
        ether 52:54:00:d0:6e:0b  txqueuelen 500  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@cs1 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘cs2’
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘cs2’
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘cs2’
CRS-2673: Attempting to stop ‘ora.crf’ on ‘cs2’
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘cs2’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘cs2’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘cs2’
CRS-2673: Attempting to stop ‘ora.drivers.acfs’ on ‘cs2’
CRS-2677: Stop of ‘ora.drivers.acfs’ on ‘cs2’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘cs2’ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘cs2’ succeeded
CRS-2677: Stop of ‘ora.gpnpd’ on ‘cs2’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘cs2’ succeeded
CRS-2677: Stop of ‘ora.mdnsd’ on ‘cs2’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘cs2’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘cs2’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘cs2’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘cs2’
CRS-2677: Stop of ‘ora.cssd’ on ‘cs2’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘cs2’
CRS-2673: Attempting to stop ‘ora.driver.afd’ on ‘cs2’
CRS-2677: Stop of ‘ora.driver.afd’ on ‘cs2’ succeeded
CRS-2677: Stop of ‘ora.gipcd’ on ‘cs2’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘cs2’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@cs1 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

手动启动节点1的集群软件

[root@cs1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       cs1                      STABLE
               ONLINE  ONLINE       cs2                      STABLE
ora.CRS.dg
               ONLINE  ONLINE       cs1                      STABLE
               ONLINE  ONLINE       cs2                      STABLE
ora.DATA.dg
               ONLINE  ONLINE       cs1                      STABLE
               ONLINE  ONLINE       cs2                      STABLE
ora.DN.dg
               OFFLINE OFFLINE      cs1                      STABLE
               OFFLINE OFFLINE      cs2                      STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       cs1                      STABLE
               ONLINE  ONLINE       cs2                      STABLE
ora.chad
               ONLINE  ONLINE       cs1                      STABLE
               ONLINE  ONLINE       cs2                      STABLE
ora.net1.network
               ONLINE  ONLINE       cs1                      STABLE
               ONLINE  ONLINE       cs2                      STABLE
ora.ons
               ONLINE  ONLINE       cs1                      STABLE
               ONLINE  ONLINE       cs2                      STABLE
ora.proxy_advm
               OFFLINE OFFLINE      cs1                      STABLE
               OFFLINE OFFLINE      cs2                      STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       cs1                      STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       cs2                      STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       cs1                      STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       cs1                      169.254.207.24 88.88
                                                             .88.190,STABLE
ora.asm
      1        ONLINE  ONLINE       cs1                      Started,STABLE
      2        ONLINE  ONLINE       cs2                      Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cs.db
      1        ONLINE  ONLINE       cs1                      Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db,STABLE
      2        ONLINE  ONLINE       cs2                      Open,HOME=/u01/app/o
                                                             racle/product/12.2.0
                                                             /db,STABLE
ora.cs1.vip
      1        ONLINE  ONLINE       cs1                      STABLE
ora.cs2.vip
      1        ONLINE  ONLINE       cs2                      STABLE
ora.cvu
      1        ONLINE  ONLINE       cs1                      STABLE
ora.gns
      1        ONLINE  ONLINE       cs2                      STABLE
ora.gns.vip
      1        ONLINE  ONLINE       cs2                      STABLE
ora.mgmtdb
      1        ONLINE  OFFLINE      cs1                      Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       cs1                      STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       cs1                      STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       cs2                      STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       cs1                      STABLE
--------------------------------------------------------------------------------

到此恢复正常状态。

PLSQL通过Oracle 11g客户端连接Oracle 12c服务器错误 ORA-28040

PLSQL通过Oracle 11g客户端连接Oracle 12c服务器错误 ORA-28040
环境描述:
oracle服务器端版本:oracle 12.2.0.1.0
oracle客户端版本:oracle 11.2.0.1.0
PLSQL是11.4

因为PLSQL连接数据库也是要通过Oracle客户端,那么使用11g客户端访问oracle 12c应该也会得到如下错误:

C:\Users\Administrator>sqlplus sys/abcd@shardcat as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Sat Dec 20 22:22:07 2014

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

ERROR:
ORA-28040: No matching authentication protocol

查看关于错误的详细描述:

[oracle@shard1 admin]$ oerr ora 28040
28040, 0000, "No matching authentication protocol"
// *Cause:  There was no acceptable authentication protocol for
//          either client or server.
// *Action: The administrator should set the values of the
//          SQLNET.ALLOWED_LOGON_VERSION_SERVER and
//          SQLNET.ALLOWED_LOGON_VERSION_CLIENT parameters, on both the
//          client and on the server, to values that match the minimum
//          version software supported in the system.
//          This error is also raised when the client is authenticating to
//          a user account which was created without a verifier suitable for
//          the client software version. In this situation, that account's
//          password must be reset, in order for the required verifier to
//          be generated and allow authentication to proceed successfully.

在服务器端查看表:dba_users,PASSWORD_VERSIONS 列是:11G 12C

SQL> select username, account_status,password_versions from dba_users where account_status='OPEN';

USERNAME                       ACCOUNT_STATUS                                                   PASSWORD_VERSIONS
------------------------------ ---------------------------------------------------------------- ----------------------------------
SYS                            OPEN                                                             11G 12C
SYSTEM                         OPEN                                                             11G 12C
GSMCATUSER                     OPEN                                                             11G 12C
JY                             OPEN                                                             11G 12C
MYGDSADMIN                     OPEN                                                             11G 12C
APP_SCHEMA                     OPEN                                                             11G 12C

6 rows selected.

服务器端:修改 sqlnet.ora 配置:(配置修改后,不需要重启oracle服务器)我这里是新创建的sqlnet.ora,因为原来没有创建

[oracle@shard1 admin]$ ls -lrt
总用量 12
-rw-r--r--. 1 oracle oinstall 1441 8月  28 2015 shrept.lst
drwxr-xr-x. 2 oracle oinstall   61 10月 12 2017 samples
-rw-r-----. 1 oracle oinstall  960 1月  18 2018 tnsnames.ora
-rw-r--r--  1 oracle oinstall  504 2月  17 12:58 listener.ora
[oracle@shard1 admin]$ vi sqlnet.ora
SQLNET.ALLOWED_LOGON_VERSION_SERVER=11
SQLNET.ALLOWED_LOGON_VERSION_CLIENT=11

客户端:再次尝试连接,提示用户名密码错误:

C:\Users\Administrator>sqlplus sys/abcd@shardcat as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Sat Dec 20 22:16:06 2014

Copyright (c) 1982, 2010, Oracle.  All rights reserved.

ERROR:
ORA-01017: invalid username/password; logon denied

修改服务器端 sqlnet.ora 后,需要重新登录sqlplus,再修改用户密码,否则修改用户密码后,标记的密码版本仍然为11G 12C;重新登录sqlplus,修改scott用户密码,并查看 PASSWORD_VERSIONS,多了一个 10G

[oracle@shard1 ~]$ sqlplus sys/abcd@shardcat as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Mon Feb 17 13:07:28 2020

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> alter user sys identified by "abcd";

User altered.


SQL> select username, account_status,password_versions from dba_users where account_status='OPEN';

USERNAME                       ACCOUNT_STATUS                                                   PASSWORD_VERSIONS
------------------------------ ---------------------------------------------------------------- ----------------------------------
SYS                            OPEN                                                             10G 11G 12C
SYSTEM                         OPEN                                                             11G 12C
GSMCATUSER                     OPEN                                                             11G 12C
JY                             OPEN                                                             11G 12C
MYGDSADMIN                     OPEN                                                             11G 12C
APP_SCHEMA                     OPEN                                                             11G 12C

6 rows selected.

客户端:再次尝试登录oracle 12c,成功:

C:\Users\Administrator>sqlplus sys/abcd@shardcat as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Sat Dec 20 22:22:39 2014

Copyright (c) 1982, 2010, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL>

总结:oracle11g 客户端连接 oracle 12c服务器,需要在服务器端配置 sqlnet.ora,并重新修改用户密码。

Oracle Linux 6.7中 Oracle 11.2.0.4 RAC集群CRS异常处理

最近一个月单位上的Oracle RAC集群CRS异常导致集群命令无法使用,执行crsctl stat res -t命令出现如下错误:

[grid@db1 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

[grid@db2 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

但是数据库可以正常访问,业务系统也运行正常

[root@db1 ~]# ps -ef | grep pmon
root       8024 204594  0 21:11 pts/0    00:00:00 grep pmon
grid      77120      1  0 Dec21 ?        00:04:21 asm_pmon_+ASM1
oracle    77790      1  0 Dec21 ?        00:05:18 ora_pmon_CAIWU1
oracle    77794      1  0 Dec21 ?        00:05:08 ora_pmon_dadb1
oracle    77848      1  0 Dec21 ?        00:05:39 ora_pmon_chdyl1
oracle    77910      1  0 Dec21 ?        00:07:47 ora_pmon_RLZY1

[root@db2 ~]# ps -ef | grep pmon
grid      27745      1  0 Dec21 ?        00:04:21 asm_pmon_+ASM2
oracle    28393      1  0 Dec21 ?        00:05:21 ora_pmon_dadb2
oracle    28569      1  0 Dec21 ?        00:04:58 ora_pmon_CAIWU2
oracle    28573      1  0 Dec21 ?        00:05:36 ora_pmon_chdyl2
oracle    28583      1  0 Dec21 ?        00:07:49 ora_pmon_RLZY2

查看ASM磁盘组的状态,发现OCR磁盘组确实offline了

[grid@db1 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576   3072000  2679522                0         2679522              0             N  ARCH/
MOUNTED  EXTERN  N         512   4096  1048576    204800   151138                0          151138              0             N  CWDATA/
MOUNTED  EXTERN  N         512   4096  1048576    512000   472546                0          472546              0             N  DADATA/
MOUNTED  EXTERN  N         512   4096  1048576   3072000   595334                0          595334              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576   1843200   609953                0          609953              0             N  SBDATA/

[grid@db2 ~]$ asmcmd lsdg
State    Type    Rebal  Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512   4096  1048576   3072000  2679522                0         2679522              0             N  ARCH/
MOUNTED  EXTERN  N         512   4096  1048576    204800   151138                0          151138              0             N  CWDATA/
MOUNTED  EXTERN  N         512   4096  1048576    512000   472546                0          472546              0             N  DADATA/
MOUNTED  EXTERN  N         512   4096  1048576   3072000   595334                0          595334              0             N  DATA/
MOUNTED  EXTERN  N         512   4096  1048576   1843200   609953                0          609953              0             N  SBDATA/

手工将crsdg上线,命令能够执行成功,但执行crsctl stat res -t命令仍然报错。

[grid@db1 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Mon Dec 30 21:15:33 2019

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup ocr mount;

Diskgroup altered.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

[grid@db1 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

[grid@db2 ~]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Mon Dec 30 21:15:05 2019

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup ocr mount;

Diskgroup altered.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options

[grid@db2 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

检查节点db1的alert_+ASM1.log有如下报错,说是不能访问OCR磁盘组的相关磁盘而强制dismount了OCR磁盘,但使用dd命令测试是能够访问的

Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_77212.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
WARNING: requested mirror side 1 of virtual extent 0 logical extent 0 offset 102400 is not allocated; I/O request failed
WARNING: requested mirror side 2 of virtual extent 0 logical extent 1 offset 102400 is not allocated; I/O request failed
Errors in file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_77212.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
Sat Dec 28 05:30:48 2019
SQL> alter diskgroup OCR check /* proxy */ 
ORA-15032: not all alterations performed
ORA-15001: diskgroup "OCR" does not exist or is not mounted
ERROR: alter diskgroup OCR check /* proxy */
NOTE: client exited [77184]
Sat Dec 28 05:30:49 2019
NOTE: [crsd.bin@db1 (TNS V1-V3) 35285] opening OCR file
Sat Dec 28 05:30:51 2019
NOTE: [crsd.bin@db1 (TNS V1-V3) 35305] opening OCR file
Sat Dec 28 05:30:53 2019
NOTE: [crsd.bin@db1 (TNS V1-V3) 35322] opening OCR file
Sat Dec 28 05:30:55 2019
NOTE: [crsd.bin@db1 (TNS V1-V3) 35346] opening OCR file
Sat Dec 28 05:30:57 2019
NOTE: [crsd.bin@db1 (TNS V1-V3) 35363] opening OCR file
Sat Dec 28 05:31:00 2019
NOTE: [crsd.bin@db1 (TNS V1-V3) 35459] opening OCR file
Sat Dec 28 05:31:02 2019
NOTE: [crsd.bin@db1 (TNS V1-V3) 35481] opening OCR file
Sat Dec 28 05:31:04 2019
NOTE: [crsd.bin@db1 (TNS V1-V3) 35520] opening OCR file
Sat Dec 28 05:31:06 2019
NOTE: [crsd.bin@db1 (TNS V1-V3) 35539] opening OCR file
Sat Dec 28 05:31:08 2019
NOTE: [crsd.bin@db1 (TNS V1-V3) 35557] opening OCR file
Sat Dec 28 21:00:10 2019
Warning: VKTM detected a time drift.
Time drifts can result in an unexpected behavior such as time-outs. Please check trace file for more details.

检查错误日志

[root@db1 ~]# more /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_77212.trc
Trace file /u01/app/grid/diag/asm/+asm/+ASM1/trace/+ASM1_ora_77212.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
ORACLE_HOME = /u01/app/11.2.0/grid
System name:    Linux
Node name:      db1
Release:        3.8.13-68.3.4.el6uek.x86_64
Version:        #2 SMP Tue Jul 14 15:03:36 PDT 2015
Machine:        x86_64
Instance name: +ASM1
Redo thread mounted by this instance: 0 
Oracle process number: 24
Unix process pid: 77212, image: oracle@db1 (TNS V1-V3)


*** 2019-12-28 05:30:44.894
*** SESSION ID:(2929.3) 2019-12-28 05:30:44.894
*** CLIENT ID:() 2019-12-28 05:30:44.894
*** SERVICE NAME:() 2019-12-28 05:30:44.894
*** MODULE NAME:(crsd.bin@db1 (TNS V1-V3)) 2019-12-28 05:30:44.894
*** ACTION NAME:() 2019-12-28 05:30:44.894
 
Received ORADEBUG command (#1) 'CLEANUP_KFK_FD' from process 'Unix process pid: 35253, image: '

*** 2019-12-28 05:30:44.895
Finished processing ORADEBUG command (#1) 'CLEANUP_KFK_FD'

*** 2019-12-28 05:30:48.235
WARNING:failed xlate 1 
ORA-15078: ASM diskgroup was forcibly dismounted
ksfdrfms:Mirror Read file=+OCR.255.4294967295 fob=0x9b00e5d8 bufp=0x7f5dd012ba00 blkno=25 nbytes=4096
WARNING:failed xlate 1 
WARNING: requested mirror side 1 of virtual extent 0 logical extent 0 offset 102400 is not allocated; I/O request failed
ksfdrfms:Read failed from mirror side=1 logical extent number=0 dskno=65535
WARNING:failed xlate 1 
WARNING: requested mirror side 2 of virtual extent 0 logical extent 1 offset 102400 is not allocated; I/O request failed
ksfdrfms:Read failed from mirror side=2 logical extent number=1 dskno=65535
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted

检查节点db1的alertdb1.log有如下报错,也是说不能访问OCR磁盘组的相关磁盘

2019-12-28 05:30:48.468: 
[/u01/app/11.2.0/grid/bin/oraagent.bin(77466)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:1:4} in /u01/app/11.2.0/grid/log/db1/agent/crsd/oraagent_grid/oraagent_grid.lo
g.
2019-12-28 05:30:48.468: 
[/u01/app/11.2.0/grid/bin/oraagent.bin(77684)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) {0:7:332} in /u01/app/11.2.0/grid/log/db1/agent/crsd/oraagent_oracle/oraagent_o
racle.log.
2019-12-28 05:30:48.471: 
[/u01/app/11.2.0/grid/bin/orarootagent.bin(77482)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:5:11497} in /u01/app/11.2.0/grid/log/db1/agent/crsd/orarootagent_root/
orarootagent_root.log.
2019-12-28 05:30:48.480: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:30:50.003: 
[crsd(35285)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:30:50.021: 
[crsd(35285)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:30:50.520: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:30:51.918: 
[crsd(35305)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:30:51.929: 
[crsd(35305)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:30:52.557: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:30:53.945: 
[crsd(35322)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:30:53.956: 
[crsd(35322)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:30:54.595: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:30:55.976: 
[crsd(35346)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:30:55.988: 
[crsd(35346)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:30:56.633: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:30:58.010: 
[crsd(35363)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:30:58.020: 
[crsd(35363)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:30:58.669: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:31:00.043: 
[crsd(35459)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:31:00.054: 
[crsd(35459)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:31:00.706: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:31:02.093: 
[crsd(35481)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:31:02.103: 
[crsd(35481)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:31:02.742: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:31:04.109: 
[crsd(35520)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:31:04.119: 
[crsd(35520)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:31:04.777: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:31:06.141: 
[crsd(35539)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:31:06.151: 
[crsd(35539)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:31:06.810: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:31:08.181: 
[crsd(35557)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:31:08.191: 
[crsd(35557)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db1/crsd/crsd.log.
2019-12-28 05:31:08.846: 
[ohasd(33022)]CRS-2765:Resource 'ora.crsd' has failed on server 'db1'.
2019-12-28 05:31:08.847: 
[ohasd(33022)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2019-12-28 05:31:08.848: 
[ohasd(33022)]CRS-2769:Unable to failover resource 'ora.crsd'.

检查节点db1的oraagent_grid.log文件有如下报错,显示OCR磁盘组的状态被改变为offline了。

2019-12-28 05:30:16.531: [    AGFW][511039232]{1:30746:2} Agent received the message: AGENT_HB[Engine] ID 12293:113720
2019-12-28 05:30:37.808: [    AGFW][511039232]{1:30746:9373} Agent received the message: RESOURCE_STOP[ora.OCR.dg db1 1] ID 4099:113730
2019-12-28 05:30:37.808: [    AGFW][511039232]{1:30746:9373} Preparing STOP command for: ora.OCR.dg db1 1
2019-12-28 05:30:37.808: [    AGFW][511039232]{1:30746:9373} ora.OCR.dg db1 1 state changed from: ONLINE to: STOPPING
2019-12-28 05:30:37.809: [ora.OCR.dg][513140480]{1:30746:9373} [stop] (:CLSN00108:) clsn_agent::stop {
2019-12-28 05:30:37.810: [ora.OCR.dg][513140480]{1:30746:9373} [stop] DgpAgent::stop: enter { 
2019-12-28 05:30:37.810: [ora.OCR.dg][513140480]{1:30746:9373} [stop] getResAttrib: attrib name USR_ORA_OPI value true len 4
2019-12-28 05:30:37.810: [ora.OCR.dg][513140480]{1:30746:9373} [stop] Agent::flagUsrOraOpiIsSet(true) reason not dependency
2019-12-28 05:30:37.810: [ora.OCR.dg][513140480]{1:30746:9373} [stop] DgpAgent::stop: tha exit }
2019-12-28 05:30:37.810: [ora.OCR.dg][513140480]{1:30746:9373} [stop] DgpAgent::stopSingle status:2 }
2019-12-28 05:30:37.810: [ora.OCR.dg][513140480]{1:30746:9373} [stop] (:CLSN00108:) clsn_agent::stop }
2019-12-28 05:30:37.810: [    AGFW][513140480]{1:30746:9373} Command: stop for resource: ora.OCR.dg db1 1 completed with status: SUCCESS
2019-12-28 05:30:37.810: [ora.OCR.dg][513140480]{1:30746:9373} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0
2019-12-28 05:30:37.811: [    AGFW][511039232]{1:30746:9373} Agent sending reply for: RESOURCE_STOP[ora.OCR.dg db1 1] ID 4099:113730
2019-12-28 05:30:37.838: [ora.OCR.dg][513140480]{1:30746:9373} [check] DgpAgent::runCheck: asm stat asmRet 0
2019-12-28 05:30:37.839: [ora.OCR.dg][513140480]{1:30746:9373} [check] DgpAgent::getConnxn connected
2019-12-28 05:30:37.844: [ora.OCR.dg][513140480]{1:30746:9373} [check] DgpAgent::queryDgStatus excp no data found
2019-12-28 05:30:37.844: [ora.OCR.dg][513140480]{1:30746:9373} [check] DgpAgent::queryDgStatus no data found in v$asm_diskgroup_stat
2019-12-28 05:30:37.844: [ora.OCR.dg][513140480]{1:30746:9373} [check] DgpAgent::queryDgStatus dgName OCR ret 1
2019-12-28 05:30:37.845: [    AGFW][511039232]{1:30746:9373} ora.OCR.dg db1 1 state changed from: STOPPING to: OFFLINE
2019-12-28 05:30:37.845: [    AGFW][511039232]{1:30746:9373} Agent sending last reply for: RESOURCE_STOP[ora.OCR.dg db1 1] ID 4099:113730
2019-12-28 05:30:43.889: [ora.asm][503641856]{1:30746:2} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0
2019-12-28 05:30:43.920: [ora.asm][503641856]{1:30746:2} [check] AsmProxyAgent::check clsagfw_res_status 0
2019-12-28 05:30:48.465: [ CRSCOMM][521545472] IpcC: IPC client connection 6c to member 0 has been removed
2019-12-28 05:30:48.465: [CLSFRAME][521545472] Removing IPC Member:{Relative|Node:0|Process:0|Type:1}
2019-12-28 05:30:48.465: [CLSFRAME][521545472] Disconnected from CRSD:db1 process: {Relative|Node:0|Process:0|Type:1}
2019-12-28 05:30:48.474: [   AGENT][511039232]{0:1:4} {0:1:4} Created alert : (:CRSAGF00117:) :  Disconnected from server, Agent is shutting down.
2019-12-28 05:30:48.474: [    AGFW][511039232]{0:1:4} Agent is exiting with exit code: 1

检查节点db2的alert_+ASM2.log有如下报错,出现了类似的”Waited 15 secs for write IO to PST disk 0 in group 1″信息,这说明对OCR磁盘组执行写操作时超时了15秒.

Sat Dec 28 03:02:51 2019
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 3.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 5.
WARNING: Waited 15 secs for write IO to PST disk 1 in group 5.
WARNING: Waited 15 secs for write IO to PST disk 2 in group 5.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 5.
WARNING: Waited 15 secs for write IO to PST disk 1 in group 5.
WARNING: Waited 15 secs for write IO to PST disk 2 in group 5.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 6.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 6.
Sat Dec 28 03:02:51 2019
NOTE: process _b000_+asm2 (45488) initiating offline of disk 0.3916160907 (OCR1) with mask 0x7e in group 5
NOTE: process _b000_+asm2 (45488) initiating offline of disk 1.3916160906 (OCR2) with mask 0x7e in group 5
NOTE: process _b000_+asm2 (45488) initiating offline of disk 2.3916160905 (OCR3) with mask 0x7e in group 5
NOTE: checking PST: grp = 5
GMON checking disk modes for group 5 at 19 for pid 27, osid 45488
ERROR: no read quorum in group: required 2, found 0 disks
NOTE: checking PST for grp 5 done.
NOTE: initiating PST update: grp = 5, dsk = 0/0xe96bdf8b, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 5, dsk = 1/0xe96bdf8a, mask = 0x6a, op = clear
NOTE: initiating PST update: grp = 5, dsk = 2/0xe96bdf89, mask = 0x6a, op = clear
GMON updating disk modes for group 5 at 20 for pid 27, osid 45488
ERROR: no read quorum in group: required 2, found 0 disks
Sat Dec 28 03:02:51 2019
NOTE: cache dismounting (not clean) group 5/0x8F5B2F9F (OCR) 
NOTE: messaging CKPT to quiesce pins Unix process pid: 45490, image: oracle@db2 (B001)
Sat Dec 28 03:02:51 2019
NOTE: halting all I/Os to diskgroup 5 (OCR)
Sat Dec 28 03:02:52 2019
NOTE: LGWR doing non-clean dismount of group 5 (OCR)
NOTE: LGWR sync ABA=23.100 last written ABA 23.100
WARNING: Offline for disk OCR1 in mode 0x7f failed.
WARNING: Offline for disk OCR2 in mode 0x7f failed.
WARNING: Offline for disk OCR3 in mode 0x7f failed.
Sat Dec 28 03:02:52 2019
kjbdomdet send to inst 1
detach from dom 5, sending detach message to inst 1
Sat Dec 28 03:02:52 2019
List of instances:
 1 2
Dirty detach reconfiguration started (new ddet inc 1, cluster inc 36)
 Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 5 invalid = TRUE 
 0 GCS resources traversed, 0 cancelled
Dirty Detach Reconfiguration complete
Sat Dec 28 03:02:52 2019
WARNING: dirty detached from domain 5
NOTE: cache dismounted group 5/0x8F5B2F9F (OCR) 
SQL> alter diskgroup OCR dismount force /* ASM SERVER:2405117855 */ 
Sat Dec 28 03:02:52 2019
NOTE: cache deleting context for group OCR 5/0x8f5b2f9f
GMON dismounting group 5 at 21 for pid 28, osid 45490
NOTE: Disk OCR1 in mode 0x7f marked for de-assignment
NOTE: Disk OCR2 in mode 0x7f marked for de-assignment
NOTE: Disk OCR3 in mode 0x7f marked for de-assignment
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 5
Sat Dec 28 03:03:03 2019
WARNING: Waited 27 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 27 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 24 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 24 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 27 secs for write IO to PST disk 0 in group 3.
WARNING: Waited 27 secs for write IO to PST disk 0 in group 3.
WARNING: Waited 21 secs for write IO to PST disk 0 in group 4.
WARNING: Waited 21 secs for write IO to PST disk 0 in group 4.
WARNING: Waited 27 secs for write IO to PST disk 0 in group 6.
WARNING: Waited 27 secs for write IO to PST disk 0 in group 6.
Sat Dec 28 03:03:03 2019
ASM Health Checker found 1 new failures
Sat Dec 28 03:03:22 2019
SUCCESS: diskgroup OCR was dismounted
SUCCESS: alter diskgroup OCR dismount force /* ASM SERVER:2405117855 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group OCR
Sat Dec 28 03:03:22 2019
NOTE: diskgroup resource ora.OCR.dg is offline
Sat Dec 28 03:03:22 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Sat Dec 28 05:30:34 2019
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 6.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 6.
Sat Dec 28 05:30:37 2019
 Received dirty detach msg from inst 1 for dom 5
Sat Dec 28 05:30:37 2019
List of instances:
 1 2
Dirty detach reconfiguration started (new ddet inc 2, cluster inc 36)
 Global Resource Directory partially frozen for dirty detach
* dirty detach - domain 5 invalid = TRUE 
 0 GCS resources traversed, 0 cancelled
freeing rdom 5
Dirty Detach Reconfiguration complete
Sat Dec 28 05:30:37 2019
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
WARNING: requested mirror side 1 of virtual extent 5 logical extent 0 offset 704512 is not allocated; I/O request failed
WARNING: requested mirror side 2 of virtual extent 5 logical extent 1 offset 704512 is not allocated; I/O request failed
Errors in file /u01/app/grid/diag/asm/+asm/+ASM2/trace/+ASM2_ora_27831.trc:
ORA-15078: ASM diskgroup was forcibly dismounted
ORA-15078: ASM diskgroup was forcibly dismounted
Sat Dec 28 05:30:37 2019
SQL> alter diskgroup OCR check /* proxy */ 
ORA-15032: not all alterations performed
ORA-15001: diskgroup "OCR" does not exist or is not mounted
ERROR: alter diskgroup OCR check /* proxy */
Sat Dec 28 05:30:44 2019
WARNING: Waited 20 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 20 secs for write IO to PST disk 0 in group 2.
Sat Dec 28 05:30:48 2019
NOTE: client exited [27819]
Sat Dec 28 05:30:49 2019
NOTE: [crsd.bin@db2 (TNS V1-V3) 142641] opening OCR file
Sat Dec 28 05:30:51 2019
NOTE: [crsd.bin@db2 (TNS V1-V3) 142660] opening OCR file
Sat Dec 28 05:30:53 2019
NOTE: [crsd.bin@db2 (TNS V1-V3) 142678] opening OCR file
Sat Dec 28 05:30:55 2019
NOTE: [crsd.bin@db2 (TNS V1-V3) 142696] opening OCR file
Sat Dec 28 05:30:57 2019
NOTE: [crsd.bin@db2 (TNS V1-V3) 142723] opening OCR file
Sat Dec 28 05:30:59 2019
NOTE: [crsd.bin@db2 (TNS V1-V3) 142744] opening OCR file
Sat Dec 28 05:31:01 2019
NOTE: [crsd.bin@db2 (TNS V1-V3) 142773] opening OCR file
Sat Dec 28 05:31:03 2019
NOTE: [crsd.bin@db2 (TNS V1-V3) 142792] opening OCR file
Sat Dec 28 05:31:05 2019
NOTE: [crsd.bin@db2 (TNS V1-V3) 142806] opening OCR file
Sat Dec 28 05:31:07 2019
NOTE: [crsd.bin@db2 (TNS V1-V3) 142821] opening OCR file
Sat Dec 28 06:18:42 2019
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 1.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.
WARNING: Waited 15 secs for write IO to PST disk 0 in group 2.

检查节点db2的alertdb2.log有如下报错,也是说不能访问OCR磁盘组的相关磁盘,但使用dd命令测试是可以访问的。

2019-12-28 05:30:48.019: 
[/u01/app/11.2.0/grid/bin/oraagent.bin(28268)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_oracle' disconnected from server. Details at (:CRSAGF00117:) {0:7:73} in /u01/app/11.2.0/grid/log/db2/agent/crsd/oraagent_oracle/oraagent_oracle.log.
2019-12-28 05:30:48.019: 
[/u01/app/11.2.0/grid/bin/scriptagent.bin(37953)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/scriptagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:9:8} in /u01/app/11.2.0/grid/log/db2/agent/crsd/scriptagent_grid/scriptagent_grid.log.
2019-12-28 05:30:48.020: 
[/u01/app/11.2.0/grid/bin/oraagent.bin(28009)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/oraagent_grid' disconnected from server. Details at (:CRSAGF00117:) {0:1:10} in /u01/app/11.2.0/grid/log/db2/agent/crsd/oraagent_grid/oraagent_grid.log.
2019-12-28 05:30:48.021: 
[/u01/app/11.2.0/grid/bin/orarootagent.bin(28025)]CRS-5822:Agent '/u01/app/11.2.0/grid/bin/orarootagent_root' disconnected from server. Details at (:CRSAGF00117:) {0:5:373} in /u01/app/11.2.0/grid/log/db2/agent/crsd/orarootagent_root/orarootagent_root.log.
2019-12-28 05:30:48.024: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:30:49.410: 
[crsd(142641)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:49.420: 
[crsd(142641)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:50.063: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:30:51.442: 
[crsd(142660)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:51.451: 
[crsd(142660)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:52.100: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:30:53.471: 
[crsd(142678)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:53.480: 
[crsd(142678)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:54.138: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:30:55.507: 
[crsd(142696)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:55.517: 
[crsd(142696)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:56.176: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:30:57.551: 
[crsd(142723)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:57.560: 
[crsd(142723)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:58.216: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:30:59.592: 
[crsd(142744)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:30:59.602: 
[crsd(142744)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:31:00.253: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:31:01.627: 
[crsd(142773)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:31:01.636: 
[crsd(142773)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:31:02.290: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:31:03.658: 
[crsd(142792)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:31:03.668: 
[crsd(142792)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:31:04.327: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:31:05.701: 
[crsd(142806)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:31:05.711: 
[crsd(142806)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:31:06.365: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:31:07.726: 
[crsd(142821)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:31:07.735: 
[crsd(142821)]CRS-0804:Cluster Ready Service aborted due to Oracle Cluster Registry error [PROC-26: Error while accessing the physical storage
]. Details at (:CRSD00111:) in /u01/app/11.2.0/grid/log/db2/crsd/crsd.log.
2019-12-28 05:31:08.402: 
[ohasd(13034)]CRS-2765:Resource 'ora.crsd' has failed on server 'db2'.
2019-12-28 05:31:08.402: 
[ohasd(13034)]CRS-2771:Maximum restart attempts reached for resource 'ora.crsd'; will not restart.
2019-12-28 05:31:08.403: 
[ohasd(13034)]CRS-2769:Unable to failover resource 'ora.crsd'.

检查节点db2的oraagent_grid.log文件有如下报错:

2019-12-28 05:29:59.329: [    AGFW][3601811200]{2:6928:2} Agent received the message: AGENT_HB[Engine] ID 12293:273304
2019-12-28 05:30:17.162: [ora.LISTENER_SCAN2.lsnr][3592312576]{1:34166:403} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl. 
2019-12-28 05:30:17.267: [ora.LISTENER_SCAN2.lsnr][3592312576]{1:34166:403} [check] execCmd ret = 0
2019-12-28 05:30:17.267: [ora.LISTENER_SCAN2.lsnr][3592312576]{1:34166:403} [check] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1
2019-12-28 05:30:17.298: [ora.LISTENER_SCAN2.lsnr][3592312576]{1:34166:403} [check] ScanLsnrAgent::checkDependentVipResource: scanVipResource = ora.scan2.vip, statRet = 0
2019-12-28 05:30:17.881: [ora.LISTENER_SCAN3.lsnr][2950686464]{1:34166:403} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl. 
2019-12-28 05:30:17.986: [ora.LISTENER_SCAN3.lsnr][2950686464]{1:34166:403} [check] execCmd ret = 0
2019-12-28 05:30:17.987: [ora.LISTENER_SCAN3.lsnr][2950686464]{1:34166:403} [check] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1
2019-12-28 05:30:18.019: [ora.LISTENER_SCAN3.lsnr][2950686464]{1:34166:403} [check] ScanLsnrAgent::checkDependentVipResource: scanVipResource = ora.scan3.vip, statRet = 0
2019-12-28 05:30:27.292: [ora.asm][2950686464]{2:6928:2} [check] CrsCmd::ClscrsCmdData::stat entity 1 statflag 33 useFilter 0
2019-12-28 05:30:27.319: [ora.asm][2950686464]{2:6928:2} [check] AsmProxyAgent::check clsagfw_res_status 0
2019-12-28 05:30:34.522: [ora.ons][2950686464]{2:6928:2} [check] getOracleHomeAttrib: oracle_home = /u01/app/11.2.0/grid
2019-12-28 05:30:34.522: [ora.ons][2950686464]{2:6928:2} [check] Utils:execCmd action = 3 flags = 6 ohome = /u01/app/11.2.0/grid/opmn/ cmdname = onsctli. 
2019-12-28 05:30:34.627: [ora.ons][2950686464]{2:6928:2} [check] (:CLSN00010:)ons is running ...
2019-12-28 05:30:34.627: [ora.ons][2950686464]{2:6928:2} [check] (:CLSN00010:)
2019-12-28 05:30:34.628: [ora.ons][2950686464]{2:6928:2} [check] execCmd ret = 0
2019-12-28 05:30:37.858: [ USRTHRD][3575973632]{1:30748:9373} Processing the event CRS_RESOURCE_STATE_CHANGE
2019-12-28 05:30:38.652: [ora.LISTENER.lsnr][3594413824]{2:6928:2} [check] Utils:execCmd action = 3 flags = 38 ohome = (null) cmdname = lsnrctl. 
2019-12-28 05:30:38.757: [ora.LISTENER.lsnr][3594413824]{2:6928:2} [check] execCmd ret = 0
2019-12-28 05:30:48.017: [ CRSCOMM][3612317440] IpcC: IPC client connection 6c to member 0 has been removed
2019-12-28 05:30:48.017: [CLSFRAME][3612317440] Removing IPC Member:{Relative|Node:0|Process:0|Type:1}
2019-12-28 05:30:48.017: [CLSFRAME][3612317440] Disconnected from CRSD:db2 process: {Relative|Node:0|Process:0|Type:1}
2019-12-28 05:30:48.020: [   AGENT][3601811200]{0:1:10} {0:1:10} Created alert : (:CRSAGF00117:) :  Disconnected from server, Agent is shutting down.
2019-12-28 05:30:48.020: [    AGFW][3601811200]{0:1:10} Agent is exiting with exit code: 1

检查2个节点的/var/log/messages日志,发现2个节点均有多路径相关的错误信息,但相关的磁盘是用于备份的,不是用于生产数据库所使用的

Dec 30 05:25:31 db1 multipathd: backup2: sdcr - emc_clariion_checker: query command indicates error
Dec 30 05:25:31 db1 multipathd: checker failed path 69:240 in map backup2
Dec 30 05:25:31 db1 kernel: device-mapper: multipath: Failing path 69:240.
Dec 30 05:25:31 db1 multipathd: backup2: remaining active paths: 3
Dec 30 05:25:37 db1 multipathd: backup2: sdcr - emc_clariion_checker: Active path is healthy.
Dec 30 05:25:37 db1 multipathd: 69:240: reinstated
Dec 30 05:25:37 db1 multipathd: backup2: remaining active paths: 4
Dec 30 05:25:37 db1 kernel: sd 5:0:3:2: emc: ALUA failover mode detected
Dec 30 05:25:37 db1 kernel: sd 5:0:3:2: emc: at SP A Port 5 (owned, default SP A)
Dec 30 05:26:03 db1 kernel: qla2xxx [0000:05:00.1]-801c:5: Abort command issued nexus=5:3:4 --  1 2002.
Dec 30 06:03:35 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 06:15:23 db1 multipathd: backup3: sdcq - emc_clariion_checker: Read error for WWN 600601608b203300d563752524c1e611.  Sense data are 0x0/0x0/0x0.
Dec 30 06:15:23 db1 kernel: qla2xxx [0000:05:00.1]-801c:5: Abort command issued nexus=5:3:1 --  1 2002.
Dec 30 06:15:23 db1 kernel: device-mapper: multipath: Failing path 69:224.
Dec 30 06:15:23 db1 multipathd: checker failed path 69:224 in map backup3
Dec 30 06:15:23 db1 multipathd: backup3: remaining active paths: 3
Dec 30 06:15:28 db1 multipathd: backup3: sdcq - emc_clariion_checker: Active path is healthy.
Dec 30 06:15:28 db1 multipathd: 69:224: reinstated
Dec 30 06:15:28 db1 multipathd: backup3: remaining active paths: 4
Dec 30 06:15:28 db1 kernel: sd 5:0:3:1: emc: ALUA failover mode detected
Dec 30 06:15:28 db1 kernel: sd 5:0:3:1: emc: at SP A Port 5 (owned, default SP A)
Dec 30 06:59:29 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 07:53:22 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 07:55:11 db1 multipathd: sdct: couldn't get asymmetric access state
Dec 30 07:55:11 db1 multipathd: backup4: load table [0 2147483648 multipath 2 queue_if_no_path retain_attached_hw_handler 1 emc 2 1 round-robin 0 2 1 70:16 1 66:240 1 round-robin 0 2 1 8:64 1 67:80 1]
Dec 30 07:55:11 db1 kernel: sd 5:0:3:4: emc: ALUA failover mode detected
Dec 30 07:55:11 db1 kernel: sd 5:0:3:4: emc: at SP A Port 5 (owned, default SP A)
Dec 30 07:55:11 db1 kernel: sd 4:0:3:4: emc: ALUA failover mode detected
Dec 30 07:55:11 db1 kernel: sd 4:0:3:4: emc: at SP A Port 4 (owned, default SP A)
Dec 30 07:55:35 db1 multipathd: backup2: sdcr - emc_clariion_checker: Read error for WWN 600601608b203300d663752524c1e611.  Sense data are 0x0/0x0/0x0.
Dec 30 07:55:35 db1 multipathd: checker failed path 69:240 in map backup2
Dec 30 07:55:35 db1 multipathd: backup2: remaining active paths: 3
Dec 30 07:55:35 db1 kernel: device-mapper: multipath: Failing path 69:240.
Dec 30 07:55:40 db1 multipathd: backup2: sdcr - emc_clariion_checker: Active path is healthy.
Dec 30 07:55:40 db1 multipathd: 69:240: reinstated
Dec 30 07:55:40 db1 multipathd: backup2: remaining active paths: 4
Dec 30 07:55:40 db1 kernel: sd 5:0:3:2: emc: ALUA failover mode detected
Dec 30 07:55:40 db1 kernel: sd 5:0:3:2: emc: at SP A Port 5 (owned, default SP A)
Dec 30 08:39:47 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 08:43:36 db1 multipathd: mpathb: load table [0 20971520 multipath 2 queue_if_no_path retain_attached_hw_handler 1 emc 2 1 round-robin 0 2 1 69:208 1 66:176 1 round-robin 0 2 1 8:0 1 67:16 1]
Dec 30 08:43:36 db1 kernel: sd 5:0:3:0: emc: ALUA failover mode detected
Dec 30 08:43:36 db1 kernel: sd 5:0:3:0: emc: at SP A Port 5 (owned, default SP A)
Dec 30 08:43:36 db1 kernel: sd 4:0:3:0: emc: ALUA failover mode detected
Dec 30 08:43:36 db1 kernel: sd 4:0:3:0: emc: at SP A Port 4 (owned, default SP A)
Dec 30 09:24:04 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 10:13:09 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 11:06:07 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 12:07:36 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 13:08:58 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 14:00:19 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 14:52:20 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 15:40:45 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 16:34:38 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 17:09:56 db1 auditd[15975]: Audit daemon rotating log files
Dec 30 17:38:16 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 18:59:38 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
Dec 30 19:54:43 db1 CLSD: The clock on host db1 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.

经查看集群相关日志可以确定,由于存储磁盘出现IO问题(或光线闪断、或IO延迟),导致集群CRS异常宕机。但是,比较奇怪的是,虽然CSR掉线了,ASM实例和DB实例却好好的,还可以正常使用。查询oracle support发现一篇文章1581864.1?提到ASM CRS仲裁盘访问超时与隐藏参数_asm_hbeatiowait有关系,而ASM的隐藏参数_asm_hbeatiowait由于操作系统多路径Multipath配置的polling_interval有关,具体的故障原因是操作系统盘的判断访问超时远大于数据库ASM仲裁盘访问超时,导致ORACLE RAC判定ASM中仲裁盘无法访问从而将仲裁盘强制Offline。解决的思路是:首先,确定操作系统polling_interval参数与数据库ASM隐藏参数值_asm_hbeatiowait,将_asm_hbeatiowait的值调整到比polling_interval值大即可。
下面是具体的解决操作:
1、查看数据库RAC ASM的_asm_hbeatiowait值(默认是15秒):

SQL> col ksppinm for a20
SQL> col ksppstvl for a40
SQL> col ksppdesc for a80
SQL> SELECT   ksppinm, ksppstvl, ksppdesc
  2     FROM   x$ksppi x, x$ksppcv y
  3    WHERE   x.indx = y.indx AND  ksppinm = '_asm_hbeatiowait';

KSPPINM              KSPPSTVL                                 KSPPDESC
-------------------- ---------------------------------------- --------------------------------------------------------------------------------
_asm_hbeatiowait     15                                       number of secs to wait for PST Async Hbeat IO return

2、查看操作存储盘访问超时时间(Oracle Linux 6.7默认是30秒)

[root@db1 ~]# cat /sys/block/sdb/device/timeout 
30

[root@db2 ~]# cat /sys/block/sdb/device/timeout 
30

3、将_asm_hbeatiowait 的值调整为45秒(该参数是静态参数,需要重启集群)

SQL>  alter system set "_asm_hbeatiowait"=45 scope=spfile sid='*';

System altered.

4.重启集群

5.检查集群状态

[grid@db1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ARCH.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.CWDATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.DADATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.DATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.LISTENER.lsnr
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.OCR.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.SBKDATA.dg
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.asm
               ONLINE  ONLINE       db1                      Started             
               ONLINE  ONLINE       db2                      Started             
ora.gsd
               OFFLINE OFFLINE      db1                                          
               OFFLINE OFFLINE      db2                                          
ora.net1.network
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
ora.ons
               ONLINE  ONLINE       db1                                          
               ONLINE  ONLINE       db2                                          
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       db2                                          
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       db1                                          
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       db1                                          
ora.caiwu.db
      1        ONLINE  ONLINE       db1                      Open                
      2        ONLINE  ONLINE       db2                      Open                
ora.chdyl.db
      1        ONLINE  ONLINE       db1                      Open                
      2        ONLINE  ONLINE       db2                      Open                
ora.cvu
      1        ONLINE  ONLINE       db1                                          
ora.dadb.db
      1        ONLINE  ONLINE       db1                      Open                
      2        ONLINE  ONLINE       db2                      Open                
ora.db1.vip
      1        ONLINE  ONLINE       db1                                          
ora.db2.vip
      1        ONLINE  ONLINE       db2                                          
ora.oc4j
      1        ONLINE  ONLINE       db1                                          
ora.rlzy.db
      1        ONLINE  ONLINE       db1                      Open                
      2        ONLINE  ONLINE       db2                      Open                
ora.scan1.vip
      1        ONLINE  ONLINE       db2                                          
ora.scan2.vip
      1        ONLINE  ONLINE       db1                                          
ora.scan3.vip
      1        ONLINE  ONLINE       db1                                          


到此处理完成。

RMAN删除归档日志出现RMAN-0813错误的处理

数据库备份日志报错:RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process,DB版本为11.2.0.4,根据备策略,每天备份归档,并且自动删除档,保留一个星期的归档,手功执如下命令:

delete noprompt archivelog until time 'sysdate-7';

依然报错,信息如下:

archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_24/thread_1_seq_2328.1241.1027887401 thread=1 sequence=2328
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_24/thread_1_seq_2329.1242.1027887719 thread=1 sequence=2329
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_24/thread_1_seq_2330.1243.1027887813 thread=1 sequence=2330
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_24/thread_1_seq_2331.1244.1027887873 thread=1 sequence=2331
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_25/thread_1_seq_2332.1245.1027902623 thread=1 sequence=2332
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_25/thread_1_seq_2333.1246.1027902719 thread=1 sequence=2333
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_25/thread_1_seq_2334.1247.1027908493 thread=1 sequence=2334
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_25/thread_1_seq_2335.1248.1027911721 thread=1 sequence=2335
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_26/thread_1_seq_2336.1249.1027989005 thread=1 sequence=2336
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_26/thread_1_seq_2337.1250.1027989067 thread=1 sequence=2337
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_26/thread_1_seq_2338.1251.1027990617 thread=1 sequence=2338
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_26/thread_1_seq_2339.1252.1028041713 thread=1 sequence=2339
RMAN-08137: WARNING: archived log not deleted, needed for standby or upstream capture process
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_27/thread_1_seq_2340.1253.1028075413 thread=1 sequence=2340
...............

在MOS搜索发现这是11g中的一个bug,描述的原因是在11g中,RMAN查看的是v$database中的MIN_REQUIRED_CAPTURE_CHANGE#,而不是DBA_CAPTURE。默认情况下,该值每6小时更新一次。因此,在默认情况下,v$database中的值将比当前值晚6个小时。
给出的解决方法是需要在delete 后面加上force。
手动执行delete noprompt force archivelog until time ‘sysdate-7’, 成功将归档删除。

deleted archived log
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_24/thread_1_seq_2330.1243.1027887813 RECID=18420 STAMP=1027887814
deleted archived log
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_24/thread_1_seq_2331.1244.1027887873 RECID=18421 STAMP=1027887875
deleted archived log
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_25/thread_1_seq_2332.1245.1027902623 RECID=18422 STAMP=1027902628
deleted archived log
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_25/thread_1_seq_2333.1246.1027902719 RECID=18423 STAMP=1027902721
deleted archived log
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_25/thread_1_seq_2334.1247.1027908493 RECID=18424 STAMP=1027908500
deleted archived log
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_25/thread_1_seq_2335.1248.1027911721 RECID=18425 STAMP=1027911722
deleted archived log
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_26/thread_1_seq_2336.1249.1027989005 RECID=18426 STAMP=1027989010
deleted archived log
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_26/thread_1_seq_2337.1250.1027989067 RECID=18427 STAMP=1027989069
deleted archived log
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_26/thread_1_seq_2338.1251.1027990617 RECID=18428 STAMP=1027990618
deleted archived log
archived log file name=+CWDATA/sjjhdg/archivelog/2019_12_26/thread_1_seq_2339.1252.1028041713 RECID=18429 STAMP=1028041715
...........
Deleted 800 objects
Proudly powered by WordPress | Indrajeet by Sus Hill.