Oracle 12c 使用RMAN搭建物理备库(RAC to RAC)

一.实例环境说明
主库环境说明
操作系统:Oracle Linux 7.1
数据库版本:12.2.0.1
IP:10.10.10.190/10.10.10.191
主机名:cs1/cs2
数据库名:cs
Db_unique_name:cs
实例名(SID)cs1/cs2

备库环境说明
操作系统:Oracle Linux 7.1
数据库版本:12.2.0.1
IP:10.10.10.171/172
主机名:jytest1/jytest2
数据库名:cs
Db_unique_nmae:cs_dg
实例名(SID)cs1/cs2

二.配置过程
2.1修改主库启用归档与force logging
首先检查主库是否启用归档

[oracle@cs1 ~]$ sqlplus /nolog

SQL*Plus: Release 12.2.0.1.0 Production on Fri May 11 12:54:07 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

SQL> conn sys/abcd@cs as sysdba
Connected.
SQL> archive log list
Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            /u01/app/oracle/product/12.2.0/db/dbs/arch
Oldest online log sequence     135
Current log sequence           136
SQL>

[oracle@cs2 ~]$ sqlplus /nolog

SQL*Plus: Release 12.2.0.1.0 Production on Fri May 11 12:54:22 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

SQL> conn sys/abcd@cs as sysdba
Connected.
SQL> archive log list
Database log mode              No Archive Mode
Automatic archival             Disabled
Archive destination            /u01/app/oracle/product/12.2.0/db/dbs/arch
Oldest online log sequence     116
Current log sequence           117
SQL>

现在主库没有启用归档,执行下面的命令来对主库启用归档

SQL> alter system set log_archive_dest_1='location=+data/arch/' scope=both sid='*';

System altered.

SQL> show parameter log_archive_for

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_format                   string      %t_%s_%r.dbf

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.

SQL> alter system set log_archive_dest_1='location=+data/arch/' scope=both sid='*';

System altered.

SQL> show parameter log_archive_for

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
log_archive_format                   string      %t_%s_%r.dbf
SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  8794848 bytes
Variable Size             671092000 bytes
Database Buffers         1459617792 bytes
Redo Buffers                7979008 bytes
Database mounted.
SQL> alter database archivelog;

Database altered.

SQL> alter database open;

Database altered.

 
SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            +DATA/arch/
Oldest online log sequence     136
Next log sequence to archive   137
Current log sequence           137

这里主库已经启用了归档

启用force logging

SQL> alter database force logging;

Database altered.

SQL> select force_logging from v$database;

FORCE_LOGGING
---------------------------------------
YES

SQL> select force_logging from v$database;

FORCE_LOGGING
---------------------------------------
YES

2.2 给备库创建密码文件
这里通过复制主库的密码文件来创建备库密码文件

[oracle@jytest1 dbs]$ scp oracle@10.10.10.190:/u01/app/oracle/product/12.2.0/db/dbs/orapwcs1 /u01/app/oracle/product/12.2.0/db/dbs/orapwcs2
The authenticity of host '10.10.10.190 (10.10.10.190)' can't be established.
ECDSA key fingerprint is 7f:1f:9a:0f:8b:d1:e0:17:32:08:12:73:d8:1d:9c:da.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.10.10.190' (ECDSA) to the list of known hosts.
oracle@10.10.10.190's password: 
orapwcs1                                                                                                                                                                                                  100% 3584     3.5KB/s   00:00    
[oracle@jytest2 dbs]$ ls -lrt


[oracle@jytest2 dbs]$ scp oracle@10.10.10.190:/u01/app/oracle/product/12.2.0/db/dbs/orapwcs1 /u01/app/oracle/product/12.2.0/db/dbs/orapwcs2
The authenticity of host '10.10.10.190 (10.10.10.190)' can't be established.
ECDSA key fingerprint is 7f:1f:9a:0f:8b:d1:e0:17:32:08:12:73:d8:1d:9c:da.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.10.10.190' (ECDSA) to the list of known hosts.
oracle@10.10.10.190's password: 
orapwcs1                                                                                                                                                                                                  100% 3584     3.5KB/s   00:00    
[oracle@jytest2 dbs]$ ls -lrt

2.3.给备库创建参数文件
使用主库的参数文件进行创建

SQL> create pfile='/u01/app/oracle/product/12.2.0/db/dbs/initcs.ora' from spfile='+DATA/CS/PARAMETERFILE/spfile.287.970602765';

File created.

[oracle@jytest1 dbs]$ cat initcs1_temp.ora

*.audit_file_dest='/u01/app/oracle/admin/cs/adump'
*.audit_trail='db'
*.cluster_database=true
*.compatible='12.2.0'
*.control_files='+FRA/CS_DG/CONTROLFILE/control01.ctl'
*.db_block_size=8192
*.db_name='cs'
*.db_unique_name='cs_dg'
*.db_create_file_dest='+FRA'
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=csXDB)'
*.enable_pluggable_database=true
family:dw_helper.instance_mode='read-only'
log_archive_config='DG_CONFIG=(cs,cs_dg)'
cs2.instance_number=2
cs1.instance_number=1 
*.log_archive_format='%t_%s_%r.dbf'
*.log_archive_max_processes=30
*.log_archive_dest_1='LOCATION=+test/arch/cs_dg/ valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=cs_dg'
*.log_archive_dest_2='service=cs LGWR ASYNC valid_for=(online_logfiles,primary_role) db_unique_name=cs ' 
*.nls_language='AMERICAN'
*.nls_territory='AMERICA'
*.open_cursors=300
*.pga_aggregate_target=1g
*.processes=320
*.remote_login_passwordfile='exclusive'
*.sga_target=2g
cs2.thread=2
cs1.thread=1
cs2.undo_tablespace='UNDOTBS2'
cs1.undo_tablespace='UNDOTBS1'
*.db_file_name_convert= '+DATA/CS/','+FRA/CS_DG/'
*.fal_client='cs_dg'
*.fal_server='cs'
*.log_file_name_convert= '+DATA/CS/','+FRA/CS_DG/'
*.standby_file_management='auto'
SQL> create spfile='+fra/cs/parameterfile/spfilecs.ora' from pfile='/u01/app/oracle/product/12.2.0/db/dbs/initcs1_temp.ora';

File created.

[oracle@jytest1 dbs]$ vi initcs1.ora
spfile='+fra/cs/parameterfile/spfilecs.ora'

[oracle@jytest2 dbs]$ vi initcs2.ora
spfile='+fra/cs/parameterfile/spfilecs.ora'

2.4为主库和备库配置监听
主库是rac已经配置了监听,备库也是RAC需要对其配置监听
备库:节点1

[grid@jytest1 admin]$ vi listener.ora
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))))                # line added by Agent
MGMTLSNR=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=MGMTLSNR))))            # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))))                # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ASMNET1LSNR_ASM=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM))))              # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_ASMNET1LSNR_ASM=ON               # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_ASMNET1LSNR_ASM=SUBNET         # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=OFF             # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER=SUBNET                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3=OFF             # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_MGMTLSNR=ON              # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_MGMTLSNR=SUBNET                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2=OFF             # line added by Agent

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = cs2)
      (ORACLE_HOME =/u01/app/oracle/product/12.2.0/db)
      (GLOBAL_DBNAME=cs_dg)
    )
  )

备库:节点2

[grid@jytest2 admin]$ vi listener.ora
LISTENER_SCAN2=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2))))                # line added by Agent
MGMTLSNR=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=MGMTLSNR))))            # line added by Agent
LISTENER_SCAN3=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3))))                # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER))))            # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1))))                # line added by Agent
ASMNET1LSNR_ASM=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=ASMNET1LSNR_ASM))))              # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_ASMNET1LSNR_ASM=ON               # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_ASMNET1LSNR_ASM=SUBNET         # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN1=OFF             # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON              # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER=SUBNET                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN3=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN3=OFF             # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_MGMTLSNR=ON              # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_MGMTLSNR=SUBNET                # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN2=ON                # line added by Agent
VALID_NODE_CHECKING_REGISTRATION_LISTENER_SCAN2=OFF             # line added by Agent

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = cs2)
      (ORACLE_HOME =/u01/app/oracle/product/12.2.0/db)
      (GLOBAL_DBNAME=cs_dg)
    )
  )
[grid@jytest1 admin]$ srvctl stop listener -n jytest1
[grid@jytest1 admin]$ srvctl stop listener -n jytest2
[grid@jytest1 admin]$ srvctl start listener -n jytest1
[grid@jytest1 admin]$ srvctl start listener -n jytest2

[grid@jytest1 admin]$ lsnrctl status

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 14-MAY-2018 22:33:06

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                14-MAY-2018 22:32:08
Uptime                    0 days 0 hr. 0 min. 57 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/product/12.2.0/crs/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/jytest1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.171)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.175)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=jytest1.jydba.net)(PORT=5500))(Security=(my_wallet_directory=/u01/app/oracle/product/12.2.0/db/admin/jy/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_CRS" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_DATA" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_FRA" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "+ASM_TEST" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "5f9ac6865e87549fe053ab828a0ade94" has 1 instance(s).
  Instance "jy1", status READY, has 1 handler(s) for this service...
Service "cs_dg" has 1 instance(s).
  Instance "cs1", status UNKNOWN, has 1 handler(s) for this service...
Service "jy" has 1 instance(s).
  Instance "jy1", status READY, has 1 handler(s) for this service...
Service "jyXDB" has 1 instance(s).
  Instance "jy1", status READY, has 1 handler(s) for this service...
Service "jy_srv" has 1 instance(s).
  Instance "jy1", status READY, has 1 handler(s) for this service...
Service "jypdb" has 1 instance(s).
  Instance "jy1", status READY, has 1 handler(s) for this service...
The command completed successfully



[grid@jytest2 ~]$ lsnrctl status

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 14-MAY-2018 22:33:10

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                14-MAY-2018 22:32:16
Uptime                    0 days 0 hr. 0 min. 54 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/product/12.2.0/crs/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/jytest2/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.172)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.176)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=jytest2.jydba.net)(PORT=5500))(Security=(my_wallet_directory=/u01/app/oracle/product/12.2.0/db/admin/jy/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "+ASM_CRS" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "+ASM_DATA" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "+ASM_FRA" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "+ASM_TEST" has 1 instance(s).
  Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "5f9ac6865e87549fe053ab828a0ade94" has 1 instance(s).
  Instance "jy2", status READY, has 1 handler(s) for this service...
Service "cs_dg" has 1 instance(s).
  Instance "cs2", status UNKNOWN, has 1 handler(s) for this service...
Service "jy" has 1 instance(s).
  Instance "jy2", status READY, has 1 handler(s) for this service...
Service "jyXDB" has 1 instance(s).
  Instance "jy2", status READY, has 1 handler(s) for this service...
Service "jypdb" has 1 instance(s).
  Instance "jy2", status READY, has 1 handler(s) for this service...
The command completed successfully

2.5为主库和备库创建Oracle Net服务名
主库:节点1

[oracle@cs1 admin]$ vi tnsnames.ora 
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/12.2.0/db/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.
CS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST =10.10.10.149)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = cs)
    )
  )
CS_DG =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST =10.10.10.175)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = cs_dg)
      (UR=A)
    )
  )

主库:节点2

[oracle@cs2 admin]$ vi  tnsnames.ora 
# tnsnames.ora Network Configuration File: /u01/app/oracle/product/12.2.0/db/network/admin/tnsnames.ora
# Generated by Oracle configuration tools.

CS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST =10.10.10.134)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = cs)
    )
  )

CS_DG =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST =10.10.10.176)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = cs_dg)
      (UR=A)
    )
  )

备库:节点1

[oracle@jytest1 admin]$ vi tnsnames.ora 
# tnsnames.ora.jytest1 Network Configuration File: /u01/app/oracle/product/12.2.0/db/network/admin/tnsnames.ora.jytest1
# Generated by Oracle configuration tools.
CS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST =10.10.10.149)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = cs)
    )
  )

备库:节点2

[oracle@jytest2 admin]$ vi tnsnames.ora 
# tnsnames.ora.jytest2 Network Configuration File: /u01/app/oracle/product/12.2.0/db/network/admin/tnsnames.ora.jytest2
# Generated by Oracle configuration tools.
CS =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST =10.10.10.134)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = cs)
    )
  )

使用备份创建备库

[oracle@jytest1 admin]$ export ORACLE_SID=cs1
[oracle@jytest1 admin]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Tue May 15 00:06:43 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.

Total System Global Area 2147483648 bytes
Fixed Size                  8794848 bytes
Variable Size             721423648 bytes
Database Buffers         1409286144 bytes
Redo Buffers                7979008 bytes
[oracle@cs1 admin]$ rman target sys/abcd@cs auxiliary sys/abcd@cs_dg

Recovery Manager: Release 12.2.0.1.0 - Production on Mon May 14 19:43:21 2018

Copyright (c) 1982, 2017, Oracle and/or its affiliates.  All rights reserved.

connected to target database: CS (DBID=1386528187)
connected to auxiliary database: CS (not mounted)

RMAN> duplicate target database for standby from active database nofilenamecheck;

Starting Duplicate Db at 14-MAY-18
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=197 instance=cs1 device type=DISK

contents of Memory Script:
{
   backup as copy reuse
   targetfile  '+DATA/CS/PASSWORD/pwdcs.271.970601731' auxiliary format 
 '/u01/app/oracle/product/12.2.0/db/dbs/orapwcs1'   ;
}
executing Memory Script

Starting backup at 14-MAY-18
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=291 instance=cs1 device type=DISK
Finished backup at 14-MAY-18

contents of Memory Script:
{
   restore clone from service  'cs' standby controlfile;
}
executing Memory Script

Starting restore at 14-MAY-18
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cs
channel ORA_AUX_DISK_1: restoring control file
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:03
output file name=+FRA/CS_DG/CONTROLFILE/control01.ctl
Finished restore at 14-MAY-18

contents of Memory Script:
{
   sql clone 'alter database mount standby database';
}
executing Memory Script

sql statement: alter database mount standby database
RMAN-05529: warning: DB_FILE_NAME_CONVERT resulted in invalid ASM names; names changed to disk group only.

contents of Memory Script:
{
   set newname for tempfile  1 to 
 "+FRA";
   set newname for tempfile  2 to 
 "+FRA";
   switch clone tempfile all;
   set newname for datafile  1 to 
 "+FRA";
   set newname for datafile  3 to 
 "+FRA";
   set newname for datafile  4 to 
 "+FRA";
   set newname for datafile  5 to 
 "+FRA";
   set newname for datafile  6 to 
 "+FRA";
   set newname for datafile  7 to 
 "+FRA";
   set newname for datafile  8 to 
 "+FRA";
   set newname for datafile  9 to 
 "+FRA";
   restore
   from  nonsparse   from service 
 'cs'   clone database
   ;
   sql 'alter system archive log current';
}
executing Memory Script

executing command: SET NEWNAME

executing command: SET NEWNAME

renamed tempfile 1 to +FRA in control file
renamed tempfile 2 to +FRA in control file

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

executing command: SET NEWNAME

Starting restore at 14-MAY-18
using channel ORA_AUX_DISK_1

channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cs
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00001 to +FRA
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:16
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cs
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00003 to +FRA
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:45
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cs
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00004 to +FRA
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:04
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cs
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00005 to +FRA
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:08
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cs
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00006 to +FRA
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:07
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cs
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00007 to +FRA
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:02
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cs
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00008 to +FRA
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:04
channel ORA_AUX_DISK_1: starting datafile backup set restore
channel ORA_AUX_DISK_1: using network backup set from service cs
channel ORA_AUX_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_AUX_DISK_1: restoring datafile 00009 to +FRA
channel ORA_AUX_DISK_1: restore complete, elapsed time: 00:00:03
Finished restore at 14-MAY-18

sql statement: alter system archive log current

contents of Memory Script:
{
   switch clone datafile all;
}
executing Memory Script

datafile 1 switched to datafile copy
input datafile copy RECID=12 STAMP=976150821 file name=+FRA/CS_DG/DATAFILE/system.342.976150731
datafile 3 switched to datafile copy
input datafile copy RECID=13 STAMP=976150821 file name=+FRA/CS_DG/DATAFILE/sysaux.341.976150747
datafile 4 switched to datafile copy
input datafile copy RECID=14 STAMP=976150821 file name=+FRA/CS_DG/DATAFILE/undotbs1.340.976150793
datafile 5 switched to datafile copy
input datafile copy RECID=15 STAMP=976150821 file name=+FRA/CS_DG/67369AA1C9AA3E71E053BE828A0A8262/DATAFILE/system.347.976150795
datafile 6 switched to datafile copy
input datafile copy RECID=16 STAMP=976150822 file name=+FRA/CS_DG/67369AA1C9AA3E71E053BE828A0A8262/DATAFILE/sysaux.351.976150803
datafile 7 switched to datafile copy
input datafile copy RECID=17 STAMP=976150822 file name=+FRA/CS_DG/DATAFILE/users.350.976150811
datafile 8 switched to datafile copy
input datafile copy RECID=18 STAMP=976150823 file name=+FRA/CS_DG/67369AA1C9AA3E71E053BE828A0A8262/DATAFILE/undotbs1.348.976150813
datafile 9 switched to datafile copy
input datafile copy RECID=19 STAMP=976150823 file name=+FRA/CS_DG/DATAFILE/undotbs2.345.976150817
Finished Duplicate Db at 14-MAY-18
[oracle@jytest1 admin]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Tue May 15 01:05:35 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> select instance_name,status from v$instance;

INSTANCE_NAME    STATUS
---------------- ------------
cs1              MOUNTED


SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
+FRA/CS_DG/DATAFILE/system.342.976150731
+FRA/CS_DG/DATAFILE/sysaux.341.976150747
+FRA/CS_DG/DATAFILE/undotbs1.340.976150793
+FRA/CS_DG/67369AA1C9AA3E71E053BE828A0A8262/DATAFILE/system.347.976150795
+FRA/CS_DG/67369AA1C9AA3E71E053BE828A0A8262/DATAFILE/sysaux.351.976150803
+FRA/CS_DG/DATAFILE/users.350.976150811
+FRA/CS_DG/67369AA1C9AA3E71E053BE828A0A8262/DATAFILE/undotbs1.348.976150813
+FRA/CS_DG/DATAFILE/undotbs2.345.976150817

8 rows selected.

对物理备库创建备重做日志文件,查询主库的联机重做日志文件,备重做日志文件的大小应该与主库联机重做日志文件的大小相同,备重做日志文件组的数量应该比主库联机重做日志文件组多一组,计算公式为(maximum # of logfiles +1) * maximum # of threads SQL> select member from v$logfile;

MEMBER
——————————————————————————————————————————————————————————————————–
+DATA/CS/ONLINELOG/group_2.277.970601985
+DATA/CS/ONLINELOG/group_1.278.970601985
+DATA/CS/ONLINELOG/group_3.285.970602759
+DATA/CS/ONLINELOG/group_4.286.970602761

SQL> select group#,thread#,bytes/1024/1024 from v$log;

GROUP# THREAD# BYTES/1024/1024
———- ———- —————
1 1 200
2 1 200
3 2 200
4 2 200

SQL> select member from v$logfile;

MEMBER
——————————————————————————————————————————————————————————————————–
+FRA/CS_DG/ONLINELOG/group_2.346.976150835
+FRA/CS_DG/ONLINELOG/group_1.349.976150825
+FRA/CS_DG/ONLINELOG/group_3.344.976150843
+FRA/CS_DG/ONLINELOG/group_4.352.976150855

SQL> select group#,thread#,bytes/1024/1024 from v$log;

GROUP# THREAD# BYTES/1024/1024
———- ———- —————
1 1 200
2 1 200
3 2 200
4 2 200

SQL> alter database add standby logfile thread 1 group 5(‘+FRA/CS_DG/ONLINELOG/redo05.log’) size 200M;

Database altered.

SQL> alter database add standby logfile thread 1 group 6(‘+FRA/CS_DG/ONLINELOG/redo06.log’) size 200M;

Database altered.

SQL> alter database add standby logfile thread 1 group 7(‘+FRA/CS_DG/ONLINELOG/redo07.log’) size 200M;

Database altered.

SQL> alter database add standby logfile thread 2 group 8(‘+FRA/CS_DG/ONLINELOG/redo08.log’) size 200M;

Database altered.

SQL> alter database add standby logfile thread 2 group 9(‘+FRA/CS_DG/ONLINELOG/redo09.log’) size 200M;

Database altered.

SQL> alter database add standby logfile thread 2 group 10(‘+FRA/CS_DG/ONLINELOG/redo10.log’) size 200M;

Database altered.

设置主库相关初始化参数

log_archive_config='DG_CONFIG=(cs,cs_dg)'
log_archive_dest_1='LOCATION=+data/arch/ valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=cs'
log_archive_dest_2='SERVICE=cs_dg LGWR ASYNC valid_for=(ONLINE_LOGFILES,PRIMARY_ROLE) db_unique_name=cs_dg'
log_archive_dest_state_1=enable
log_archive_dest_state_2=enable
remote_login_passwordfile=exclusive
log_archive_format= %t_%s_%r.dbf
log_archive_max_processes=30

主库以备库角色运行时需要额外设置的参数。这些参数当主库被转换为备库角色运行时生效:

fal_server='cs_dg'
fal_client='cs'
db_file_name_convert='+FRA/CS_DG/', '+DATA/CS/'
log_file_name_convert='+FRA/CS_DG/', '+DATA/CS/'
standby_file_management='auto'
SQL> alter system set log_archive_dest_1='LOCATION=+data/arch/ valid_for=(ALL_LOGFILES,ALL_ROLES) db_unique_name=cs' scope=both sid='*';

System altered.

SQL> alter system set log_archive_dest_2='SERVICE=cs_dg LGWR ASYNC valid_for=(ONLINE_LOGFILES,PRIMARY_ROLE) db_unique_name=cs_dg' scope=both sid='*';

System altered.

SQL> alter system set log_archive_dest_state_1=enable scope=both sid='*';

System altered.

SQL> alter system set log_archive_dest_state_2=enable scope=both sid='*';

System altered.

SQL> alter system set log_archive_max_processes=30 scope=both sid='*';

System altered.

SQL> alter system set log_archive_config='DG_CONFIG=(cs,cs_dg)' scope=both sid='*';

System altered.

SQL> alter system set fal_server='cs_dg' scope=both sid='*';

System altered.

SQL> alter system set fal_client='cs' scope=both sid='*';

System altered.

SQL> alter system set db_file_name_convert='+FRA/CS_DG/', '+DATA/CS/' scope=spfile  sid='*';

System altered.

SQL> alter system set log_file_name_convert='+FRA/CS_DG/', '+DATA/CS/' scope=spfile sid='*';

System altered.

SQL> alter system set standby_file_management='auto' scope=both sid='*';

System altered.

给主库创建备重做日志文件,备重做日志文件组比联机重做日志文件多一组,主库是RAC数据库,在创建备重做日志文件时需要指定thread号:

SQL> alter database add standby logfile thread 1 group 5('+DATA/CS/ONLINELOG/redo05.log') size 200M;

Database altered.

SQL> alter database add standby logfile thread 1 group 6('+DATA/CS/ONLINELOG/redo06.log') size 200M;

Database altered.

SQL> alter database add standby logfile thread 1 group 7('+DATA/CS/ONLINELOG/redo07.log') size 200M;

Database altered.

SQL> alter database add standby logfile thread 2 group 8('+DATA/CS/ONLINELOG/redo08.log') size 200M;

Database altered.

SQL> alter database add standby logfile thread 2 group 9('+DATA/CS/ONLINELOG/redo09.log') size 200M;

Database altered.

SQL> alter database add standby logfile thread 2 group 10('+DATA/CS/ONLINELOG/redo10.log') size 200M;

Database altered.

Oracle用户使用srvctl在备库的另一节点注册备库和备库实例

[oracle@jytest1 dbs]$ srvctl add database -db cs_dg -oraclehome /u01/app/oracle/product/12.2.0/db/ -dbtype RAC -spfile +FRA/CS_DG/PARAMETERFILE/spfilecs.ora -pwfile +FRA/CS_DG/PASSWORD/pwdcs  -role physical_standby -startoption open -stopoption immediate -dbname cs -policy automatic -diskgroup data,fra,test

[oracle@jytest1 dbs]$ srvctl config database -db cs_dg
Database unique name: cs_dg
Database name: cs
Oracle home: /u01/app/oracle/product/12.2.0/db/
Oracle user: oracle
Spfile: +FRA/CS_DG/PARAMETERFILE/spfilecs.ora
Password file: +FRA/CS_DG/PASSWORD/pwdcs
Domain: 
Start options: open
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools: 
Disk Groups: DATA,FRA,TEST
Mount point paths: 
Services: 
Type: RAC
Start concurrency: 
Stop concurrency: 
OSDBA group: dba
OSOPER group: oper
Database instances: 
Configured nodes: 
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services: 
Database is administrator managed

添加实例

[oracle@jytest1 dbs]$ srvctl add instance -db cs_dg -instance cs1 -node jytest1
[oracle@jytest1 dbs]$ srvctl add instance -db cs_dg -instance cs2 -node jytest2
[oracle@jytest1 dbs]$ srvctl config database -db cs_dg
Database unique name: cs_dg
Database name: cs
Oracle home: /u01/app/oracle/product/12.2.0/db/
Oracle user: oracle
Spfile: +FRA/CS_DG/PARAMETERFILE/spfilecs.ora
Password file: +FRA/CS_DG/PASSWORD/pwdcs
Domain: 
Start options: open
Stop options: immediate
Database role: PHYSICAL_STANDBY
Management policy: AUTOMATIC
Server pools: 
Disk Groups: DATA,FRA,TEST
Mount point paths: 
Services: 
Type: RAC
Start concurrency: 
Stop concurrency: 
OSDBA group: dba
OSOPER group: oper
Database instances: cs1,cs2
Configured nodes: jytest1,jytest2
CSS critical: no
CPU count: 0
Memory target: 0
Maximum memory: 0
Default network number for database services: 
Database is administrator managed

[grid@jytest1 ~]$ srvctl start database -db cs_dg

[grid@jytest1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       jytest1                  STABLE
               ONLINE  ONLINE       jytest2                  STABLE
ora.CRS.dg
               ONLINE  ONLINE       jytest1                  STABLE
               ONLINE  ONLINE       jytest2                  STABLE
ora.DATA.dg
               ONLINE  ONLINE       jytest1                  STABLE
               ONLINE  ONLINE       jytest2                  STABLE
ora.FRA.dg
               ONLINE  ONLINE       jytest1                  STABLE
               ONLINE  ONLINE       jytest2                  STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       jytest1                  STABLE
               ONLINE  ONLINE       jytest2                  STABLE
ora.TEST.dg
               ONLINE  ONLINE       jytest1                  STABLE
               ONLINE  ONLINE       jytest2                  STABLE
ora.chad
               ONLINE  ONLINE       jytest1                  STABLE
               ONLINE  ONLINE       jytest2                  STABLE
ora.net1.network
               ONLINE  ONLINE       jytest1                  STABLE
               ONLINE  ONLINE       jytest2                  STABLE
ora.ons
               ONLINE  ONLINE       jytest1                  STABLE
               ONLINE  ONLINE       jytest2                  STABLE
ora.proxy_advm
               OFFLINE OFFLINE      jytest1                  STABLE
               OFFLINE OFFLINE      jytest2                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       jytest2                  STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       jytest1                  STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       jytest1                  STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       jytest1                  169.254.123.145 88.8
                                                             8.88.1,STABLE
ora.asm
      1        ONLINE  ONLINE       jytest1                  Started,STABLE
      2        ONLINE  ONLINE       jytest2                  Started,STABLE
      3        ONLINE  OFFLINE                               STABLE
ora.cs_dg.db
      1        ONLINE  ONLINE       jytest1                  Open,Readonly,HOME=/
                                                             u01/app/oracle/produ
                                                             ct/12.2.0/db/,STABLE
      2        ONLINE  ONLINE       jytest2                  Open,Readonly,HOME=/
                                                             u01/app/oracle/produ
                                                             ct/12.2.0/db/,STABLE
ora.cvu
      1        ONLINE  ONLINE       jytest1                  STABLE

ora.jy.jy_srv.svc
      1        ONLINE  ONLINE       jytest1                  STABLE
ora.jytest1.vip
      1        ONLINE  ONLINE       jytest1                  STABLE
ora.jytest2.vip
      1        ONLINE  ONLINE       jytest2                  STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       jytest1                  Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       jytest1                  STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       jytest2                  STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       jytest1                  STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       jytest1                  STABLE
--------------------------------------------------------------------------------
[oracle@jytest1 ~]$ export ORACLE_SID=cs1
[oracle@jytest1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Tue May 15 23:02:41 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> set line 120
SQL> select name, database_role, open_mode from gv$database;

NAME      DATABASE_ROLE    OPEN_MODE
--------- ---------------- --------------------
CS        PHYSICAL STANDBY READ ONLY
CS        PHYSICAL STANDBY READ ONLY

[oracle@jytest2 dbs]$ export ORACLE_SID=cs2
[oracle@jytest2 dbs]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Tue May 15 23:02:47 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> set line 120
SQL> select name, database_role, open_mode from gv$database;

NAME      DATABASE_ROLE    OPEN_MODE
--------- ---------------- --------------------
CS        PHYSICAL STANDBY READ ONLY
CS        PHYSICAL STANDBY READ ONLY

在备库的单个节点上执行实时重做应用

SQL> alter database recover managed standby database using current logfile disconnect from session;

Database altered.

alert日志信息如下

Media Recovery Log +TEST/arch/cs_dg/2_129_970601983.dbf
2018-05-15T23:04:44.911711+08:00
Completed: alter database recover managed standby database using current logfile disconnect from session
2018-05-15T23:04:45.116759+08:00
Media Recovery Log +TEST/arch/cs_dg/1_145_970601983.dbf
2018-05-15T23:04:45.676517+08:00
Media Recovery Log +TEST/arch/cs_dg/2_130_970601983.dbf
2018-05-15T23:05:31.642093+08:00
Media Recovery Log +TEST/arch/cs_dg/2_131_970601983.dbf

在主库创建表空间test

SQL> create tablespace test;

Tablespace created.

SQL> select tablespace_name from dba_tablespaces;

TABLESPACE_NAME
------------------------------
SYSTEM
SYSAUX
UNDOTBS1
TEMP
USERS
UNDOTBS2
TEST

7 rows selected.


SQL> select con_id,ts#,name from v$tablespace;

    CON_ID        TS# NAME
---------- ---------- ------------------------------
         1          1 SYSAUX
         1          0 SYSTEM
         1          2 UNDOTBS1
         1          4 USERS
         1          3 TEMP
         2          0 SYSTEM
         2          1 SYSAUX
         2          2 UNDOTBS1
         2          3 TEMP
         1          5 UNDOTBS2
         1          6 TEST

SQL> select ts#,name from v$datafile where ts#=6 and con_id=1;

       TS#  NAME
    ----------  --------------------------------------------------------------------------------
         6  +DATA/CS/DATAFILE/test.326.976211663

[oracle@jytest1 trace]$ tail -f alert_cs1.log 
2018-05-15T23:04:44.736977+08:00
NOTE: dependency between database cs_dg and diskgroup resource ora.TEST.dg is established
2018-05-15T23:04:44.842580+08:00
Media Recovery Log +TEST/arch/cs_dg/2_129_970601983.dbf
2018-05-15T23:04:44.911711+08:00
Completed: alter database recover managed standby database using current logfile disconnect from session
2018-05-15T23:04:45.116759+08:00
Media Recovery Log +TEST/arch/cs_dg/1_145_970601983.dbf
2018-05-15T23:04:45.676517+08:00
Media Recovery Log +TEST/arch/cs_dg/2_130_970601983.dbf
2018-05-15T23:05:31.642093+08:00
Media Recovery Log +TEST/arch/cs_dg/2_131_970601983.dbf
2018-05-15T23:06:11.885256+08:00
Media Recovery Log +TEST/arch/cs_dg/1_146_970601983.dbf
2018-05-15T23:06:26.490187+08:00
Media Recovery Log +TEST/arch/cs_dg/2_132_970601983.dbf
2018-05-15T23:06:36.761337+08:00
Media Recovery Log +TEST/arch/cs_dg/2_133_970601983.dbf
2018-05-15T23:06:38.014959+08:00
Media Recovery Log +TEST/arch/cs_dg/1_147_970601983.dbf
2018-05-15T23:06:38.932380+08:00
Media Recovery Log +TEST/arch/cs_dg/1_148_970601983.dbf
2018-05-15T23:06:40.372178+08:00
Media Recovery Log +TEST/arch/cs_dg/2_134_970601983.dbf
2018-05-15T23:06:40.994801+08:00
Media Recovery Log +TEST/arch/cs_dg/1_149_970601983.dbf
2018-05-15T23:06:41.656032+08:00
Media Recovery Log +TEST/arch/cs_dg/2_135_970601983.dbf
2018-05-15T23:06:47.456319+08:00
Media Recovery Waiting for thread 2 sequence 136 (in transit)
2018-05-15T23:06:47.474190+08:00
Recovery of Online Redo Log: Thread 2 Group 9 Seq 136 Reading mem 0
  Mem# 0: +FRA/CS_DG/ONLINELOG/redo09.log
2018-05-15T23:06:52.286510+08:00
Media Recovery Waiting for thread 1 sequence 150 (in transit)
2018-05-15T23:06:52.338688+08:00
Recovery of Online Redo Log: Thread 1 Group 6 Seq 150 Reading mem 0
  Mem# 0: +FRA/CS_DG/ONLINELOG/redo06.log
2018-05-15T23:08:48.743309+08:00
Successfully added datafile 10 to media recovery
Datafile #10: '+FRA/CS_DG/DATAFILE/test.327.976230527'

从上面信息可以看到在备库创建的表空间test的数据文件名,下面从备库进行查询来验证

SQL>  select con_id,ts#,name from v$tablespace;

    CON_ID        TS# NAME
---------- ---------- ------------------------------
         1          1 SYSAUX
         1          0 SYSTEM
         1          2 UNDOTBS1
         1          4 USERS
         1          3 TEMP
         2          0 SYSTEM
         2          1 SYSAUX
         2          2 UNDOTBS1
         2          3 TEMP
         1          5 UNDOTBS2
         1          6 TEST

11 rows selected.
SQL> select ts#,name from v$datafile where ts#=6 and con_id=1;

       TS#  NAME
----------  ------------------------------------------------------------------------------------------------------------------------
         6  +FRA/CS_DG/DATAFILE/test.327.976230527

验证主库

SQL> create user c##test identified by "test" default tablespace users temporary tablespace temp;

User created.

SQL> grant dba,connect,resource to c##test;

Grant succeeded.

SQL> create table c##test.t1 as select * from dba_objects where 1=2;

Table created.

备库

SQL> desc c##test.t1
 Name                                                              Null?    Type
 ----------------------------------------------------------------- -------- --------------------------------------------
 OWNER                                                                      VARCHAR2(128)
 OBJECT_NAME                                                                VARCHAR2(128)
 SUBOBJECT_NAME                                                             VARCHAR2(128)
 OBJECT_ID                                                                  NUMBER
 DATA_OBJECT_ID                                                             NUMBER
 OBJECT_TYPE                                                                VARCHAR2(23)
 CREATED                                                                    DATE
 LAST_DDL_TIME                                                              DATE
 TIMESTAMP                                                                  VARCHAR2(19)
 STATUS                                                                     VARCHAR2(7)
 TEMPORARY                                                                  VARCHAR2(1)
 GENERATED                                                                  VARCHAR2(1)
 SECONDARY                                                                  VARCHAR2(1)
 NAMESPACE                                                                  NUMBER
 EDITION_NAME                                                               VARCHAR2(128)
 SHARING                                                                    VARCHAR2(18)
 EDITIONABLE                                                                VARCHAR2(1)
 ORACLE_MAINTAINED                                                          VARCHAR2(1)
 APPLICATION                                                                VARCHAR2(1)
 DEFAULT_COLLATION                                                          VARCHAR2(100)
 DUPLICATED                                                                 VARCHAR2(1)
 SHARDED                                                                    VARCHAR2(1)
 CREATED_APPID                                                              NUMBER
 CREATED_VSNID                                                              NUMBER
 MODIFIED_APPID                                                             NUMBER
 MODIFIED_VSNID                                                             NUMBER

主库:节点1

SQL> insert into c##test.t1 select * from dba_objects;

73390 rows created.

SQL> commit;

Commit complete.

SQL> select count(*) from c##test.t1;

  COUNT(*)
----------
     73390

备库:节点1

SQL> select count(*) from c##test.t1;

  COUNT(*)
----------
     73390

主库:节点2

SQL> insert into c##test.t1 select * from c##test.t1;

73390 rows created.

SQL> commit;

Commit complete.

SQL> select count(*) from c##test.t1;

  COUNT(*)
----------
    146780

备库: 节点2

SQL> select count(*) from c##test.t1;

  COUNT(*)
----------
    146780

主库:

SQL> truncate table c##test.t1;

Table truncated.

SQL> select count(*) from c##test.t1;

  COUNT(*)
----------
         0

备库:

SQL> select count(*) from c##test.t1;

  COUNT(*)
----------
         0

验证物理备库是否执行正确
在创建物理备库后并且设置重做传输服务,可能想要验证主库的数据库修改是否会成功的传输到备库。对于备库可以查询v$managed_standby视图来验证重做是否被从主库传输到备库并应用。

[oracle@jytest1 ~]$ export ORACLE_SID=cs1
[oracle@jytest1 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.2.0.1.0 Production on Wed May 16 22:41:10 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production

SQL> select client_process,process,thread#,sequence#,status from v$managed_standby where client_process='LGWR' or process='MRP0';

CLIENT_P PROCESS      THREAD#  SEQUENCE# STATUS
-------- --------- ---------- ---------- ------------
LGWR     RFS                1        151 IDLE
N/A      MRP0               2        139 APPLYING_LOG

上面的查询对于使用CLIENT_PROCESS为LGWR的主库会显示一行记录,它指示重做传输工作正常并且主重做线程将会被发送到备库。 如果主库是RAC数据库,那么对于使用CLIENT_PROCESS为LGWR的当前活动的每个主库实例都会显示一行记录。上面的查询对于MRP也行显示一行。如果MRP的状态显示为APPLYING_LOG并且SEQUENCE#等于主库当前正被发送的日志
序列号,那么备库已经解决了所有的日志差异并且当前处于实时应用日志模式。

查询主库当前正被发送日志的序列号为139与上面的MRP进程所显示的sequence#(139)相同

SQL> select group#,thread#,sequence#,status from v$log;

    GROUP#    THREAD#  SEQUENCE# STATUS
---------- ---------- ---------- ----------------
         1          1        151 CURRENT
         2          1        150 INACTIVE
         3          2        139 CURRENT
         4          2        138 INACTIVE

注意MRP进程可能显示的sequence#比主库当前被发送的日志序列号小,那么这就表示正在应用的归档重做日志文件与发送的日志文件之间存在差异并且它并没有赶上。一旦所有差异被解决,相同的查询将显示MRP正在应用当前sequence#。

修改/dev/shm大小造成Oracle 12c集群启动故障

由于维护人员修改Oracle Linux 7中的/dev/shm大小造成其大小小于Oracle实例的MEMORY_TARGET或者SGA_TARGET而导致集群不能启动(CRS-4535,CRS-4000)

[grid@jtp1 ~]$ crsctl stat res -t
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.

检查asm磁盘的权限是否问题,发现磁盘权限正常

[root@jtp3 ~]# ls -lrt /dev/asm*
brw-rw----. 1 grid oinstall 8, 128 Apr  3  2018 /dev/asmdisk07
brw-rw----. 1 grid oinstall 8,  48 Apr  3  2018 /dev/asmdisk02
brw-rw----. 1 grid oinstall 8,  96 Apr  3  2018 /dev/asmdisk05
brw-rw----. 1 grid oinstall 8, 112 Apr  3  2018 /dev/asmdisk06
brw-rw----. 1 grid oinstall 8,  64 Apr  3  2018 /dev/asmdisk03
brw-rw----. 1 grid oinstall 8,  80 Apr  3  2018 /dev/asmdisk04
brw-rw----. 1 grid oinstall 8,  32 Apr  3  2018 /dev/asmdisk01

重启crs

[root@jtp1 bin]# ./crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'jtp1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'jtp1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'jtp1'
CRS-2677: Stop of 'ora.mdnsd' on 'jtp1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'jtp1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'jtp1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'jtp1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'jtp1' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'jtp1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'jtp1'
CRS-2673: Attempting to stop 'ora.evmd' on 'jtp1'
CRS-2677: Stop of 'ora.ctssd' on 'jtp1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'jtp1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'jtp1'
CRS-2677: Stop of 'ora.cssd' on 'jtp1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'jtp1'
CRS-2673: Attempting to stop 'ora.driver.afd' on 'jtp1'
CRS-2677: Stop of 'ora.driver.afd' on 'jtp1' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'jtp1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'jtp1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
[root@jtp1 bin]# ./crsctl start crs
CRS-4123: Oracle High Availability Services has been started.

查看crs的alert.log发现磁盘组不能加载

[root@jtp1 ~]# tail -f /u01/app/grid/diag/crs/jtp1/crs/trace/alert.log
2018-04-02 18:30:21.227 [OHASD(8143)]CRS-8500: Oracle Clusterware OHASD process is starting with operating system process ID 8143
2018-04-02 18:30:21.230 [OHASD(8143)]CRS-0714: Oracle Clusterware Release 12.2.0.1.0.
2018-04-02 18:30:21.245 [OHASD(8143)]CRS-2112: The OLR service started on node jtp1.
2018-04-02 18:30:21.262 [OHASD(8143)]CRS-8017: location: /etc/oracle/lastgasp has 2 reboot advisory log files, 0 were announced and 0 errors occurred
2018-04-02 18:30:21.262 [OHASD(8143)]CRS-1301: Oracle High Availability Service started on node jtp1.
2018-04-02 18:30:21.567 [ORAROOTAGENT(8214)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 8214
2018-04-02 18:30:21.600 [CSSDAGENT(8231)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 8231
2018-04-02 18:30:21.607 [CSSDMONITOR(8241)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 8241
2018-04-02 18:30:21.620 [ORAAGENT(8225)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 8225
2018-04-02 18:30:22.146 [ORAAGENT(8316)]CRS-8500: Oracle Clusterware ORAAGENT process is starting with operating system process ID 8316
2018-04-02 18:30:22.211 [MDNSD(8335)]CRS-8500: Oracle Clusterware MDNSD process is starting with operating system process ID 8335
2018-04-02 18:30:22.215 [EVMD(8337)]CRS-8500: Oracle Clusterware EVMD process is starting with operating system process ID 8337
2018-04-02 18:30:23.259 [GPNPD(8369)]CRS-8500: Oracle Clusterware GPNPD process is starting with operating system process ID 8369
2018-04-02 18:30:24.275 [GPNPD(8369)]CRS-2328: GPNPD started on node jtp1.
2018-04-02 18:30:24.283 [GIPCD(8433)]CRS-8500: Oracle Clusterware GIPCD process is starting with operating system process ID 8433
2018-04-02 18:30:26.296 [CSSDMONITOR(8464)]CRS-8500: Oracle Clusterware CSSDMONITOR process is starting with operating system process ID 8464
2018-04-02 18:30:28.299 [CSSDAGENT(8482)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 8482
2018-04-02 18:30:28.496 [OCSSD(8497)]CRS-8500: Oracle Clusterware OCSSD process is starting with operating system process ID 8497
2018-04-02 18:30:29.538 [OCSSD(8497)]CRS-1713: CSSD daemon is started in hub mode
2018-04-02 18:30:36.015 [OCSSD(8497)]CRS-1707: Lease acquisition for node jtp1 number 1 completed
2018-04-02 18:30:37.087 [OCSSD(8497)]CRS-1605: CSSD voting file is online: AFD:CRS1; details in /u01/app/grid/diag/crs/jtp1/crs/trace/ocssd.trc.
2018-04-02 18:30:37.103 [OCSSD(8497)]CRS-1672: The number of voting files currently available 1 has fallen to the minimum number of voting files required 1.
2018-04-02 18:30:46.237 [OCSSD(8497)]CRS-1601: CSSD Reconfiguration complete. Active nodes are jtp1 .
2018-04-02 18:30:48.514 [OCTSSD(9302)]CRS-8500: Oracle Clusterware OCTSSD process is starting with operating system process ID 9302
2018-04-02 18:30:48.535 [OCSSD(8497)]CRS-1720: Cluster Synchronization Services daemon (CSSD) is ready for operation.
2018-04-02 18:30:50.626 [OCTSSD(9302)]CRS-2407: The new Cluster Time Synchronization Service reference node is host jtp1.
2018-04-02 18:30:50.627 [OCTSSD(9302)]CRS-2401: The Cluster Time Synchronization Service started on host jtp1.
2018-04-02 18:31:04.202 [ORAROOTAGENT(8214)]CRS-5019: All OCR locations are on ASM disk groups [CRS], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc".
2018-04-02 18:41:00.225 [ORAROOTAGENT(8214)]CRS-5818: Aborted command 'start' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:9:3} in /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc.
2018-04-02 18:41:03.757 [ORAROOTAGENT(8214)]CRS-5017: The resource action "ora.storage start" encountered the following error:
2018-04-02 18:41:03.757+Storage agent start action aborted. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc".
2018-04-02 18:41:03.760 [OHASD(8143)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.storage'. Details at (:CRSPE00221:) {0:9:3} in /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd.trc.
2018-04-02 18:42:09.921 [ORAROOTAGENT(8214)]CRS-5019: All OCR locations are on ASM disk groups [CRS], and none of these disk groups are mounted. Details are at "(:CLSN00140:)" in "/u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc".

检查跟踪文件,发现查询ASM_DISCOVERY_ADDRESS与ASM_DISCOVERY_ADDRESS属性时出现

[root@jtp1 ~]# more /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc
Trace file /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root.trc
Oracle Database 12c Clusterware Release 12.2.0.1.0 - Production Copyright 1996, 2016 Oracle. All rights reserved.

*** TRACE CONTINUED FROM FILE /u01/app/grid/diag/crs/jtp1/crs/trace/ohasd_orarootagent_root_93.trc ***

2018-04-02 18:42:09.165 : CSSCLNT:3554666240: clsssterm: terminating context (0x7f03c0229390)
2018-04-02 18:42:09.165 : default:3554666240: clsCredDomClose: Credctx deleted 0x7f03c0459470
2018-04-02 18:42:09.166 :    GPNP:3554666240: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode='remote'
2018-04-02 18:42:09.253 : CSSCLNT:3554666240: clsssinit: initialized context: (0x7f03c045c2c0) flags 0x115
2018-04-02 18:42:09.253 : CSSCLNT:3554666240: clsssterm: terminating context (0x7f03c045c2c0)
2018-04-02 18:42:09.254 :   CLSNS:3554666240: clsns_SetTraceLevel:trace level set to 1.
2018-04-02 18:42:09.254 :    GPNP:3554666240: clsgpnp_dbmsGetItem_profile: [at clsgpnp_dbms.c:399] Result: (0) CLSGPNP_OK. (:GPNP00401:)got ASM-Profile.Mode='remote'
2018-04-02 18:42:09.257 : default:3554666240: Inited LSF context: 0x7f03c04f0420
2018-04-02 18:42:09.260 : CLSCRED:3554666240: clsCredCommonInit: Inited singleton credctx.
2018-04-02 18:42:09.260 : CLSCRED:3554666240: (:CLSCRED0101:)clsCredDomInitRootDom: Using user given storage context for repository access.
2018-04-02 18:42:09.294 : USRTHRD:3554666240: {0:9:3} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS

2018-04-02 18:42:09.300 : USRTHRD:3554666240: {0:9:3} 8033 Error 4 querying length of attr ASM_DISCOVERY_ADDRESS

2018-04-02 18:42:09.356 : CLSCRED:3554666240: (:CLSCRED1079:)clsCredOcrKeyExists: Obj dom : SYSTEM.credentials.domains.root.ASM.Self.5c82286a084bcf37ffa014144074e5dd.root not found
2018-04-02 18:42:09.356 : USRTHRD:3554666240: {0:9:3} 7755 Error 4 opening dom root in 0x7f03c064c980

检查ASM的alert.log 发现/dev/shm大小小于MEMORY_TARGET大小,并且给出了/dev/shm应该被设置的最小值

[root@jtp1 ~]# tail -f /u01/app/grid/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log
WARNING: ASM does not support ipclw. Switching to skgxp
WARNING: ASM does not support ipclw. Switching to skgxp
WARNING: ASM does not support ipclw. Switching to skgxp
* instance_number obtained from CSS = 1, checking for the existence of node 0...
* node 0 does not exist. instance_number = 1
Starting ORACLE instance (normal) (OS id: 9343)
2018-04-02T18:31:00.187055+08:00
CLI notifier numLatches:7 maxDescs:2301
2018-04-02T18:31:00.193961+08:00
WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to be mounted for at least 1140850688 bytes. /dev/shm is either not mounted or is mounted with available space less than this size. Please fix this so that MEMORY_TARGET can work as expected. Current available is 1073573888 and used is 167936 bytes. Ensure that the mount point is /dev/shm for this directory.

修改/dev/shm的大小可以通过修改/etc/fstab来实现,将/dev/shm的大小修改为12G

[root@jtp1 bin]# df -h
Filesystem           Size  Used Avail Use% Mounted on
/dev/mapper/ol-root   49G   42G  7.9G  85% /
devtmpfs              12G   28K   12G   1% /dev
tmpfs                1.0G  164K  1.0G   1% /dev/shm
tmpfs                1.0G  9.3M 1015M   1% /run
tmpfs                1.0G     0  1.0G   0% /sys/fs/cgroup
/dev/sda1           1014M  141M  874M  14% /boot
[root@jtp1 bin]# vi /etc/fstab

#
# /etc/fstab
# Created by anaconda on Sat Mar 18 15:27:13 2017
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/ol-root     /                       xfs     defaults        0 0
UUID=ca5854cd-0125-4954-a5c4-1ac42c9a0f70 /boot                   xfs     defaults        0 0
/dev/mapper/ol-swap     swap                    swap    defaults        0 0


tmpfs                   /dev/shm                tmpfs   defaults,size=12G        0 0
tmpfs                   /run                    tmpfs   defaults,size=12G        0 0
tmpfs                  /sys/fs/cgroup           tmpfs   defaults,size=12G        0 0

重启集群后,再次检查集群资源状态恢复正常

--------------------------------------------------------------------------------
[grid@jtp1 ~]$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       jtp1                  STABLE
               ONLINE  ONLINE       jtp2                  STABLE
ora.CRS.dg
               ONLINE  ONLINE       jtp1                  STABLE
               ONLINE  ONLINE       jtp2                  STABLE
ora.DATA.dg
               ONLINE  ONLINE       jtp1                  STABLE
               ONLINE  ONLINE       jtp2                  STABLE
ora.FRA.dg
               ONLINE  ONLINE       jtp1                  STABLE
               ONLINE  ONLINE       jtp2                  STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       jtp1                  STABLE
               ONLINE  ONLINE       jtp2                  STABLE
ora.TEST.dg
               ONLINE  ONLINE       jtp1                  STABLE
               ONLINE  ONLINE       jtp2                  STABLE
ora.chad
               ONLINE  ONLINE       jtp1                  STABLE
               ONLINE  ONLINE       jtp2                  STABLE
ora.net1.network
               ONLINE  ONLINE       jtp1                  STABLE
               ONLINE  ONLINE       jtp2                  STABLE
ora.ons
               ONLINE  ONLINE       jtp1                  STABLE
               ONLINE  ONLINE       jtp2                  STABLE
ora.proxy_advm
               OFFLINE OFFLINE      jtp1                  STABLE
               OFFLINE OFFLINE      jtp2                  STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       jtp1                  STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       jtp2                  STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       jtp2                  STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       jtp2                  169.254.237.250 88.8
                                                             8.88.2,STABLE
ora.asm
      1        ONLINE  ONLINE       jtp1                  Started,STABLE
      2        ONLINE  ONLINE       jtp2                  Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       jtp2                  STABLE
ora.jy.db
      1        ONLINE  OFFLINE                               STABLE
      2        ONLINE  OFFLINE                               STABLE
ora.jtp1.vip
      1        ONLINE  ONLINE       jtp1                  STABLE
ora.jtp2.vip
      1        ONLINE  ONLINE       jtp2                  STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       jtp2                  Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       jtp2                  STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       jtp1                  STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       jtp2                  STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       jtp2                  STABLE
--------------------------------------------------------------------------------

到此集群恢复正常

Oracle 12C ORA-12545 While Connecting to RAC through SCAN Name

操作环境为Oracle Linux 7.1,Oracle 12.2 RAC数据库,SCAN IP是使用GNS方式创建,在使用SCAN IP登录数据库时出现如下错误

SQL> conn sys/abcd@cs as sysdba
ERROR:
ORA-12545: Connect failed because target host or object does not exist

错误信息直译是目标主机或对象不存在。

查看scan的配置信息状态正常

[grid@cs1 ~]$ srvctl config scan
SCAN name: cs-cluster-scan.cs-cluster.jy.net, Network: 1
Subnet IPv4: 10.10.10.0/255.255.255.0/ens160, dhcp
Subnet IPv6: 
SCAN 1 IPv4 VIP: -/scan1-vip/10.10.10.143
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes: 
SCAN VIP is individually disabled on nodes: 
SCAN 2 IPv4 VIP: -/scan2-vip/10.10.10.141
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes: 
SCAN VIP is individually disabled on nodes: 
SCAN 3 IPv4 VIP: -/scan3-vip/10.10.10.142
SCAN VIP is enabled.
SCAN VIP is individually enabled on nodes: 
SCAN VIP is individually disabled on nodes: 

查看scan的监听信息状态正常

[grid@cs1 ~]$ srvctl config scan_listener
SCAN Listener LISTENER_SCAN1 exists. Port: TCP:1521
Registration invited nodes: 
Registration invited subnets: 
SCAN Listener is enabled.
SCAN Listener is individually enabled on nodes: 
SCAN Listener is individually disabled on nodes: 
SCAN Listener LISTENER_SCAN2 exists. Port: TCP:1521
Registration invited nodes: 
Registration invited subnets: 
SCAN Listener is enabled.
SCAN Listener is individually enabled on nodes: 
SCAN Listener is individually disabled on nodes: 
SCAN Listener LISTENER_SCAN3 exists. Port: TCP:1521
Registration invited nodes: 
Registration invited subnets: 
SCAN Listener is enabled.
SCAN Listener is individually enabled on nodes: 
SCAN Listener is individually disabled on nodes: 

[grid@cs2 ~]$ lsnrctl status LISTENER_SCAN1

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 13-MAR-2018 11:05:13

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN1
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                12-MAR-2018 19:10:06
Uptime                    0 days 15 hr. 55 min. 7 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/product/12.2.0/crs/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/cs2/listener_scan1/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN1)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.143)(PORT=1521)))
Services Summary...
Service "-MGMTDBXDB" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "66fa07fcd41a56f0e053be828a0abc70" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "_mgmtdb" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "cs" has 2 instance(s).
  Instance "cs1", status READY, has 1 handler(s) for this service...
  Instance "cs2", status READY, has 1 handler(s) for this service...
Service "csXDB" has 2 instance(s).
  Instance "cs1", status READY, has 1 handler(s) for this service...
  Instance "cs2", status READY, has 1 handler(s) for this service...
Service "gimr_dscrep_10" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
The command completed successfully

[grid@cs1 ~]$ lsnrctl status LISTENER_SCAN2

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 13-MAR-2018 11:05:29

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN2)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN2
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                12-MAR-2018 19:08:54
Uptime                    0 days 15 hr. 56 min. 35 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/product/12.2.0/crs/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/cs1/listener_scan2/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN2)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.141)(PORT=1521)))
Services Summary...
Service "-MGMTDBXDB" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "66fa07fcd41a56f0e053be828a0abc70" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "_mgmtdb" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "cs" has 2 instance(s).
  Instance "cs1", status READY, has 1 handler(s) for this service...
  Instance "cs2", status READY, has 1 handler(s) for this service...
Service "csXDB" has 2 instance(s).
  Instance "cs1", status READY, has 1 handler(s) for this service...
  Instance "cs2", status READY, has 1 handler(s) for this service...
Service "gimr_dscrep_10" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
The command completed successfully
[grid@cs1 ~]$ lsnrctl status LISTENER_SCAN3

LSNRCTL for Linux: Version 12.2.0.1.0 - Production on 13-MAR-2018 11:05:32

Copyright (c) 1991, 2016, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN3)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER_SCAN3
Version                   TNSLSNR for Linux: Version 12.2.0.1.0 - Production
Start Date                12-MAR-2018 19:08:52
Uptime                    0 days 15 hr. 56 min. 40 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/product/12.2.0/crs/network/admin/listener.ora
Listener Log File         /u01/app/grid/diag/tnslsnr/cs1/listener_scan3/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER_SCAN3)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.142)(PORT=1521)))
Services Summary...
Service "-MGMTDBXDB" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "66fa07fcd41a56f0e053be828a0abc70" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "_mgmtdb" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "cs" has 2 instance(s).
  Instance "cs1", status READY, has 1 handler(s) for this service...
  Instance "cs2", status READY, has 1 handler(s) for this service...
Service "csXDB" has 2 instance(s).
  Instance "cs1", status READY, has 1 handler(s) for this service...
  Instance "cs2", status READY, has 1 handler(s) for this service...
Service "gimr_dscrep_10" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
The command completed successfully

在通过SCAN Name来连接RAC数据库时,客户端可以解析所有有完整域名的SCAN Name与VIP Name,但是不能解析没有域名的SCAN Name与VIP Name。通过以下操作可以解决这个问题。

1.在数据库级别使用有完整域名的VIP Name或VIP来设置pfile或spfile文件中的local_listener参数

alter system set local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=10.10.10.140)(PORT=1521))))' scope=both sid='cs1';

or
alter system set local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=cs1-vip.jy.net)(PORT=1521))))' scope=both sid='cs1';

and
alter system register;

在RAC的每个节点都执行类似上面的操作,但我的环境中local_listener的设置是正确的

SQL> show parameter local_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
local_listener                       string      (DESCRIPTION=(ADDRESS_LIST=(AD
                                                 DRESS=(PROTOCOL=TCP)(HOST=10.1
                                                 0.10.140)(PORT=1521))))

2.在客户端的hosts文件中增加tnsnames.ora文件中带完整域名的SCAN Name

[root@cs1 ~]# vi /etc/hosts
....
10.10.10.141 cs-cluster-scan.cs-cluster.jy.net
10.10.10.142 cs-cluster-scan.cs-cluster.jy.net
10.10.10.143 cs-cluster-scan.cs-cluster.jy.net

3.再次登录

[oracle@cs11 ~]$ tnsping cs

TNS Ping Utility for Linux: Version 12.2.0.1.0 - Production on 13-MAR-2018 10:40:15

Copyright (c) 1997, 2016, Oracle.  All rights reserved.

Used parameter files:


Used TNSNAMES adapter to resolve the alias
Attempting to contact (DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = cs-cluster-scan.cs-cluster.jy.net)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = cs)))
OK (0 msec)


[oracle@cs1 admin]$ sqlplus /nolog

SQL*Plus: Release 12.2.0.1.0 Production on Tue Mar 13 13:12:32 2018

Copyright (c) 1982, 2016, Oracle.  All rights reserved.

SQL> conn sys/abcd@cs as sysdba
Connected.

到此问题解决了,可以使用SCAN Name来登录数据库了。

Linux中两块device的minor number相同而造成RAC不能启动的问题

Oracle Linux 7.1在安装Oracle 12.2 Grid软件在执行root.sh脚本时,当第一个节点执行成功后,在第二个节点执行出现如下错误

[root@cs2 /]# ./u01/app/product/12.2.0/crs/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/product/12.2.0/crs

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/product/12.2.0/crs/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/cs2/crsconfig/rootcrs_cs2_2018-03-09_04-33-22PM.log
2018/03/09 16:34:23 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2018/03/09 16:34:23 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2018/03/09 16:34:54 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2018/03/09 16:34:54 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2018/03/09 16:35:03 CLSRSC-363: User ignored prerequisites during installation
2018/03/09 16:35:03 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2018/03/09 16:35:05 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2018/03/09 16:35:06 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2018/03/09 16:35:10 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2018/03/09 16:35:12 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2018/03/09 16:35:13 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2018/03/09 16:35:16 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2018/03/09 16:35:24 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2018/03/09 16:35:24 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2018/03/09 16:35:26 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2018/03/09 16:35:42 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2018/03/09 16:36:42 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2018/03/09 16:38:08 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cs2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cs2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2018/03/09 16:38:37 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2018/03/09 16:38:39 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cs2'
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cs2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cs2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'cs2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'cs2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cs2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2018/03/09 16:39:07 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'cs2'
CRS-2672: Attempting to start 'ora.evmd' on 'cs2'
CRS-2676: Start of 'ora.mdnsd' on 'cs2' succeeded
CRS-2676: Start of 'ora.evmd' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'cs2'
CRS-2676: Start of 'ora.gpnpd' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'cs2'
CRS-2676: Start of 'ora.gipcd' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'cs2'
CRS-2676: Start of 'ora.cssdmonitor' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'cs2'
CRS-2672: Attempting to start 'ora.diskmon' on 'cs2'
CRS-2676: Start of 'ora.diskmon' on 'cs2' succeeded
CRS-2883: Resource 'ora.cssdmonitor' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2018/03/09 16:39:46 CLSRSC-378: Failed to get the configured node role for the local node
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
2018/03/09 16:39:48 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /u01/app/product/12.2.0/crs/crs/install/crsinstall.pm line 1494.
The command '/u01/app/product/12.2.0/crs/perl/bin/perl -I/u01/app/product/12.2.0/crs/perl/lib -I/u01/app/product/12.2.0/crs/crs/install /u01/app/product/12.2.0/crs/crs/install/rootcrs.pl ' 

execution failed

出错了,如是再次执行

[root@cs2 /]# ./u01/app/product/12.2.0/crs/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/product/12.2.0/crs

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/product/12.2.0/crs/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/cs2/crsconfig/rootcrs_cs2_2018-03-09_04-51-03PM.log
2018/03/09 16:51:07 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2018/03/09 16:51:07 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2018/03/09 16:51:07 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2018/03/09 16:51:08 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2018/03/09 16:51:11 CLSRSC-363: User ignored prerequisites during installation
2018/03/09 16:51:11 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2018/03/09 16:51:12 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2018/03/09 16:51:12 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2018/03/09 16:51:15 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2018/03/09 16:51:17 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2018/03/09 16:51:18 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2018/03/09 16:51:21 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2018/03/09 16:51:22 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2018/03/09 16:52:09 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2018/03/09 16:52:10 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2018/03/09 16:52:26 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.service'
2018/03/09 16:52:48 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2018/03/09 16:52:51 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2018/03/09 16:52:53 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2018/03/09 16:52:55 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cs2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'cs2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'cs2'
CRS-2673: Attempting to stop 'ora.evmd' on 'cs2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'cs2'
CRS-2677: Stop of 'ora.drivers.acfs' on 'cs2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'cs2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'cs2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'cs2' succeeded
CRS-2673: Attempting to stop 'ora.driver.afd' on 'cs2'
CRS-2673: Attempting to stop 'ora.gipcd' on 'cs2'
CRS-2677: Stop of 'ora.driver.afd' on 'cs2' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'cs2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cs2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2018/03/09 16:53:14 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'cs2'
CRS-2672: Attempting to start 'ora.evmd' on 'cs2'
CRS-2676: Start of 'ora.mdnsd' on 'cs2' succeeded
CRS-2676: Start of 'ora.evmd' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'cs2'
CRS-2676: Start of 'ora.gpnpd' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'cs2'
CRS-2676: Start of 'ora.gipcd' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'cs2'
CRS-2676: Start of 'ora.cssdmonitor' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'cs2'
CRS-2672: Attempting to start 'ora.diskmon' on 'cs2'
CRS-2676: Start of 'ora.diskmon' on 'cs2' succeeded
CRS-1705: Found 0 configured voting files but 1 voting files are required, terminating to ensure data integrity; details at (:CSSNM00021:) in /u01/app/grid/diag/crs/cs2/crs/trace/ocssd.trc
CRS-2883: Resource 'ora.cssdmonitor' failed during Clusterware stack start.
CRS-4406: Oracle High Availability Services synchronous start failed.
CRS-4000: Command Start failed, or completed with errors.
2018/03/09 16:53:53 CLSRSC-378: Failed to get the configured node role for the local node
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4000: Command Status failed, or completed with errors.
2018/03/09 16:53:55 CLSRSC-117: Failed to start Oracle Clusterware stack
Died at /u01/app/product/12.2.0/crs/crs/install/crsinstall.pm line 1494.
The command '/u01/app/product/12.2.0/crs/perl/bin/perl -I/u01/app/product/12.2.0/crs/perl/lib -I/u01/app/product/12.2.0/crs/crs/install /u01/app/product/12.2.0/crs/crs/install/rootcrs.pl ' 

execution failed

还是出错了,但错误信息与第一次执行有所不同了,有以下内容

CRS-1705: Found 0 configured voting files but 1 voting files are required, terminating to ensure data integrity; details at (:CSSNM00021:) in /u01/app/grid/diag/crs/cs2/crs/trace/ocssd.trc

如果查看跟踪文件/u01/app/grid/diag/crs/cs2/crs/trace/ocssd.trc

[root@cs2 ~]# more /u01/app/grid/diag/crs/cs2/crs/trace/ocssd.trc
Trace file /u01/app/grid/diag/crs/cs2/crs/trace/ocssd.trc
Oracle Database 12c Clusterware Release 12.2.0.1.0 - Production Copyright 1996, 2016 Oracle. All rights reserved.
 default:2950017088: 1: clskec:has:CLSU:910 4 args[CLSD00302][mod=clsdadr.c][loc=(:CLSD00302:)][msg=clsdAdrInit: Trace file size and number of segments fetched from environemnt variable: 

ORA_DAEMON_TRACE_FILE_OPTIONS filesize=52428800,n
umsegments=10]

    CLSB:2950017088: Argument count (argc) for this daemon is 3
    CLSB:2950017088: Argument 0 is: /u01/app/product/12.2.0/crs/bin/ocssd.bin
    CLSB:2950017088: Argument 1 is: 
    CLSB:2950017088: Argument 2 is: OSD2PORTIF
2018-03-09 16:39:26.799 :    CSSD:2950017088: (TLM) Starting CSS daemon, version 12.2.0.1.0 with uniqueness value 1520584766
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CSSD, LogLevel = 2
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CSSDNMC, LogLevel = 2
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CSSDGMPC, LogLevel = 2
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CSSDGMCC, LogLevel = 2
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CSSDBCMC, LogLevel = 2
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CSSDGNS, LogLevel = 2
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CLSF, LogLevel = 0
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CLSFA, LogLevel = 3
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = SKGFD, LogLevel = 0
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = GPNP, LogLevel = 1
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CLSINET, LogLevel = 1
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CLSNS, LogLevel = 3
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = OCRAPI, LogLevel = 1
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = OCRCLI, LogLevel = 1
2018-03-09 16:39:26.799 :    CSSD:2950017088: clsu_load_ENV_levels: Module = CLSCEVT, LogLevel = 0
2018-03-09 16:39:26.803 :    CSSD:2950017088: clssscInitGlobalCTX: not in a container
    CSSD:2950017088: clsugetconf : Configuration type [4]. 
2018-03-09 16:39:26.809 :    CSSD:2950017088: clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
2018-03-09 16:39:26.809 :    CSSD:2950017088: clssscagStartAgLsnr: Failed to get auth location from OLR, constructing manually 
2018-03-09 16:39:26.809 :    CSSD:2950017088: clssscagStartAgLsnr: auth location '/u01/app/product/12.2.0/crs/auth/css/' 
2018-03-09 16:39:26.809 :    CSSD:2950017088: clssscGPNPInit: PERF_TIME Initializing GPNP
2018-03-09 16:39:26.810 :    GPNP:2950017088: clsgpnp_Init: HMAC hash type in context is SHA-512
2018-03-09 16:39:26.810 :    GPNP:2950017088: clsgpnp_Init: [at clsgpnp0.c:684] '/u01/app/product/12.2.0/crs' in effect as GPnP home base.
2018-03-09 16:39:26.810 :    GPNP:2950017088: clsgpnp_Init: [at clsgpnp0.c:750] GPnP pid=29348, cli= GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:init, apitl:0, tstenv:0, devenv:0, 

envopt:0, flags=3
2018-03-09 16:39:26.815 :    GPNP:2950017088: clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:403] Using FS Wallet Location : /u01/app/product/12.2.0/crs/gpnp/cs2/wallets/peer/

2018-03-09 16:39:26.815 :    GPNP:2950017088: clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:415] Wallet readable. Path: /u01/app/product/12.2.0/crs/gpnp/cs2/wallets/peer/

2018-03-09 16:39:26.817 :    CSSD:2782349056: clssscthrdmain: Starting thread clssscAgListener
2018-03-09 16:39:26.817 :    CSSD:2782349056: clsssclsnrsetup: endp 0x6b for ipc://agent_ag_cs2_
2018-03-09 16:39:26.817 :    CSSD:2782349056: clssscagOpenAgentEndp: listening on ipc://agent_ag_cs2_ (0x6b)
2018-03-09 16:39:26.818 :    CSSD:2782349056: clsssclsnrsetup: endp 0x7c for ipc://monitor_ag_cs2_
2018-03-09 16:39:26.818 :    CSSD:2782349056: clssscagOpenAgentEndp: listening on ipc://monitor_ag_cs2_ (0x7c)
2018-03-09 16:39:26.818 :    CSSD:2782349056: clssscUpdateInitState: Set state to 0x00080000, based on prior state of 0x00000000 and requested change of 0x00080000
2018-03-09 16:39:26.839 :    GPNP:2950017088: clsgpnp_profileCallUrlInt: [at clsgpnp.c:2238] get-profile call to url "ipc://GPNPD_cs2" disco "" [f=0 claimed- host: cname: cguid: cli:gpnp 

p:29348 role: seq: ep: auth: diag:[]] 
2018-03-09 16:39:26.839 : default:2950017088: clsvactversion:4: Retrieving Active Version from local storage.
2018-03-09 16:39:26.842 : default:2950017088: clsvactversion:4: Retrieving Active Version from local storage.
2018-03-09 16:39:26.851 :    GPNP:2950017088: clsgpnp_profileCallUrlInt: [at clsgpnp.c:2381] Result: (0) CLSGPNP_OK. Successful get-profile CALL to remote "ipc://GPNPD_cs2" disco ""
2018-03-09 16:39:26.851 :    CSSD:2950017088: clssscGPNPInit: Successfully initialized GPNP
2018-03-09 16:39:26.851 :    CSSD:2950017088: clssscInitGlobalCTX: PERF_TIME starting CLSFA for Flex
2018-03-09 16:39:26.851 :    CSSD:2950017088: clssscInitGlobalCTX: PERF_TIME started CLSFA for Flex
2018-03-09 16:39:26.851 :    CSSD:2950017088: Starting CSS daemon in big-clustered mode with a role of hub
2018-03-09 16:39:26.851 :    CSSD:2950017088: clssscInitGlobalCTX: Environment is production
2018-03-09 16:39:26.851 :    CSSD:2950017088: clssscInitGlobalCTX: Core file size limit extended
2018-03-09 16:39:26.852 :    CSSD:2950017088: clssscInitGlobalCTX: GIPCHA down 0
2018-03-09 16:39:26.855 :    CSSD:2950017088: clssscqueue_init: queue(0xe708a0), max(0)
2018-03-09 16:39:26.855 :    CSSD:2950017088: clssscUpdateInitState: Set state to 0x00080002, based on prior state of 0x00080000 and requested change of 0x00000002
2018-03-09 16:39:26.855 :    CSSD:2950017088: osdctx->majik=OSD2PORTIF 
2018-03-09 16:39:26.855 :    CSSD:2950017088: ipmi_ipv4 = ()
2018-03-09 16:39:26.855 :    CSSD:2950017088: clssscModifyWalletIP: Running as user grid
2018-03-09 16:39:26.855 :    CSSD:2950017088: clssscModifyEnvironment: unable to update wallet for IPMI
2018-03-09 16:39:26.857 :    CSSD:2950017088: clssscWaitOnInitState: Waiting on requested state 0x00001000, current state 0x00080002, timeout 4294967295
2018-03-09 16:39:26.858 :  CLSDMT:2779895552: PID for the Process [29348], connkey CSSD
2018-03-09 16:39:26.858 :  CLSDMT:2779895552: ERROR: Empty pid name for proc CSSD
2018-03-09 16:39:26.859 :    CSSD:2778318592: clssscthrdmain: Starting thread Alarm
2018-03-09 16:39:26.859 :    CSSD:2778318592: clssscAlarmThread: Thread Spawned
2018-03-09 16:39:26.859 :    CSSD:2778318592: clssscUpdateInitState: Set state to 0x00081002, based on prior state of 0x00080002 and requested change of 0x00001000
2018-03-09 16:39:26.859 :    CSSD:2950017088: clssscWaitOnInitState: returning 1, requested state 0x00001000, current state 0x00081002
2018-03-09 16:39:26.859 :    CSSD:2950017088: clssscqueue_init: queue(0xf1c670), max(0)
2018-03-09 16:39:26.860 :    CSSD:2950017088: clssscStartActivityThread: Waiting for INIT
2018-03-09 16:39:26.860 :    CSSD:2950017088: clssscWaitOnInitState: Waiting on requested state 0x00800000, current state 0x00081002, timeout 4294967295
2018-03-09 16:39:26.860 :    CSSD:2776741632: clssscthrdmain: Starting thread clssscActivityThread
2018-03-09 16:39:26.860 :    CSSD:2776741632: clssscUpdateInitState: Set state to 0x00881002, based on prior state of 0x00081002 and requested change of 0x00800000
2018-03-09 16:39:26.860 :    CSSD:2950017088: clssscWaitOnInitState: returning 1, requested state 0x00800000, current state 0x00881002
2018-03-09 16:39:26.860 :    CSSD:2950017088: clssscGetParameterProfile: profile fetch failed for parameter ocrid (4) with return code CLSGPNP_NOT_FOUND(5)
2018-03-09 16:39:26.860 :    CSSD:2950017088: clssscmain: OCRID is 0
2018-03-09 16:39:26.860 :    CSSD:2950017088: clssscmain: Cluster GUID is 89e968f37e66cf79bf4a5b7353e8e859
2018-03-09 16:39:26.860 :    CSSD:2950017088: clssscUpdateInitState: Set state to 0x00881006, based on prior state of 0x00881002 and requested change of 0x00000004
2018-03-09 16:39:26.860 : CSSDBCM:2950017088: clssbcmInitialize: flags(0)
2018-03-09 16:39:26.860 : CSSDBCM:2950017088: clssbcm_SetGlobal: global(0xe69560)
2018-03-09 16:39:26.860 : CSSDBCM:2950017088: clssbcmContext: context pointer(0x7fff90720590), flags(0)
2018-03-09 16:39:26.861 : CSSDBCM:2950017088: clssbcmNewEvent: event(0xe704c0), name(BCCM State), value(1)
2018-03-09 16:39:26.861 : CSSDBCM:2950017088: clssbcmNewEvent: event(0xe664b0), name(BCCM Work State), value(0)
2018-03-09 16:39:26.861 : CSSDBCM:2950017088: clssbcm_NewWorkThread: Spawn status(1)
2018-03-09 16:39:26.862 :    CSSD:2775164672: clssscthrdmain: Starting thread BCCM_Worker
2018-03-09 16:39:26.862 : CSSDBCM:2775164672: clssbcm_worker: Spawned with thread(0xe673f0)
2018-03-09 16:39:26.862 : CSSDBCM:2950017088: clssbcmContext: context(0xe6bb30) created
2018-03-09 16:39:26.862 : CSSDBCM:2950017088: clssbcmContext: context pointer(0x7fff90720590), flags(0)
2018-03-09 16:39:26.862 : CSSDBCM:2950017088: clssbcmNewEvent: event(0xd07650), name(BCCM State), value(1)
2018-03-09 16:39:26.863 : CSSDBCM:2950017088: clssbcmNewEvent: event(0xd07790), name(BCCM Work State), value(0)
2018-03-09 16:39:26.863 : CSSDBCM:2950017088: clssbcm_NewWorkThread: Spawn status(1)
2018-03-09 16:39:26.864 :    CSSD:2773587712: clssscthrdmain: Starting thread BCCM_Worker
2018-03-09 16:39:26.864 : CSSDBCM:2773587712: clssbcm_worker: Spawned with thread(0xd07a90)
2018-03-09 16:39:26.864 : CSSDBCM:2950017088: clssbcmContext: context(0xd0ecd0) created
2018-03-09 16:39:26.864 : CSSDBCM:2950017088: clssbcmSetGIPCTraceLevel: context(0xe6bb30), trace(2)
2018-03-09 16:39:26.864 : CSSDBCM:2950017088: clssbcmSetGIPCTraceLevel: context(0xd0ecd0), trace(2)
2018-03-09 16:39:26.864 :    CSSD:2950017088: clssscmain: PERF_TIME initializing GNS for Flex
2018-03-09 16:39:26.864 : CSSDGNS:2950017088: clssscBQInitialize: Initialized GNS Work queue

2018-03-09 16:39:26.864 :    CSSD:2950017088: clssscWaitOnInitState: Waiting on requested state 0x00040000, current state 0x00881006, timeout 4294967295
2018-03-09 16:39:26.865 :    CSSD:2772010752: clssscthrdmain: Starting thread GNSWork
2018-03-09 16:39:26.865 :    CSSD:2772010752: clssscUpdateInitState: Set state to 0x008c1006, based on prior state of 0x00881006 and requested change of 0x00040000
2018-03-09 16:39:26.866 :   CLSNS:2772010752: clsns_SetTraceLevel:trace level set to 1.
2018-03-09 16:39:26.866 : CSSDGNS:2772010752: clssgnsCrsQuery: Querying CRS for resource type "ora.gns.type".
2018-03-09 16:39:26.867 :    CSSD:2950017088: clssscWaitOnInitState: returning 1, requested state 0x00040000, current state 0x008c1006
2018-03-09 16:39:26.867 :    CSSD:2950017088: clssscmain: PERF_TIME initialized GNS for Flex
2018-03-09 16:39:26.868 :    CSSD:2950017088: clssscWaitOnInitState: Waiting on requested state 0x00000400, current state 0x008c1006, timeout 4294967295
2018-03-09 16:39:26.869 :    CSSD:2770433792: clssscthrdmain: Starting thread BCNMServer
2018-03-09 16:39:26.869 :    CSSD:2770433792: clssbnmsThread: Spawned
2018-03-09 16:39:26.869 :    CSSD:2770433792: clssbnmInitBCNMCommon: Initialialization complete for comm ctx(0xd0fbb8)
2018-03-09 16:39:26.869 :    CSSD:2770433792: clssscUpdateInitState: Set state to 0x008c1406, based on prior state of 0x008c1006 and requested change of 0x00000400
2018-03-09 16:39:26.869 :    CSSD:2770433792: clssscWaitOnInitState: Waiting on requested state 0x00020000, current state 0x008c1406, timeout 4294967295
2018-03-09 16:39:26.871 :    CSSD:2950017088: clssscWaitOnInitState: returning 1, requested state 0x00000400, current state 0x008c1406
2018-03-09 16:39:26.872 :    CSSD:2950017088: clssscWaitOnInitState: Waiting on requested state 0x00000800, current state 0x008c1406, timeout 4294967295
2018-03-09 16:39:26.873 :    CSSD:2768856832: clssscthrdmain: Starting thread BCNMClient
2018-03-09 16:39:26.873 :    CSSD:2768856832: clssbnmcThread: Thread Spawned
2018-03-09 16:39:26.873 :    CSSD:2768856832: clssbnmInitBCNMCommon: Initialialization complete for comm ctx(0xecc348)
2018-03-09 16:39:26.873 :    CSSD:2768856832: clssscUpdateInitState: Set state to 0x008c1c06, based on prior state of 0x008c1406 and requested change of 0x00000800
2018-03-09 16:39:26.874 : CSSDGNS:2772010752: clssgnsCrsQuery: CRS is not ready. Cannot query GNS resource state.
2018-03-09 16:39:26.874 : CLSCEVT:2772010752: clsce_subscribe 0x7f568011b540 filter='^(CRS_RESOURCE_PROFILE_CHANGE|CRS_RESOURCE_STATE_CHANGE),.*?NAME='ora.gns'', flags=0, handler=0x58fa24, 

arg=0xd082c0
2018-03-09 16:39:26.875 :    CSSD:2950017088: clssscWaitOnInitState: returning 1, requested state 0x00000800, current state 0x008c1c06
2018-03-09 16:39:26.876 :    CSSD:2950017088: clssscGetParameterOLR: OLR fetch for parameter node number hint (6) failed with rc 21
2018-03-09 16:39:26.886 :    CSSD:2950017088: clssgmGMPInitCtx: GMP context init done
2018-03-09 16:39:26.886 :    CSSD:2950017088: clssscUpdateInitState: Set state to 0x008c1e06, based on prior state of 0x008c1c06 and requested change of 0x00000200
2018-03-09 16:39:26.887 : CLSCEVT:2772010752: (:CLSCE0022:)clsce_subscribe 0x7f568011b540 successfully subscribed : 0
2018-03-09 16:39:26.888 :    CSSD:2764130048: clssscthrdmain: Starting thread GM Peer Lsnr
2018-03-09 16:39:26.888 :    CSSD:2764130048: clssgmPeerListener: Spawned for node
2018-03-09 16:39:26.888 :    CSSD:2764130048: clssscWaitOnInitState: Waiting on requested state 0x00000001, current state 0x008c1e06, timeout 4294967295
2018-03-09 16:39:26.890 :    CSSD:2762553088: clssscthrdmain: Starting thread GMC Req Thrd
2018-03-09 16:39:26.890 :    CSSD:2762553088: clssgmGMCRequestThread: Starting
2018-03-09 16:39:26.890 :    CSSD:2950017088: clsswtStartWrkthrds: Workerthread GMPLstnrWorkerThread, thrdnum 1 spawn success
2018-03-09 16:39:26.891 :    CSSD:2950017088: clsswtStartWrkthrds: Workerthread GMPLstnrWorkerThread, thrdnum 2 spawn success
2018-03-09 16:39:26.892 :    CSSD:2950017088: clsswtStartWrkthrds: Workerthread GMPLstnrWorkerThread, thrdnum 3 spawn success
2018-03-09 16:39:26.893 :    CSSD:2950017088: clsswtStartWrkthrds: Workerthread GMCReqWorkerThread, thrdnum 1 spawn success
2018-03-09 16:39:26.893 :    CSSD:2950017088: clsswtStartWrkthrds: Workerthread GMCReqWorkerThread, thrdnum 2 spawn success
2018-03-09 16:39:26.894 :    CSSD:2950017088: clsswtStartWrkthrds: Workerthread GMCReqWorkerThread, thrdnum 3 spawn success
2018-03-09 16:39:26.896 :    CSSD:2760976128: clssscthrdmain: Starting thread GMPLstnrWorkerThread
2018-03-09 16:39:26.896 :    CSSD:2760976128: clssgmPeerWorkerThread: thrdname GMPLstnrWorkerThread, num 35, wrkthrdnum 1
2018-03-09 16:39:26.897 :    CSSD:2759399168: clssscthrdmain: Starting thread GMPLstnrWorkerThread
2018-03-09 16:39:26.897 :    CSSD:2759399168: clssgmPeerWorkerThread: thrdname GMPLstnrWorkerThread, num 35, wrkthrdnum 2
2018-03-09 16:39:26.897 :    CSSD:2757822208: clssscthrdmain: Starting thread GMPLstnrWorkerThread
2018-03-09 16:39:26.898 :    CSSD:2757822208: clssgmPeerWorkerThread: thrdname GMPLstnrWorkerThread, num 35, wrkthrdnum 3
2018-03-09 16:39:26.899 :    CSSD:2756245248: clssscthrdmain: Starting thread GMCReqWorkerThread
2018-03-09 16:39:26.899 :    CSSD:2756245248: clssgmpcGMCReqWorkerThread: thrdname GMCReqWorkerThread, num 36, wrkthrdnum 1 spawned
2018-03-09 16:39:26.899 :    CSSD:2754668288: clssscthrdmain: Starting thread GMCReqWorkerThread
2018-03-09 16:39:26.899 :    CSSD:2754668288: clssgmpcGMCReqWorkerThread: thrdname GMCReqWorkerThread, num 36, wrkthrdnum 2 spawned
2018-03-09 16:39:26.900 :    CSSD:2753091328: clssscthrdmain: Starting thread GMCReqWorkerThread
2018-03-09 16:39:26.900 :    CSSD:2753091328: clssgmpcGMCReqWorkerThread: thrdname GMCReqWorkerThread, num 36, wrkthrdnum 3 spawned
2018-03-09 16:39:26.901 :    CSSD:2950017088: clssscGetParameterOLR: OLR fetch for parameter auth rep (9) failed with rc 21
2018-03-09 16:39:26.901 :    CSSD:2950017088: clssgmceventsub_construct: eventsub(0x13e0610), gmc(0x139adf0), name(Event Subscription Cache)
2018-03-09 16:39:26.901 :    CSSD:2950017088: clssgmGMCInitCtx: GMC context init done
2018-03-09 16:39:26.902 :    CSSD:2011944704: clssscthrdmain: Starting thread GMClientListener
2018-03-09 16:39:26.902 :    CSSD:2011944704: clssgmclientlsnr: Spawned
2018-03-09 16:39:26.902 :    CSSD:2011944704: clssscqueue_init: queue(0x7f5658032cd0), max(0)
2018-03-09 16:39:26.902 :    CSSD:2011944704: clssscqueue_init: queue(0x7f5658032fd0), max(0)
2018-03-09 16:39:26.903 :    CSSD:2011944704: clsssclsnrsetup: endp 0x20a for clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_cs2_)(GIPCID=00000000-00000000-29348))
2018-03-09 16:39:26.903 :    CSSD:2011944704: clssgmclientlsnr: listening on clsc://(ADDRESS=(PROTOCOL=ipc)(KEY=OCSSD_LL_cs2_)(GIPCID=00000000-00000000-29348))
2018-03-09 16:39:26.903 :    CSSD:2011944704: clssgmceventsub_construct: eventsub(0x7f565805a340), gmc(0x139adf0), name(Event Subscription Cache)
2018-03-09 16:39:26.903 :    CSSD:2011944704: clssscWaitOnInitState: Waiting on requested state 0x00000001, current state 0x008c1e06, timeout 4294967295
2018-03-09 16:39:26.905 :    CSSD:2010367744: clssscthrdmain: Starting thread GMDeathCheck
2018-03-09 16:39:26.905 :    CSSD:2010367744: clssgmDeathChkThread: Spawned
2018-03-09 16:39:26.907 :    CSSD:2008790784: clssscthrdmain: Starting thread GMPResponse
2018-03-09 16:39:26.907 :    CSSD:2950017088: clssscUpdateInitState: Set state to 0x008c1e46, based on prior state of 0x008c1e06 and requested change of 0x00000040
2018-03-09 16:39:26.908 :    CSSD:2950017088: clssscGetParameterProfile: buffer passed for parameter ASM discovery (3) is too short, required 25, passed 20
2018-03-09 16:39:26.908 :    CSSD:2950017088: clssnmReadDiscoveryProfile: voting file discovery string(/dev/sd*,/dev/asm*,AFD:*)
2018-03-09 16:39:26.908 :    CSSD:2950017088: clssnkInit: NK generic layer initializing.
2018-03-09 16:39:26.909 :    CSSD:2007213824: clssscthrdmain: Starting thread clssnmvDDiscThread
2018-03-09 16:39:26.910 :    GPNP:2007213824: clsgpnp_Init: HMAC hash type in context is SHA-512
2018-03-09 16:39:26.910 :    GPNP:2007213824: clsgpnp_Init: [at clsgpnp0.c:684] '/u01/app/product/12.2.0/crs' in effect as GPnP home base.
2018-03-09 16:39:26.910 :    GPNP:2007213824: clsgpnp_Init: [at clsgpnp0.c:750] GPnP pid=29348, cli=clsuGpnpg GPNP comp tracelevel=1, depcomp tracelevel=0, tlsrc:init, apitl:0, tstenv:0, 

devenv:0, envopt:0, flags=2003
2018-03-09 16:39:26.915 :    GPNP:2007213824: clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:403] Using FS Wallet Location : /u01/app/product/12.2.0/crs/gpnp/cs2/wallets/peer/

2018-03-09 16:39:26.915 :    GPNP:2007213824: clsgpnpkwf_initwfloc: [at clsgpnpkwf.c:415] Wallet readable. Path: /u01/app/product/12.2.0/crs/gpnp/cs2/wallets/peer/

2018-03-09 16:39:26.940 :    CLSF:2007213824: Allocated CLSF context
2018-03-09 16:39:26.940 :    CSSD:2007213824: clssnmvDDiscThread: using discovery string /dev/sd*,/dev/asm*,AFD:* for initial discovery 
2018-03-09 16:39:26.940 :   SKGFD:2007213824: Discovery with str:/dev/sd*,/dev/asm*,AFD:*:

2018-03-09 16:39:26.940 :   SKGFD:2007213824: UFS discovery with :/dev/sd*:

2018-03-09 16:39:26.940 :   SKGFD:2007213824: Execute glob on the string /dev/sd*

2018-03-09 16:39:26.940 :   SKGFD:2007213824: running stat on disk:/dev/sda2
2018-03-09 16:39:26.940 :   SKGFD:2007213824: running stat on disk:/dev/sda1
2018-03-09 16:39:26.940 :   SKGFD:2007213824: running stat on disk:/dev/sdc
2018-03-09 16:39:26.940 :   SKGFD:2007213824: running stat on disk:/dev/sdb
2018-03-09 16:39:26.940 :   SKGFD:2007213824: running stat on disk:/dev/sdf
2018-03-09 16:39:26.940 :   SKGFD:2007213824: running stat on disk:/dev/sde
2018-03-09 16:39:26.940 :   SKGFD:2007213824: running stat on disk:/dev/sdd
2018-03-09 16:39:26.940 :   SKGFD:2007213824: running stat on disk:/dev/sda
2018-03-09 16:39:27.128 :    CSSD:2782349056: clssscagSelect: endpoint(0x85) authenticated with user(root)
2018-03-09 16:39:27.128 :    CSSD:2782349056: clssscagProcessInitialMsg: Handshake successful with agent 1
2018-03-09 16:39:27.128 :    CSSD:2782349056: clssscagProcAgReq: got a successful connection
2018-03-09 16:39:27.128 :    CSSD:2782349056: clssscagProcAgReq: Sending initdata
2018-03-09 16:39:27.128 :    CSSD:2782349056: clssscagProcessInitialMsg: notify agent 1 that it is active
2018-03-09 16:39:27.128 :    CSSD:2782349056: clssscagProcessInitialMsg: connection from agent 1, endp 0x85 - agents joined 0
2018-03-09 16:39:27.718 :    CSSD:2782349056: clssscagSelect: endpoint(0x74) authenticated with user(root)
2018-03-09 16:39:27.718 :    CSSD:2782349056: clssscagProcessInitialMsg: Handshake successful with agent 0
2018-03-09 16:39:27.718 :    CSSD:2782349056: clssscagProcAgReq: got a successful connection
2018-03-09 16:39:27.718 :    CSSD:2782349056: clssscagProcAgReq: Sending initdata
2018-03-09 16:39:27.718 :    CSSD:2782349056: clssscagProcessInitialMsg: connection from agent 0, endp 0x74 - agents joined 0
2018-03-09 16:39:27.868 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:28.868 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:29.868 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:30.689 :   SKGFD:2007213824: Fetching UFS disk :/dev/sda:

2018-03-09 16:39:30.689 :    CLSF:2007213824: Ignoring 0-byte file /dev/sda

2018-03-09 16:39:30.689 :   SKGFD:2007213824: Fetching UFS disk :/dev/sdd:

2018-03-09 16:39:30.689 :    CLSF:2007213824: Ignoring 0-byte file /dev/sdd

2018-03-09 16:39:30.689 :   SKGFD:2007213824: Fetching UFS disk :/dev/sde:

2018-03-09 16:39:30.689 :    CLSF:2007213824: Ignoring 0-byte file /dev/sde

2018-03-09 16:39:30.689 :   SKGFD:2007213824: Fetching UFS disk :/dev/sdf:

2018-03-09 16:39:30.689 :    CLSF:2007213824: Ignoring 0-byte file /dev/sdf

2018-03-09 16:39:30.689 :   SKGFD:2007213824: Fetching UFS disk :/dev/sdb:

2018-03-09 16:39:30.689 :    CLSF:2007213824: Ignoring 0-byte file /dev/sdb

2018-03-09 16:39:30.689 :   SKGFD:2007213824: Fetching UFS disk :/dev/sdc:

2018-03-09 16:39:30.689 :    CLSF:2007213824: Ignoring 0-byte file /dev/sdc

2018-03-09 16:39:30.689 :   SKGFD:2007213824: Fetching UFS disk :/dev/sda1:

2018-03-09 16:39:30.689 :    CLSF:2007213824: Ignoring 0-byte file /dev/sda1

2018-03-09 16:39:30.689 :   SKGFD:2007213824: Fetching UFS disk :/dev/sda2:

2018-03-09 16:39:30.689 :    CLSF:2007213824: Ignoring 0-byte file /dev/sda2

2018-03-09 16:39:30.689 :   SKGFD:2007213824: OSS discovery with :/dev/sd*:

2018-03-09 16:39:30.690 :   SKGFD:2007213824: Discovery with asmlib :ASM:AFD Library - Generic , version 3 (KABI_V3): str :/dev/sd*:

2018-03-09 16:39:30.690 :   SKGFD:2007213824: Discovery advancing to nxt string :/dev/asm*:

2018-03-09 16:39:30.690 :   SKGFD:2007213824: UFS discovery with :/dev/asm*:

2018-03-09 16:39:30.690 :   SKGFD:2007213824: Execute glob on the string /dev/asm*

2018-03-09 16:39:30.690 :   SKGFD:2007213824: running stat on disk:/dev/asm
2018-03-09 16:39:30.690 :   SKGFD:2007213824: running stat on disk:/dev/asmdisk05
2018-03-09 16:39:30.692 :   SKGFD:2007213824: running stat on disk:/dev/asmdisk04
2018-03-09 16:39:30.693 :   SKGFD:2007213824: running stat on disk:/dev/asmdisk03
2018-03-09 16:39:30.707 :   SKGFD:2007213824: running stat on disk:/dev/asmdisk02
2018-03-09 16:39:30.729 :   SKGFD:2007213824: running stat on disk:/dev/asmdisk01
2018-03-09 16:39:30.869 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:31.869 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:32.869 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:33.869 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:34.869 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:35.163 :   SKGFD:2007213824: Fetching UFS disk :/dev/asmdisk01:

2018-03-09 16:39:35.163 :   SKGFD:2007213824: Fetching UFS disk :/dev/asmdisk02:

2018-03-09 16:39:35.163 :   SKGFD:2007213824: Fetching UFS disk :/dev/asmdisk03:

2018-03-09 16:39:35.163 :   SKGFD:2007213824: Fetching UFS disk :/dev/asmdisk04:

2018-03-09 16:39:35.163 :   SKGFD:2007213824: Fetching UFS disk :/dev/asmdisk05:

2018-03-09 16:39:35.163 :   SKGFD:2007213824: Fetching UFS disk :/dev/asm:

2018-03-09 16:39:35.163 :    CLSF:2007213824: Ignoring 0-byte file /dev/asm

2018-03-09 16:39:35.163 :   SKGFD:2007213824: OSS discovery with :/dev/asm*:

2018-03-09 16:39:35.164 :   SKGFD:2007213824: Discovery with asmlib :ASM:AFD Library - Generic , version 3 (KABI_V3): str :/dev/asm*:

2018-03-09 16:39:35.164 :   SKGFD:2007213824: Discovery advancing to nxt string :AFD:*:

2018-03-09 16:39:35.164 :   SKGFD:2007213824: UFS discovery with :AFD:*:

2018-03-09 16:39:35.164 :   SKGFD:2007213824: Execute glob on the string /u01/app/product/12.2.0/crs/dbs/AFD:*

2018-03-09 16:39:35.869 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:36.870 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:37.870 :    CSSD:2778318592: clsssc_CLSFAInit_CB: System not ready for CLSFA initialization
2018-03-09 16:39:38.691 :   SKGFD:2007213824: OSS discovery with :AFD:*:

2018-03-09 16:39:38.692 :   SKGFD:2007213824: Discovery with asmlib :ASM:AFD Library - Generic , version 3 (KABI_V3): str :AFD:*:

2018-03-09 16:39:38.692 :   SKGFD:2007213824: Fetching asmlib disk :AFD:CRS2:

2018-03-09 16:39:38.692 :   SKGFD:2007213824: Handle 0x7f5644337dc0 from lib :UFS:: for disk :/dev/asmdisk01:

2018-03-09 16:39:38.692 :   SKGFD:2007213824: Handle 0x7f5644338d00 from lib :UFS:: for disk :/dev/asmdisk02:

2018-03-09 16:39:38.692 :   SKGFD:2007213824: Handle 0x7f5644339c40 from lib :UFS:: for disk :/dev/asmdisk03:

2018-03-09 16:39:38.693 :   SKGFD:2007213824: Handle 0x7f564433ab80 from lib :UFS:: for disk :/dev/asmdisk04:

2018-03-09 16:39:38.693 :   SKGFD:2007213824: Handle 0x7f564433bc50 from lib :UFS:: for disk :/dev/asmdisk05:

2018-03-09 16:39:38.693 :   SKGFD:2007213824: Handle 0x7f564433cd20 from lib :ASM:AFD Library - Generic , version 3 (KABI_V3): for disk :AFD:CRS2:

2018-03-09 16:39:38.693 :    CLSF:2007213824:  Warning: Voting disk: /dev/asmdisk01 is Hard mounted 

2018-03-09 16:39:38.693 :   SKGFD:2007213824: Lib :UFS:: closing handle 0x7f5644337dc0 for disk :/dev/asmdisk01:

2018-03-09 16:39:38.693 :    CLSF:2007213824:  Warning: Voting disk: /dev/asmdisk02 is Hard mounted 

2018-03-09 16:39:38.693 :   SKGFD:2007213824: Lib :UFS:: closing handle 0x7f5644338d00 for disk :/dev/asmdisk02:

2018-03-09 16:39:38.693 :    CLSF:2007213824: Read header of dev:/dev/asmdisk03:none:
2018-03-09 16:39:38.693 :   SKGFD:2007213824: Lib :UFS:: closing handle 0x7f5644339c40 for disk :/dev/asmdisk03:

2018-03-09 16:39:38.693 :    CLSF:2007213824: Read header of dev:/dev/asmdisk04:none:
2018-03-09 16:39:38.693 :   SKGFD:2007213824: Lib :UFS:: closing handle 0x7f564433ab80 for disk :/dev/asmdisk04:

2018-03-09 16:39:38.702 :    CLSF:2007213824: Read header of dev:/dev/asmdisk05:none:
2018-03-09 16:39:38.702 :   SKGFD:2007213824: Lib :UFS:: closing handle 0x7f564433bc50 for disk :/dev/asmdisk05:

2018-03-09 16:39:38.702 :   SKGFD:2007213824: Lib :ASM:AFD Library - Generic , version 3 (KABI_V3): closing handle 0x7f564433cd20 for disk :AFD:CRS2:

2018-03-09 16:39:38.702 :    CSSD:2007213824: clssnmvDiskVerify: discovered a potential voting file
2018-03-09 16:39:38.702 :   SKGFD:2007213824: Handle 0x7f564432fec0 from lib :UFS:: for disk :/dev/asmdisk01:

2018-03-09 16:39:38.702 :    CLSF:2007213824: Opened hdl:0x7f564432e790 for dev:/dev/asmdisk01:
2018-03-09 16:39:38.702 :    CSSD:2007213824: clssnmvDiskCreate: name /dev/asmdisk01 blocksz 512
2018-03-09 16:39:38.711 :    CSSD:2007213824: clssnmvDiskCreate: siteid during discovery = 00000000000000000000000000000000
2018-03-09 16:39:38.711 :    CSSD:2007213824: clssnmvDiskVerify: Successful discovery for disk /dev/asmdisk01, UID 750a78e1-ae984fcd-bfb4dbf4-4d337a77, SID 00112233-44556677-8899aabb-

ccddeeff, Pending CIN 0:1520582274:0, Committed CIN 0
:1520582274:0
2018-03-09 16:39:38.711 :    CLSF:2007213824: Closing handle:0x7f564432e790
2018-03-09 16:39:38.711 :   SKGFD:2007213824: Lib :UFS:: closing handle 0x7f564432fec0 for disk :/dev/asmdisk01:

2018-03-09 16:39:38.711 :    CSSD:2007213824: clssnmvDiskVerify: discovered a potential voting file
2018-03-09 16:39:38.711 :   SKGFD:2007213824: Handle 0x7f564432fec0 from lib :UFS:: for disk :/dev/asmdisk02:

2018-03-09 16:39:38.711 :    CLSF:2007213824: Opened hdl:0x7f564432e790 for dev:/dev/asmdisk02:
2018-03-09 16:39:38.711 :    CSSD:2007213824: clssnmvDiskCreate: name /dev/asmdisk02 blocksz 512
2018-03-09 16:39:38.712 :    CSSD:2007213824: clssnmvDiskCreate: siteid during discovery = 00000000000000000000000000000000
2018-03-09 16:39:38.712 :    CSSD:2007213824: clssnmFindVF: found VF by vdin in the discovered queue
2018-03-09 16:39:38.712 :    CSSD:2007213824: clssnmFindVF: Duplicate voting file found in the queue of previously discovered disks queued(/dev/asmdisk01|[750a78e1-ae984fcd-bfb4dbf4-

4d337a77]), found(/dev/asmdisk02|[750a78e1-ae984fcd-bfb4dbf4-4d337a77]), is not corrupted
2018-03-09 16:39:38.712 :    CLSF:2007213824: Resolved (/dev/asmdisk01,/dev/asmdisk02) to NONE
2018-03-09 16:39:38.712 :    CSSD:2007213824: clssnmvDiskCreate: Found a duplicate voting file /dev/asmdisk01 in the discovery queue which appears to be the same physical device as the 

newly discovered disk /dev/asmdisk02. Rejecting bot
h these files
2018-03-09 16:39:38.712 : default:2007213824: clssnmvDiskCreate:destroy_vdisk->vdisk:  dump of 0x0x7f56443d3760, len 10616

发现CRS磁盘组中磁盘/dev/asmdisk01中的voting file有一份重复的副本出现在/dev/asmdisk02磁盘中,因此拒绝了这些文件。

CSSD:2007213824: clssnmvDiskCreate: Found a duplicate voting file /dev/asmdisk01 in the discovery queue which appears to be the same physical device as the newly discovered disk 

/dev/asmdisk02. Rejecting both these files

如是在节点cs2上查看这两块磁盘,可以看到 /dev/asmdisk01与/dev/asmdisk02的minor number都显示为32,而cs1上的/dev/asmdisk01的minor number是16,/dev/asmdisk02的minor number为32

[root@cs2 /]# ls -lrt /dev/asm*
brw-rw---- 1 grid asmadmin 8, 32 Mar  9 12:21 /dev/asmdisk01
brw-rw---- 1 grid asmadmin 8, 32 Mar  9 12:23 /dev/asmdisk02
brw-rw---- 1 grid asmadmin 8, 48 Mar  9 12:23 /dev/asmdisk03
brw-rw---- 1 grid asmadmin 8, 64 Mar  9 12:23 /dev/asmdisk04
brw-rw---- 1 grid asmadmin 8, 80 Mar  9 12:23 /dev/asmdisk05

[root@cs1 /]# ls -lrt /dev/asm*
brw-rw---- 1 grid asmadmin 8, 48 Mar  9 12:23 /dev/asmdisk03
brw-rw---- 1 grid asmadmin 8, 64 Mar  9 12:23 /dev/asmdisk04
brw-rw---- 1 grid asmadmin 8, 80 Mar  9 12:23 /dev/asmdisk05
brw-rw---- 1 grid asmadmin 8, 32 Mar  9 15:57 /dev/asmdisk02
brw-rw---- 1 grid asmadmin 8, 16 Mar  9 15:57 /dev/asmdisk01

如果手动删除这些用来创建asm磁盘组的块设备

[root@cs2 /]# /bin/rm /dev/asmdisk01
[root@cs2 /]# /bin/rm /dev/asmdisk02
[root@cs2 /]# /bin/rm /dev/asmdisk023
[root@cs2 /]# /bin/rm /dev/asmdisk03
[root@cs2 /]# /bin/rm /dev/asmdisk04
[root@cs2 /]# /bin/rm /dev/asmdisk05
[root@cs2 /]# ls -lrt /dev/asm*
total 0

重新创建这些块设备

[root@cs2 /]# /sbin/udevadm trigger --type=devices --action=change
[root@cs2 /]# ls -lrt /dev/asm*
brw-rw---- 1 grid asmadmin 8, 16 Mar  9 17:37 /dev/asmdisk01
brw-rw---- 1 grid asmadmin 8, 32 Mar  9 17:37 /dev/asmdisk02
brw-rw---- 1 grid asmadmin 8, 64 Mar  9 17:37 /dev/asmdisk04
brw-rw---- 1 grid asmadmin 8, 80 Mar  9 17:37 /dev/asmdisk05
brw-rw---- 1 grid asmadmin 8, 48 Mar  9 17:37 /dev/asmdisk03

/dev/asm:
total 0

再次重新执行root.sh脚本就可以执行成功

[root@cs2 /]# ./u01/app/product/12.2.0/crs/root.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= grid
    ORACLE_HOME=  /u01/app/product/12.2.0/crs

Enter the full pathname of the local bin directory: [/usr/local/bin]: 
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/product/12.2.0/crs/crs/install/crsconfig_params
The log of current session can be found at:
  /u01/app/grid/crsdata/cs2/crsconfig/rootcrs_cs2_2018-03-09_05-37-57PM.log
2018/03/09 17:38:01 CLSRSC-594: Executing installation step 1 of 19: 'SetupTFA'.
2018/03/09 17:38:01 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2018/03/09 17:38:01 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
2018/03/09 17:38:02 CLSRSC-594: Executing installation step 2 of 19: 'ValidateEnv'.
2018/03/09 17:38:05 CLSRSC-363: User ignored prerequisites during installation
2018/03/09 17:38:05 CLSRSC-594: Executing installation step 3 of 19: 'CheckFirstNode'.
2018/03/09 17:38:06 CLSRSC-594: Executing installation step 4 of 19: 'GenSiteGUIDs'.
2018/03/09 17:38:06 CLSRSC-594: Executing installation step 5 of 19: 'SaveParamFile'.
2018/03/09 17:38:09 CLSRSC-594: Executing installation step 6 of 19: 'SetupOSD'.
2018/03/09 17:38:11 CLSRSC-594: Executing installation step 7 of 19: 'CheckCRSConfig'.
2018/03/09 17:38:12 CLSRSC-594: Executing installation step 8 of 19: 'SetupLocalGPNP'.
2018/03/09 17:38:15 CLSRSC-594: Executing installation step 9 of 19: 'ConfigOLR'.
2018/03/09 17:38:16 CLSRSC-594: Executing installation step 10 of 19: 'ConfigCHMOS'.
2018/03/09 17:39:03 CLSRSC-594: Executing installation step 11 of 19: 'CreateOHASD'.
2018/03/09 17:39:04 CLSRSC-594: Executing installation step 12 of 19: 'ConfigOHASD'.
2018/03/09 17:39:06 CLSRSC-594: Executing installation step 13 of 19: 'InstallAFD'.
2018/03/09 17:39:08 CLSRSC-594: Executing installation step 14 of 19: 'InstallACFS'.
2018/03/09 17:39:11 CLSRSC-594: Executing installation step 15 of 19: 'InstallKA'.
2018/03/09 17:39:12 CLSRSC-594: Executing installation step 16 of 19: 'InitConfig'.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'cs2'
CRS-2673: Attempting to stop 'ora.driver.afd' on 'cs2'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'cs2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'cs2'
CRS-2673: Attempting to stop 'ora.gipcd' on 'cs2'
CRS-2673: Attempting to stop 'ora.evmd' on 'cs2'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'cs2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'cs2'
CRS-2677: Stop of 'ora.cssdmonitor' on 'cs2' succeeded
CRS-2677: Stop of 'ora.driver.afd' on 'cs2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'cs2' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'cs2' succeeded
CRS-2677: Stop of 'ora.evmd' on 'cs2' succeeded
CRS-2677: Stop of 'ora.gipcd' on 'cs2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'cs2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cs2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2018/03/09 17:39:17 CLSRSC-594: Executing installation step 17 of 19: 'StartCluster'.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.evmd' on 'cs2'
CRS-2672: Attempting to start 'ora.mdnsd' on 'cs2'
CRS-2676: Start of 'ora.mdnsd' on 'cs2' succeeded
CRS-2676: Start of 'ora.evmd' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'cs2'
CRS-2676: Start of 'ora.gpnpd' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'cs2'
CRS-2676: Start of 'ora.gipcd' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'cs2'
CRS-2676: Start of 'ora.cssdmonitor' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'cs2'
CRS-2672: Attempting to start 'ora.diskmon' on 'cs2'
CRS-2676: Start of 'ora.diskmon' on 'cs2' succeeded
CRS-2676: Start of 'ora.cssd' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'cs2'
CRS-2672: Attempting to start 'ora.ctssd' on 'cs2'
CRS-2676: Start of 'ora.ctssd' on 'cs2' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'cs2'
CRS-2676: Start of 'ora.asm' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'cs2'
CRS-2676: Start of 'ora.storage' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'cs2'
CRS-2676: Start of 'ora.crf' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'cs2'
CRS-2676: Start of 'ora.crsd' on 'cs2' succeeded
CRS-6017: Processing resource auto-start for servers: cs2
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'cs1'
CRS-2672: Attempting to start 'ora.net1.network' on 'cs2'
CRS-2672: Attempting to start 'ora.ASMNET1LSNR_ASM.lsnr' on 'cs2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'cs1' succeeded
CRS-2676: Start of 'ora.net1.network' on 'cs2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'cs1'
CRS-2672: Attempting to start 'ora.ons' on 'cs2'
CRS-2677: Stop of 'ora.scan1.vip' on 'cs1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'cs2'
CRS-2676: Start of 'ora.scan1.vip' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'cs2'
CRS-2676: Start of 'ora.ons' on 'cs2' succeeded
CRS-2676: Start of 'ora.ASMNET1LSNR_ASM.lsnr' on 'cs2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'cs2'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'cs2' succeeded
CRS-2676: Start of 'ora.asm' on 'cs2' succeeded
CRS-6016: Resource auto-start has completed for server cs2
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2018/03/09 17:42:11 CLSRSC-343: Successfully started Oracle Clusterware stack
2018/03/09 17:42:11 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
2018/03/09 17:42:24 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2018/03/09 17:43:16 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

到此这个问题就修复了

Oracle Cluster Time Management

Cluster Time Synchronization Service(CTSS)可以检测集群中节点之间的时间同步问题。CTSS被作为Oracle CLusterware的一部分进行安装。如果它检测到在系统中存在时间同步服务(比如NTP或Chrony)或者存在时间同步服务配置是valid或broken,那么它将以观察模式运行。例如,如果集群中的任何一个节点上 存在/etc/ntp.conf文件,那么就算没有运行时间同步服务CTSS也会以观察模式运行。如果CTSS在集群的任何一个节点上都没有检测到存在时间同步服务或时间同步服务配置信息,它就会变成主动模式运行并且对集群执行时间管理。如果CTSS在集群中的一个节点上以主动模式运行,在另一个节点上以观察模式运行,并且没有NTP,时间同步软件在运行,那么你可以通过创建一个名叫/etc/ntp.conf的文件来将CTSS修改为观察模式。对于CTSS改变为观察模式 会在alert.log中写入消息。当节点加入集群时,如果CTSS以主动模式运行时,那么它将会使用集群中存在引用时钟的节点与这些加入节点进行时间比较。如果两个时间之间存在差异并且这种差异在特定设置限制,那么CTSS将执行时间同步。集群中的时钟节点由于各种原因将不能使用引用时钟(CTSS用来作为基础的时间,它所在的节点是集群中第一个启动的节点)进行同步。当发生这种情况时,CTSS执行回转时间同步,它将加速或减慢节点上的系统时间直到它们与引 用系统时间处于同步状态。在这种时间同步方法中,CTSS不会向后调整时间,它将保证系统时间单向增加。

当Orale集群软件启动时,如果CTSS以主动模式运行并且时间差异超过了设置限制(限制为24小时),那么CTSS会在alert.log中生成一个告警,并退出,那么Oracle集群软件会启动失败。如果你必须手动对加入集群的节点进行时 间调整,在启动Oracle集群软件之后CTSS可以对这些节点管理时间。当执行回转时间同步时,CTSS从来不会使用引用时钟向后调整时间。CTSS定期写包含它如何使用引用时钟来调整节点时间的告警到alert.log中。

CTSS当出现以下情况下会向Oracle集群alert.log与syslog中写告警信息:
.检查到时间发生改变
.检查到与引用时钟相比存在着显著的时间差异
.从观察模式切换到主动模式或者从主动模式切换到观察模式

使用CTSS来同步集群中的时间可以促进对Oracle集群问题的诊断,因为不需要考虑不同节点上影响时间的一系列事件了。
激活与禁止集群时间同步
为了对集群设置定时间管理服务可以激活CTSS。如果想要使用不同的集群时间同步服务可以禁止CTSS。为了激活CTSS,必须要对集群中的所有节点停止与删除第三方的时间同步服务。当激活CTSS时会检查集群的时间管理服务方式。
下面的RAC因为所有节点存在第三方的时间同步服务配置信息使得CTSS以观察方式运行
检测集群节点是否存在第三方时间同步服务

[root@jytest1 ~]# find / -name ntp.conf
/etc/ntp.conf


[root@jytest2 ~]# find / -name ntp.conf
/etc/ntp.conf


[grid@jytest1 ~]$ cluvfy comp clocksync -n all

Verifying Clock Synchronization ...
CTSS is in Observer state. Switching over to clock synchronization checks using NTP

  Verifying Network Time Protocol (NTP) ...
    Verifying '/etc/ntp.conf' ...PASSED
    Verifying '/etc/chrony.conf' ...PASSED
    Verifying '/var/run/chronyd.pid' ...PASSED
    Verifying Daemon 'chronyd' ...PASSED
    Verifying NTP daemon or service using UDP port 123 ...PASSED
    Verifying chrony daemon is synchronized with at least one external time source ...FAILED (PRVG-13606)
  Verifying Network Time Protocol (NTP) ...FAILED (PRVG-1063)
Verifying Clock Synchronization ...FAILED (PRVG-1063, PRVG-13606)

Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.


Failures were encountered during execution of CVU verification request "Clock Synchronization across the cluster nodes".

Verifying Clock Synchronization ...FAILED
  Verifying Network Time Protocol (NTP) ...FAILED
  PRVG-1063 : configuration files for more than one time synchronization
  service were found on nodes of the cluster
  configuration file "/etc/chrony.conf" was found on nodes "jytest2,jytest1"
  configuration file "/etc/ntp.conf" was found on nodes "jytest2,jytest1"

    Verifying chrony daemon is synchronized with at least one external time
    source ...FAILED
    jytest2: PRVG-13606 : chrony daemon is not synchronized with any external
             time source on node "jytest2".

    jytest1: PRVG-13606 : chrony daemon is not synchronized with any external
             time source on node "jytest1".


CVU operation performed:      Clock Synchronization across the cluster nodes
Date:                         Mar 3, 2018 12:58:07 AM
CVU home:                     /u01/app/product/12.2.0/crs/
User:                         grid

[grid@jytest2 ~]$ cluvfy comp clocksync -n all

Verifying Clock Synchronization ...
CTSS is in Observer state. Switching over to clock synchronization checks using NTP

  Verifying Network Time Protocol (NTP) ...
    Verifying '/etc/ntp.conf' ...PASSED
    Verifying '/etc/chrony.conf' ...PASSED
    Verifying '/var/run/chronyd.pid' ...PASSED
    Verifying Daemon 'chronyd' ...PASSED
    Verifying NTP daemon or service using UDP port 123 ...PASSED
    Verifying chrony daemon is synchronized with at least one external time source ...FAILED (PRVG-13606)
  Verifying Network Time Protocol (NTP) ...FAILED (PRVG-1063)
Verifying Clock Synchronization ...FAILED (PRVG-1063, PRVG-13606)

Verification of Clock Synchronization across the cluster nodes was unsuccessful on all the specified nodes.


Failures were encountered during execution of CVU verification request "Clock Synchronization across the cluster nodes".

Verifying Clock Synchronization ...FAILED
  Verifying Network Time Protocol (NTP) ...FAILED
  PRVG-1063 : configuration files for more than one time synchronization
  service were found on nodes of the cluster
  configuration file "/etc/chrony.conf" was found on nodes "jytest2,jytest1"
  configuration file "/etc/ntp.conf" was found on nodes "jytest2,jytest1"

    Verifying chrony daemon is synchronized with at least one external time
    source ...FAILED
    jytest2: PRVG-13606 : chrony daemon is not synchronized with any external
             time source on node "jytest2".

    jytest1: PRVG-13606 : chrony daemon is not synchronized with any external
             time source on node "jytest1".


CVU operation performed:      Clock Synchronization across the cluster nodes
Date:                         Mar 3, 2018 1:11:12 AM
CVU home:                     /u01/app/product/12.2.0/crs/
User:                         grid

使用grid用户执行crsctl check ctss命令来查看CTSS的运行模式,命令结果显示为观察模式

[grid@jytest1 ~]$ crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.

[grid@jytest2 ~]$ crsctl check ctss
CRS-4700: The Cluster Time Synchronization Service is in Observer mode.

现在删除集群节点中的第三方时间同步服务

[root@jytest1 ~]# rm -rf /etc/ntp.conf
[root@jytest1 ~]# rm -rf /etc/chrony.conf
[root@jytest1 ~]# rm -rf /var/run/chronyd.pid


[root@jytest2 ~]# rm -rf /etc/ntp.conf
[root@jytest2 ~]# rm -rf /etc/chrony.conf
[root@jytest2 ~]# rm -rf /var/run/chronyd.pid

再次检查CTSS的运行模式,可以看到CTSS从观察模式变为了主动模式

[grid@jytest1 ~]$ crsctl check ctss
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 100
[grid@jytest1 ~]$ cluvfy comp clocksync -n all

Verifying Clock Synchronization ...PASSED

Verification of Clock Synchronization across the cluster nodes was successful.

CVU operation performed:      Clock Synchronization across the cluster nodes
Date:                         Mar 3, 2018 1:14:39 AM
CVU home:                     /u01/app/product/12.2.0/crs/
User:                         grid

[grid@jytest2 ~]$ crsctl check ctss
CRS-4701: The Cluster Time Synchronization Service is in Active mode.
CRS-4702: Offset (in msec): 0
[grid@jytest2 ~]$ cluvfy comp clocksync -n all

Verifying Clock Synchronization ...PASSED

Verification of Clock Synchronization across the cluster nodes was successful.

CVU operation performed:      Clock Synchronization across the cluster nodes
Date:                         Mar 3, 2018 1:15:14 AM
CVU home:                     /u01/app/product/12.2.0/crs/
User:                         grid