hp rx6600两台oracle双机互备服务器其中一台经常自动关机的故障诊断

hp rx6600两台oracle数据库双机互备服务器其中一台经常自动关机,刚好在做巡检时遇到了就顺便检查一下原因.检查经常出故障的一台小机日志信息如下:

rx6600-1:[/]#cat /var/adm/syslog/syslog.log
Nov  6 10:40:35 rx6600-1 syslogd: restart
Nov  6 10:40:35 rx6600-1 vmunix: Found adjacent data tr.  Growing size.  0x32a6000 -> 0x72a6000.
Nov  6 10:40:35 rx6600-1 vmunix: Pinned PDK malloc pool: base: 0xe000000100d5a000  size=117400K
Nov  6 10:40:35 rx6600-1 vmunix: Loaded ACPI revision 2.0 tables.
Nov  6 10:40:35 rx6600-1 vmunix: MMIO on this platform supports Write Coalescing.
Nov  6 10:40:35 rx6600-1 vmunix: 
Nov  6 10:40:35 rx6600-1 vmunix: MFS is defined: base= 0xe000000100d5a000  size= 5084 KB
Nov  6 10:40:35 rx6600-1 vmunix: Unpinned PDK malloc pool: base: 0xe000000108000000  size=393216K
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: cachefs_link(): File system was registered at index 5.
Nov  6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: GPX emcpgpx_install() success.
Nov  6 10:40:35 rx6600-1 vmunix: 
Nov  6 10:40:35 rx6600-1  above message repeats 2 times
Nov  6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: DM emcpgpx_dm_install() success.
Nov  6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: VLUMD emcpgpx_vlumd_install() success.
Nov  6 10:40:35 rx6600-1 vmunix: emcp:GPX:Info: XCRYPT emcpgpx_xcrypt_install() success.
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: nfs3_link(): File system was registered at index 8.
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: mod_fs_reg: Cannot retrieve configured loading phase from KRS for module: cifs. Setting to load at INIT
Nov  6 10:40:35 rx6600-1 vmunix: 
Nov  6 10:40:35 rx6600-1 vmunix: 0 sba
Nov  6 10:40:35 rx6600-1 vmunix: 0/0 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/1/0 rmp3f01
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/1/1 rmp3f01
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/1/2 asio0
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/0 UsbOhci
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached.  Identification String: 
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/hp/Unknown/0_1
Nov  6 10:40:35 rx6600-1 vmunix:  <2.1.3.10.1008.4390.1>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/0.0 UsbMiniBus
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Keyboard/USB/Boot/hp/Unknown/0_1
Nov  6 10:40:35 rx6600-1 vmunix:  <2.305.3.100.1008.4390.1>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/0.0.0 UsbBootKeyboard
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Mouse/USB/Standard/hp/Unknown/0_1
Nov  6 10:40:35 rx6600-1 vmunix:  <2.307.3.10.1008.4390.1>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1 UsbOhci
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/hp/Multibay/0_a1
Nov  6 10:40:35 rx6600-1 vmunix:  <2.1.3.10.1008.294.161>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0 UsbMiniBus
Nov  6 10:40:35 rx6600-1 vmunix: Devices/MassStorage-SCSI/USB/BulkOnly/hp/Multibay/0_a1
Nov  6 10:40:35 rx6600-1 vmunix:  <2.310.3.150.1008.294.161>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.0 UsbBulkOnlyMS
Nov  6 10:40:35 rx6600-1 vmunix: Devices/ScsiControllerAdaptor/USB/BulkOnly/hp/Multibay
Nov  6 10:40:35 rx6600-1 vmunix:  <2.1000.3.150.1008.294>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16 UsbScsiAdaptor
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached.  Identification String: 
Nov  6 10:40:36 rx6600-1  above message repeats 5 times
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.7 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.0.16.7.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached.  Identification String: 
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Device/USB/Standard/Avocent/KVMAdaptor/1_0
Nov  6 10:40:35 rx6600-1 vmunix:  <2.1.3.10.1572.833.256>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1 UsbMiniBus
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Keyboard/USB/Boot/Avocent/KVMAdaptor/1_0
Nov  6 10:40:35 rx6600-1 vmunix:  <2.305.3.100.1572.833.256>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1.0 UsbBootKeyboard
Nov  6 10:40:35 rx6600-1 vmunix: Devices/Mouse/USB/Boot/Avocent/KVMAdaptor/1_0
Nov  6 10:40:35 rx6600-1 vmunix:  <2.307.3.100.1572.833.256>
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/1.1.1 UsbBootMouse
Nov  6 10:40:35 rx6600-1 vmunix: NOTICE: USB device attached.  Identification String: 
Nov  6 10:40:36 rx6600-1  above message repeats 2 times
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/2/2 UsbEhci
Nov  6 10:40:35 rx6600-1 vmunix: 0/0/4/0 gvid_core
Nov  6 10:40:35 rx6600-1 vmunix: 0/1 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/2 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/2/1/0 PCItoPCI
Nov  6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/2/1/0/4/0 (FC Port 1 on HBA)
Nov  6 10:40:35 rx6600-1 vmunix: 0/2/1/0/4/0 fcd
Nov  6 10:40:35 rx6600-1 vmunix: 0/2/1/0/6/0 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/3 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/3/1/0 PCItoPCI
Nov  6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/3/1/0/4/0 (FC Port 1 on HBA)
Nov  6 10:40:35 rx6600-1 vmunix: 0/3/1/0/4/0 fcd
Nov  6 10:40:35 rx6600-1 vmunix: 0/3/1/0/6/0 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/4 lba
Nov  6 10:40:35 rx6600-1 vmunix: sasd: Claimed HP PCI/PCI-X SAS MPT adapter at hardware path 0/4/1/0 
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/1/0 sasd
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/2/0 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/2/1 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/5 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0 PCItoPCI
Nov  6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/5/1/0/4/0 (FC Port 1 on HBA)
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0 fcd
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/6/0 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/6 lba
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0 PCItoPCI
Nov  6 10:40:35 rx6600-1 vmunix: fcd: Claimed HP AD193-60001 4Gb Fibre Channel port at hardware path 0/6/1/0/4/0 (FC Port 1 on HBA)
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0 fcd
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/6/0 iether
Nov  6 10:40:35 rx6600-1 vmunix: 0/7 lba
Nov  6 10:40:35 rx6600-1 vmunix: Initializing the Ultra320 SCSI Controller at 0/7/1/0. Controller firmware version is 01.03.35.70
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/0 mpt
Nov  6 10:40:35 rx6600-1 vmunix: Initializing the Ultra320 SCSI Controller at 0/7/1/1. Controller firmware version is 01.03.35.70
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/1 mpt
Nov  6 10:40:35 rx6600-1 vmunix: 120 processor
Nov  6 10:40:35 rx6600-1 vmunix: 121 processor
Nov  6 10:40:35 rx6600-1 vmunix: 122 processor
Nov  6 10:40:35 rx6600-1 vmunix: 123 processor
Nov  6 10:40:35 rx6600-1 vmunix: 124 processor
Nov  6 10:40:35 rx6600-1 vmunix: 125 processor
Nov  6 10:40:35 rx6600-1 vmunix: 126 processor
Nov  6 10:40:35 rx6600-1 vmunix: 127 processor
Nov  6 10:40:35 rx6600-1 vmunix: 250 pdh
Nov  6 10:40:35 rx6600-1 vmunix: 250/0 ipmi
Nov  6 10:40:35 rx6600-1 vmunix: 250/1 asio0
Nov  6 10:40:35 rx6600-1 vmunix: 250/2 acpi_node
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/0.7 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/0.7.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1 fcd_fcp
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.13.255.0.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.255.0.0.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.1 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.2 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.3 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/5/1/0/4/0.1.9.0.0.0.4 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1 fcd_fcp
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0 fcd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.255.0.0.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.255.0.0.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.1 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.2 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.3 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.1 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.13.0.0.0.4 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.2 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.3 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/6/1/0/4/0.1.9.0.0.0.4 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/1.7 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/7/1/1.7.0 sctl
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0 sasd_vbus
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0.0 tgt
Nov  6 10:40:35 rx6600-1 vmunix: 0/4/1/0.0.0.0.0 sdisk
Nov  6 10:40:35 rx6600-1 vmunix: Boot device's HP-UX HW path is: 0/4/1/0.0.0.0.0
Nov  6 10:40:35 rx6600-1 vmunix: 
Nov  6 10:40:35 rx6600-1 vmunix:     System Console is on the Built-In Serial Interface
Nov  6 10:40:35 rx6600-1 vmunix: iether0: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/2/1/0/6/0
Nov  6 10:40:35 rx6600-1 vmunix: iether1: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/3/1/0/6/0
Nov  6 10:40:35 rx6600-1 vmunix: iether2: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path 0/4/2/0
Nov  6 10:40:35 rx6600-1 vmunix: iether4: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/5/1/0/6/0
Nov  6 10:40:35 rx6600-1 vmunix: iether5: INITIALIZING HP AD193-60001 PCI/PCI-X 1000Base-T 4Gb FC/1000B-T Combo Adapter at hardware path 0/6/1/0/6/0
Nov  6 10:40:35 rx6600-1 vmunix: iether3: INITIALIZING HP AB352-60003 PCI/PCI-X 1000Base-T Dual-port Core at hardware path 0/4/2/1
Nov  6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x3 configured as ROOT
Nov  6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x2 configured as SWAP
Nov  6 10:40:35 rx6600-1 vmunix: Logical volume 64, 0x2 configured as DUMP
Nov  6 10:40:35 rx6600-1 vmunix:     Swap device table:  (start & size given in 512-byte blocks)
Nov  6 10:40:35 rx6600-1 vmunix:         entry 0 - major is 64, minor is 0x2; start = 0, size = 16777216
Nov  6 10:40:35 rx6600-1 vmunix:     Dump device table:  (start & size given in 1-Kbyte blocks)
Nov  6 10:40:35 rx6600-1 vmunix:         entry 0000000000000000 - major is 31, minor is 0x30000; start = 2349940, size = 8388604
Nov  6 10:40:35 rx6600-1 vmunix: Starting the STREAMS daemons-phase 1
Nov  6 10:40:35 rx6600-1 vmunix: Create STCP device files
Nov  6 10:40:35 rx6600-1 vmunix: Starting the STREAMS daemons-phase 2
Nov  6 10:40:35 rx6600-1 vmunix:      $Revision: vmunix:    B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $
Nov  6 10:40:35 rx6600-1 vmunix: Memory Information:
Nov  6 10:40:35 rx6600-1 vmunix:     physical page size = 4096 bytes, logical page size = 4096 bytes
Nov  6 10:40:35 rx6600-1 vmunix:     Physical: 25133536 Kbytes, lockable: 18994328 Kbytes, available: 22051156 Kbytes
Nov  6 10:40:35 rx6600-1 vmunix: 
Nov  6 10:40:36 rx6600-1 nettl[832]: nettl starting up.
Nov  6 10:40:48 rx6600-1 sshd[986]: Server listening on :: port 22.
Nov  6 10:40:48 rx6600-1 sshd[986]: Server listening on 0.0.0.0 port 22.
Nov  6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found CLTS loopback transport
Nov  6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found COTS loopback transport
Nov  6 10:40:49 rx6600-1 rpcbind: check_netconfig: Found COTS ORD loopback transport
Nov  6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for udp
Nov  6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for tcp
Nov  6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticlts
Nov  6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticotsord
Nov  6 10:40:49 rx6600-1 rpcbind: init_transport: check binding for ticots
Nov  6 10:40:50 rx6600-1 inetd[1100]: Reading configuration
Nov  6 10:40:50 rx6600-1 inetd[1100]: ftp/tcp: Added service, server /usr/lbin/ftpd
Nov  6 10:40:50 rx6600-1 inetd[1100]: telnet/tcp: Added service, server /usr/lbin/telnetd
Nov  6 10:40:50 rx6600-1 inetd[1100]: tftp/udp: Added service, server /usr/lbin/tftpd
Nov  6 10:40:50 rx6600-1 inetd[1100]: login/tcp: Added service, server /usr/lbin/rlogind
Nov  6 10:40:50 rx6600-1 inetd[1100]: shell/tcp: Added service, server /usr/lbin/remshd
Nov  6 10:40:50 rx6600-1 inetd[1100]: exec/tcp: Added service, server /usr/lbin/rexecd
Nov  6 10:40:50 rx6600-1 inetd[1100]: ntalk/udp: Added service, server /usr/lbin/ntalkd
Nov  6 10:40:50 rx6600-1 inetd[1100]: auth/tcp: Added service, server /usr/lbin/identd
Nov  6 10:40:50 rx6600-1 inetd[1100]: printer/tcp: Added service, server /usr/sbin/rlpdaemon
Nov  6 10:40:51 rx6600-1 inetd[1100]: daytime/tcp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: daytime/udp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: time/tcp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: echo/tcp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: echo/udp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: discard/tcp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: discard/udp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: chargen/tcp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: chargen/udp: Added service, server internal
Nov  6 10:40:51 rx6600-1 inetd[1100]: kshell/tcp: Added service, server /usr/lbin/remshd
Nov  6 10:40:51 rx6600-1 inetd[1100]: klogin/tcp: Added service, server /usr/lbin/rlogind
Nov  6 10:40:51 rx6600-1 inetd[1100]: dtspc/tcp: Added service, server /usr/dt/bin/dtspcd
Nov  6 10:40:51 rx6600-1 inetd[1100]: recserv/tcp: Added service, server /usr/lbin/recserv
Nov  6 10:40:51 rx6600-1 inetd[1100]: swat/tcp: Added service, server /opt/samba/bin/swat
Nov  6 10:40:51 rx6600-1 inetd[1100]: registrar/tcp: Added service, server /etc/opt/resmon/lbin/registrar
Nov  6 10:40:51 rx6600-1 inetd[1100]: hacl-probe/tcp: Added service, server /opt/cmom/lbin/cmomd
Nov  6 10:40:51 rx6600-1 inetd[1100]: hacl-cfg/udp: Added service, server /usr/lbin/cmclconfd
Nov  6 10:40:51 rx6600-1 inetd[1100]: hacl-cfg/tcp: Added service, server /usr/lbin/cmclconfd
Nov  6 10:40:51 rx6600-1 inetd[1100]: instl_boots/udp: Added service, server /opt/ignite/lbin/instl_bootd
Nov  6 10:40:51 rx6600-1 inetd[1100]: omni/tcp: Added service, server /opt/omni/lbin/inet
Nov  6 10:40:51 rx6600-1 inetd[1100]: rpc.cmsd/udp: Added service, server /usr/dt/bin/rpc.cmsd
Nov  6 10:40:51 rx6600-1 inetd[1100]: rpc.ttdbserver/tcp: Added service, server /usr/dt/bin/rpc.ttdbserver
Nov  6 10:40:51 rx6600-1 inetd[1100]: Configuration complete
Nov  6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: restore  (user ID real=0 effective=0)
Nov  6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: config  (user ID real=0 effective=0)
Nov  6 10:40:53 rx6600-1 EMCPP: emcpAudit: Info: cmd=powermt: save  (user ID real=0 effective=0)
Nov  6 10:40:54 rx6600-1 su: + tty?? root-sfmdb
Nov  6 10:41:06 rx6600-1 cimserver[1706]: starting
Nov  6 10:41:29 rx6600-1 cimserver[1707]: PGS10026:  THE CIM SERVER IS LISTENING ON HTTPS PORT 5,989.
Nov  6 10:41:29 rx6600-1 cimserver[1707]: PGS10028: THE CIM SERVER IS LISTENING ON THE LOCAL CONNECTION SOCKET.
Nov  6 10:41:29 rx6600-1 cimserver[1707]: PGS10030:  STARTED HP-UX WBEM Services VERSION A.02.07.
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_japanese/100dpi/"
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_japanese/75dpi/"
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/hp_korean/75dpi/"
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Cannot initialize font path element: "/usr/lib/X11/fonts/hp_chinese_t/75dpi/"
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/ttfjpn.st"
Nov  6 10:41:32 rx6600-1 FontServer[1755]: Warning: Bad font path element: "/usr/lib/X11/fonts/ifojpn.st"
Nov  6 10:41:34 rx6600-1 pwgrd: Started at Thu Nov  6 10:41:34 2014, pid = 1798
Nov  6 10:41:34 rx6600-1 diagmond[1833]: started
Nov  6 10:41:34 rx6600-1 /usr/sbin/envd[1837]: VXPBFt6/, 2"6A3vEdVCND< ~
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2180]: Setting STREAMS-HEAD high water value to 131072.
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one mpctl succeeded: ncpus = 8.
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one pmap 2
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one pmap 3
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: nfsd do_one bind 0
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: nfsd do_one bind 1
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: nfsd do_one bind 2
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: nfsd do_one bind 3
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: nfsd do_one bind 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: nfsd do_one bind 5
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd do_one bind 7
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2195]: nfsd 5 1  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2197]: nfsd 5 0  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2193]: nfsd 3 1  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2192]: nfsd 2 1  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2200]: nfsd 2 0  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2191]: nfsd 1 1  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2201]: nfsd 1 0  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2199]: nfsd 3 0  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2194]: nfsd 4 1  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2202]: nfsd 4 0  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2185]: nfsd 7 1  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2219]: nfsd 7 0  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: nfsd do_one bind 6
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2190]: nfsd 0 1  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2220]: nfsd 0 0  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2196]: nfsd 6 1  sock 4
Nov  6 10:41:50 rx6600-1 /usr/sbin/nfsd[2221]: nfsd 6 0  sock 4
Nov  6 10:41:53 rx6600-1 krsd[2300]: Delay time is 300 seconds
Nov  6 10:41:53 rx6600-1 sfd[2301]: daemon already running.
Nov  6 10:41:54 rx6600-1 sfd[2314]: starting the daemon.
Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: New event pair [0] (2,4,60)
Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: New event pair [1] (20,40,300)
Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SetLogMask:: EventLogMask set to 0x66 
Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: Using hostname localhost community public debug 0
Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: Daemon created successfully.  Starting it now
Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SNMP trap processing disabled.
Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: PP Remote Management disabled.
Nov  6 10:45:17 rx6600-1 vmunix: emcp:Mpx:Info: PowerPath Auto Host Registration on VNX-FCN00125000137 is unavailable: incompatible initiator information received from the array
Nov  6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: ***** 9} HH AY =g >/ 8f *****
Nov  6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: NB6H3,9}U}3#9$WwAY=gV5, P^U}9}HHLu< ~!#
Nov  6 10:45:42 rx6600-1 EMS [2970]: ------ EMS Event Notification ------   Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw"     (Threshold:  >= " 3")   
 Execute the following command to obtain event details:   /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 
Nov  6 10:49:14 rx6600-1 EMS [2928]: ------ EMS Event Notification ------   Value: "CRITICAL (5)" for Resource: "/system/events/ipmi_fpl/ipmi_fpl"     (Threshold:  >= " 3")    
Execute the following command to obtain event details:   /opt/resmon/bin/resdata -R 191889410 -r /system/events/ipmi_fpl/ipmi_fpl -n 191889409 -a 
Nov  6 18:48:12 rx6600-1 EMS [2970]: ------ EMS Event Notification ------   Value: "CRITICAL (5)" for Resource: "/system/events/ia64_corehw/core_hw"     (Threshold:  >= " 3")    
Execute the following command to obtain event details:   /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641922 -a 
Nov  6 19:00:00 rx6600-1 su: + tty?? root-oracle
Nov  7 08:00:00 rx6600-1 su: + tty?? root-root

从如下信息看到服务器已经出问题了,且信息已经指出可以执行
/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 命令来查看详细信息

Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: SNMP trap processing disabled.
Nov  6 10:41:54 rx6600-1 emcp_mond: PP daemon: Info: PP Remote Management disabled.
Nov  6 10:45:17 rx6600-1 vmunix: emcp:Mpx:Info: PowerPath Auto Host Registration on VNX-FCN00125000137 is unavailable: incompatible initiator information received from the array
Nov  6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: ***** 9} HH AY =g >/ 8f *****
Nov  6 10:45:42 rx6600-1 /usr/sbin/envd[1837]: NB6H3,9}U}3#9$WwAY=gV5, P^U}9}HHLu< ~!#
Nov  6 10:45:42 rx6600-1 EMS [2970]: ------ EMS Event Notification ------   Value: "MAJORWARNING (3)" for Resource: "/system/events/ia64_corehw/core_hw"     (Threshold:  >= " 3")   
Execute the following command to obtain event details:   /opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 

执行/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 命令来查看详细信息

rx6600-1:[/]#/opt/resmon/bin/resdata -R 194641922 -r /system/events/ia64_corehw/core_hw -n 194641921 -a 

ARCHIVED MONITOR DATA:

Event Time..........: Thu Nov  6 10:45:42 2014
Severity............: MAJORWARNING
Monitor.............: ia64_corehw
Event #.............: 101011              
System..............: rx6600-1

Summary:
     System temperature is out of normal range. 


Description of Error:

     The system temperature is not within normal operating range. It is higher
     than required operating range.

这个错误描述是说系统的温度超出了正常范围,下面信息说明了可能的原因

Probable Cause / Recommended Action:

     Something may be blocking the cooling intakes of the fans. Check for
     obstruction.
     One or more fans may be operating at lower speed than normal. Check the
     fan performance.

     Check for problems with the room air conditioning.

     If the problem is not fixed, the operating temperature may become
     non-recoverable, in which case there are chances that the hardware may be
     damaged.  At that temperature level, on Integrity servers, the firmware
     will shutdown the system automatically. However on HP 9000 servers, the
     action specified in the envd config file will be taken - which may be to
     shutdown the system automatically.

     For information on the sensor that generated this event, refer to FRU ID
     in Event Details section.

上面的信息是说,可能需要清理一下风机,或者风机性能出现问题,或者检查空调情况,如果不是这些原因造成那么可能是硬件出现问题了。下面的论断事件的数据:

Additional Event Data: 
     System IP Address...: 10.138.129.5
     Event Id............: 0x545ae0d600000000
     Monitor Version.....: B.01.00
     Event Class.........: System
     Client Configuration File...........:
     /var/stm/config/tools/monitor/default_ia64_corehw.clcfg 
     Client Configuration File Version...: A.01.00 
          Qualification criteria met.
               Number of events..: 1 
     Associated OS error log entry id(s): 
          None
     Additional System Data:
          System Model Number.............: ia64 hp server rx6600 
          EMS Version.....................: A.04.20 
          STM Version.....................: C.58.00 
          System Serial Number............: SGH48045VY 
     Latest information on this event:
          http://docs.hp.com/hpux/content/hardware/ems/ia64_corehw.htm#101011

v-v-v-v-v-v-v-v-v-v-v-v-v    D  E  T  A  I  L  S    v-v-v-v-v-v-v-v-v-v-v-v-v


Event Details :

     Event Date .............: Thu Nov  6 10:44:08 2014
     Sensor Number ..........: 0xdb
     Sensor Type ............: Temperature
     Sensor Class ...........: Threshold based
     Sensor Reading/Offset...: 0x07 (Offset)
     Event  Type.............: Assertion
     Entity ID ..............: 3
     Generic Message.........: 
       Temperature :  Upper non-critical - going high 
     Entity FRU Id Info......: 
       processor (Sensor ID: Processor 2)

从上面的Event Details信息可以看到,传感器类型是温度方面的问题,传感器类别是基于阈值,事件类型是断言,是说2号cpu的温度已经超过了阈值.经过检查不是机房空调,通风口堵塞问题,需要联系小机厂商来进行一步检查是什么原因造成cpu温度超过阈值,平时cpu使用率只有10%。

Proudly powered by WordPress | Indrajeet by Sus Hill.