weblogic 10.3.3 java.lang.IllegalArgumentException: Failed to properly unregister weblogic.work.RequestClassRuntimeMBeanImpl故障解决一例

weblogic 10.3.3异常终止服务了检查日志文件发现如下错误信息:

weblogic.management.DeploymentException: 
  at weblogic.application.internal.flow.ApplicationRuntimeMBeanFlow.unprepare(ApplicationRuntimeMBeanFlow.java:64)
  at weblogic.application.internal.BaseDeployment$1.previous(BaseDeployment.java:1233)
  at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:167)
  at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:159)
  at weblogic.application.internal.BaseDeployment.unprepare(BaseDeployment.java:495)
  at weblogic.application.internal.SingleModuleDeployment.unprepare(SingleModuleDeployment.java:43)
  at weblogic.application.internal.DeploymentStateChecker.unprepare(DeploymentStateChecker.java:205)
  at weblogic.deploy.internal.targetserver.AppContainerInvoker.unprepare(AppContainerInvoker.java:117)
  at weblogic.deploy.internal.targetserver.BasicDeployment.unprepare(BasicDeployment.java:287)
  at weblogic.management.deploy.internal.DeploymentAdapter$1.doUnprepare(DeploymentAdapter.java:81)
  at weblogic.management.deploy.internal.DeploymentAdapter.unprepare(DeploymentAdapter.java:224)
  at weblogic.management.deploy.internal.AppTransition$7.transitionApp(AppTransition.java:75)
  at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
  at weblogic.management.deploy.internal.ConfiguredDeployments.unprepare(ConfiguredDeployments.java:204)
  at weblogic.management.deploy.internal.ConfiguredDeployments.undeploy(ConfiguredDeployments.java:192)
  at weblogic.management.deploy.internal.DeploymentServerService.shutdownApps(DeploymentServerService.java:195)
  at weblogic.management.deploy.internal.DeploymentServerService.shutdownHelper(DeploymentServerService.java:127)
  at weblogic.application.ApplicationShutdownService.halt(ApplicationShutdownService.java:142)
  at weblogic.t3.srvr.ServerServicesManager.haltInternal(ServerServicesManager.java:504)
  at weblogic.t3.srvr.ServerServicesManager.halt(ServerServicesManager.java:336)
  at weblogic.t3.srvr.T3Srvr.shutdown(T3Srvr.java:1039)
  at weblogic.t3.srvr.T3Srvr.forceShutdown(T3Srvr.java:945)
  at weblogic.t3.srvr.T3Srvr$2.run(T3Srvr.java:958)

Caused By: java.lang.IllegalArgumentException: Failed to properly unregister weblogic.work.RequestClassRuntimeMBeanImpl@50b20090 for ObjectName com.bea:ServerRuntime=AdminServer,Name=default@plat_changde_test@null,WorkManagerRuntime=default,ApplicationRuntime=plat_changde_test,Type=RequestClassRuntime
  at weblogic.management.jmx.ObjectNameManagerBase.unregisterObject(ObjectNameManagerBase.java:219)
  at weblogic.management.jmx.ObjectNameManagerBase.unregisterObjectInstance(ObjectNameManagerBase.java:192)
  at weblogic.management.mbeanservers.internal.RuntimeMBeanAgent$1.unregisteredInternal(RuntimeMBeanAgent.java:124)
  at weblogic.management.mbeanservers.internal.RuntimeMBeanAgent$1.unregistered(RuntimeMBeanAgent.java:108)
  at weblogic.management.provider.core.RegistrationManagerBase.invokeRegistrationHandlers(RegistrationManagerBase.java:187)
  at weblogic.management.provider.core.RegistrationManagerBase.unregister(RegistrationManagerBase.java:126)
  at weblogic.management.runtime.RuntimeMBeanDelegate.unregister(RuntimeMBeanDelegate.java:287)
  at weblogic.management.runtime.RuntimeMBeanDelegate.unregisterChildren(RuntimeMBeanDelegate.java:350)
  at weblogic.management.runtime.RuntimeMBeanDelegate.unregister(RuntimeMBeanDelegate.java:274)
  at weblogic.management.runtime.RuntimeMBeanDelegate.unregisterChildren(RuntimeMBeanDelegate.java:350)
  at weblogic.management.runtime.RuntimeMBeanDelegate.unregister(RuntimeMBeanDelegate.java:274)
  at weblogic.j2ee.J2EEApplicationRuntimeMBeanImpl.unregister(J2EEApplicationRuntimeMBeanImpl.java:359)
  at weblogic.application.internal.flow.ApplicationRuntimeMBeanFlow.unprepare(ApplicationRuntimeMBeanFlow.java:62)
  at weblogic.application.internal.BaseDeployment$1.previous(BaseDeployment.java:1233)
  at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:167)
  at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:159)
  at weblogic.application.internal.BaseDeployment.unprepare(BaseDeployment.java:495)
  at weblogic.application.internal.SingleModuleDeployment.unprepare(SingleModuleDeployment.java:43)
  at weblogic.application.internal.DeploymentStateChecker.unprepare(DeploymentStateChecker.java:205)
  at weblogic.deploy.internal.targetserver.AppContainerInvoker.unprepare(AppContainerInvoker.java:117)
  at weblogic.deploy.internal.targetserver.BasicDeployment.unprepare(BasicDeployment.java:287)
  at weblogic.management.deploy.internal.DeploymentAdapter$1.doUnprepare(DeploymentAdapter.java:81)
  at weblogic.management.deploy.internal.DeploymentAdapter.unprepare(DeploymentAdapter.java:224)
  at weblogic.management.deploy.internal.AppTransition$7.transitionApp(AppTransition.java:75)
  at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
  at weblogic.management.deploy.internal.ConfiguredDeployments.unprepare(ConfiguredDeployments.java:204)
  at weblogic.management.deploy.internal.ConfiguredDeployments.undeploy(ConfiguredDeployments.java:192)
  at weblogic.management.deploy.internal.DeploymentServerService.shutdownApps(DeploymentServerService.java:195)
  at weblogic.management.deploy.internal.DeploymentServerService.shutdownHelper(DeploymentServerService.java:127)
  at weblogic.application.ApplicationShutdownService.halt(ApplicationShutdownService.java:142)
  at weblogic.t3.srvr.ServerServicesManager.haltInternal(ServerServicesManager.java:504)
  at weblogic.t3.srvr.ServerServicesManager.halt(ServerServicesManager.java:336)
  at weblogic.t3.srvr.T3Srvr.shutdown(T3Srvr.java:1039)
  at weblogic.t3.srvr.T3Srvr.forceShutdown(T3Srvr.java:945)
  at weblogic.t3.srvr.T3Srvr$2.run(T3Srvr.java:958)

故障原因

Caused By: java.lang.IllegalArgumentException: Failed to properly unregister weblogic.work.RequestClassRuntimeMBeanImpl@50b20090 for ObjectName com.bea:ServerRuntime=AdminServer,Name=default@plat_changde_test@null,WorkManagerRuntime=default,ApplicationRuntime=plat_changde_test,Type=RequestClassRuntime

在MOS上找到一篇关于这个错误的文章说原因是

When you have JDBC connection pool name same as application name in config.xml we are running into the issue.

在config.xml文件中确实存在jb_zs,jb_test,plat_changde_test,plat_changde的应用名与连接池名相同将其修改为不一样后重新weblogic解决了此问题.

oracle IO性能问题故障诊断案例

一业务系统在白天业务时间出现了严重了IO性能问题,下面是下午业务高峰时间(3-5)的awr报告
1

从等待事件来看主要都是与IO相关
2

3

4

从上面可以看到除了几个语句的逻辑读很高,其实物理不是很高,每秒产生的重做日志以及物理读也不高.

检查磁盘IO

rx6600-1:[/]#sar -d 1 10

HP-UX rx6600-1 B.11.23 U ia64    07/15/14

16:18:45   device   %busy   avque   r+w/s  blks/s  avwait  avserv
16:18:46  c39t0d3  100.00    0.50      18    1130    0.00  132.11
          c41t0d3   83.50    0.50       6     450    0.00  290.78
16:18:47   c3t0d0    0.99    0.50       2      63    0.00    7.12
          c39t0d3   91.09    0.50      10     982    0.00  115.53
          c41t0d3  100.00    0.50      12     586    0.00  291.67
16:18:48   c3t0d0    3.03    0.50       2      32    0.00   15.93
          c39t0d3  100.00    0.50       9    1034    0.00  139.76
          c41t0d3   92.93    0.50       7     388    0.00  310.07
16:18:49   c3t0d0    2.00    0.50       4      64    0.00   19.59
          c39t0d3  100.00    0.50      12    1088    0.00  127.33
          c41t0d3   86.00    0.50       8     416    0.00  251.32
16:18:50   c3t0d0    1.01    0.50       1       2    0.00    8.99
          c39t0d3  100.00    0.50      16     954    0.00  117.10
          c41t0d3  100.00    0.50       9     614    0.00  295.52
16:18:51   c3t0d0    0.99    0.50       1       8    0.00   10.60
          c39t0d3   93.07    0.50      17     913    0.00  110.59
          c41t0d3  100.00    0.50       9     350    0.00  326.92
16:18:52  c39t0d3  100.00    0.50      21    1168    0.00  127.22
          c41t0d3   88.00    0.50      11     544    0.00  252.08
16:18:53   c3t0d0    2.02    0.50       3      48    0.00   18.51
          c39t0d3   88.89    0.50      19    1164    0.00   98.25
          c41t0d3  100.00    0.50      11     630    0.00  324.39
16:18:54   c3t0d0    3.00    0.50       3      20    0.00   12.39
          c39t0d3   95.00    0.50      20     954    0.00  131.90
          c41t0d3   81.00    0.50       9     610    0.00  289.05
16:18:55   c3t0d0    9.00    0.50      11     134    0.00    8.62
          c39t0d3  100.00    0.50      19    1090    0.00  137.20
          c41t0d3  100.00    0.50      11     512    0.00  327.16

Average   c39t0d3   99.50    0.50      16    1048    0.00  123.38
Average   c41t0d3  100.00    0.50       9     510    0.00  296.44
Average    c3t0d0    2.20    0.50       3      37    0.00   12.28
rx6600-1:[/]#sar -d 1 10

HP-UX rx6600-1 B.11.23 U ia64    07/15/14

16:20:04   device   %busy   avque   r+w/s  blks/s  avwait  avserv
16:20:05   c3t0d0    1.00    0.50       1      16    0.00    8.33
          c39t0d3   98.00    0.50      16     928    0.00  114.86
          c41t0d3   98.00    0.50      10     684    0.00  266.43
16:20:06   c3t0d0    1.98    0.50       4      81    0.00    8.57
          c39t0d3   93.07    0.50      19    1251    0.00  128.81
          c41t0d3   91.09    0.50       6     475    0.00  365.83
16:20:07   c3t0d0    2.00    0.50       3      48    0.00    5.87
          c39t0d3   98.00    0.50      23    1216    0.00  113.66
          c41t0d3   98.00    0.50       8     576    0.00  307.92
16:20:08   c3t0d0    1.00    0.50       2      32    0.00    5.36
          c39t0d3  100.00    0.50      21    1132    0.00  118.47
          c41t0d3  100.00    0.50       7     592    0.00  300.71
16:20:09   c3t0d0    6.00    0.58      13     194    2.22   26.05
          c39t0d3   89.00    0.50      17    1152    0.00  123.54
          c41t0d3   87.00    0.50       8     512    0.00  298.26
16:20:10   c3t0d0    3.00    0.50       6      96    0.00   22.78
          c39t0d3   85.00    0.50      17    1136    0.00  114.79
          c41t0d3   98.00    0.50       9     592    0.00  252.52
16:20:11   c3t0d0    1.00    0.50       1       2    0.00    8.04
          c39t0d3  100.00    0.50      17    1216    0.00  138.04
          c41t0d3  100.00    0.50      12     672    0.00  291.69
16:20:12   c3t0d0    2.00    0.50       3      34    0.00    9.24
          c39t0d3   99.00    0.50      16    1024    0.00  122.11
          c41t0d3   88.00    0.50       9     476    0.00  299.79
16:20:13  c39t0d3   91.00    0.50      18    1024    0.00  111.77
          c41t0d3   92.00    0.50       3     384    0.00  396.25
16:20:14  c39t0d3   99.00    0.50      17     892    0.00  132.15
          c41t0d3  100.00    0.50      10     608    0.00  233.54

Average    c3t0d0    1.80    0.53       3      50    0.87   17.64
Average   c39t0d3   96.00    0.50      18    1097    0.00  121.54
Average   c41t0d3  100.00    0.50       8     557    0.00  290.35

在业务人员下班后重启的双机软件,但在启动数据库时停在了Completed redo application这一步

SQL> startup
ORACLE instance started.

Total System Global Area 1.0318E+10 bytes
Fixed Size                  2073176 bytes
Variable Size            3238006184 bytes
Database Buffers         7063207936 bytes
Redo Buffers               14700544 bytes
Database mounted.

从alert.log文件中可以看到如下信息:

Tue Jul 15 22:23:29 2014
Starting ORACLE instance (normal)
LICENSE_MAX_SESSION = 0
LICENSE_SESSIONS_WARNING = 0
Picked latch-free SCN scheme 3
Autotune of undo retention is turned on. 
IMODE=BR
ILAT =61
LICENSE_MAX_USERS = 0
SYS auditing is disabled
ksdpec: called for event 13740 prior to event group initialization
Starting up ORACLE RDBMS Version: 10.2.0.4.0.
System parameters with non-default values:
  processes                = 500
  sessions                 = 555
  __shared_pool_size       = 3154116608
  __large_pool_size        = 16777216
  __java_pool_size         = 33554432
  __streams_pool_size      = 33554432
  sga_target               = 10317987840
  control_files            = /sx_data/ORCL/control01.ctl, /sx_data/ORCL/control02.ctl, /sx_data/ORCL/control03.ctl
  db_block_size            = 8192
  __db_cache_size          = 7063207936
  compatible               = 10.2.0.3.0
  log_archive_dest_1       = LOCATION=/sx_data/arch_ORCL/
  log_archive_format       = %t_%s_%r.dbf
  db_file_multiblock_read_count= 16
  db_recovery_file_dest    = /oracle/flash_recovery_area
  db_recovery_file_dest_size= 2147483648
  undo_management          = AUTO
  undo_tablespace          = UNDOTBS1
  undo_retention           = 39600
  fast_start_parallel_rollback= FALSE
  remote_login_passwordfile= EXCLUSIVE
  db_domain                = 
  dispatchers              = (PROTOCOL=TCP) (SERVICE=ORCLXDB)
  local_listener           = ORCL
  job_queue_processes      = 10
  background_dump_dest     = /oracle/admin/ORCL/bdump
  user_dump_dest           = /oracle/admin/ORCL/udump
  core_dump_dest           = /oracle/admin/ORCL/cdump
  audit_file_dest          = /oracle/admin/ORCL/adump
  db_name                  = ORCL
  open_cursors             = 2000
  optimizer_index_cost_adj = 20
  optimizer_index_caching  = 90
  pga_aggregate_target     = 2576351232
PMON started with pid=2, OS id=13613
PSP0 started with pid=3, OS id=13615
MMAN started with pid=4, OS id=13617
DBW0 started with pid=5, OS id=13619
LGWR started with pid=6, OS id=13621
CKPT started with pid=7, OS id=13623
SMON started with pid=8, OS id=13625
RECO started with pid=9, OS id=13627
CJQ0 started with pid=10, OS id=13629
MMON started with pid=11, OS id=13631
Tue Jul 15 22:23:30 2014
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=12, OS id=13635
Tue Jul 15 22:23:30 2014
starting up 1 shared server(s) ...
Tue Jul 15 22:23:31 2014
ALTER DATABASE   MOUNT
Tue Jul 15 22:23:39 2014
Setting recovery target incarnation to 2
Tue Jul 15 22:23:42 2014
Successful mount of redo thread 1, with mount id 1380841571
Tue Jul 15 22:23:42 2014
Database mounted in Exclusive Mode
Completed: ALTER DATABASE   MOUNT
Tue Jul 15 22:23:42 2014
ALTER DATABASE OPEN
Tue Jul 15 22:23:47 2014
Beginning crash recovery of 1 threads
 parallel recovery started with 7 processes
Tue Jul 15 22:23:50 2014
Started redo scan
Tue Jul 15 22:23:52 2014
Completed redo scan
 336597 redo blocks read, 78835 data blocks need recovery
Tue Jul 15 22:23:52 2014
Started redo application at
 Thread 1: logseq 2270, block 29
Tue Jul 15 22:23:53 2014
Recovery of Online Redo Log: Thread 1 Group 4 Seq 2270 Reading mem 0
  Mem# 0: /sx_data/ORCL/redo04.log
Tue Jul 15 22:23:58 2014
Completed redo application

一直停在Completed redo application这,而这时的等待事件是checkpoint complete开始以为是并行恢复慢造成的,就查询了v$transaction,v$fast_start_transactions但视图中并没有进行恢复操作的事务存在.后来咨询了老熊,老熊说检查一下IO情况看是不是存储出问题了,如是再次检查存储IO性能:

rx6600-1:[/]#sar 1 10

HP-UX rx6600-1 B.11.23 U ia64    07/15/14

22:36:41    %usr    %sys    %wio   %idle
22:36:42       2       2      12      84
22:36:43       1       0      12      87
22:36:44       0       0      17      83
22:36:45       0       0      13      87
22:36:46       0       1      12      87
22:36:47       2       1      13      84
22:36:48       1       1      16      82
22:36:49       0       0      12      88
22:36:50       0       0      12      88
22:36:51       0       0      22      78

Average        1       0      14      85

从上面可以看到现在实际上并没有业务在跑居然还存在IO等待这是不正常的

rx6600-2:[/]#bdf
Filesystem          kbytes    used   avail %used Mounted on
/dev/vg00/lvol3     983040  422504  556176   43% /
/dev/vg00/lvol1    1835008  135048 1686776    7% /stand
/dev/vg00/lvol8    8912896 8535352  374824   96% /var
/dev/vg00/lvol7    7962624 2762312 5159704   35% /usr
/dev/vg00/lvol4     524288   83192  437784   16% /tmp
/dev/vg00/tmplv    2064384   93512 1847942    5% /oratmp
/dev/vg00/orasoft  10256384 3144652 6668390   32% /orasoft
/dev/vg00/oracle   20480000 5480497 14062042   28% /oracle
/dev/vg00/lvol6    9076736 5206384 3840128   58% /opt
/dev/vg00/lvol5     131072   25472  104824   20% /home
/dev/cwjcvg/cwjc_datalv
                   414973952 134188114 263239842   34% /cwjc_data
/dev/sxvg/sx_datalv
                   624689152 298665485 305654147   49% /sx_data

rx6600-2:[/]#time dd if=/dev/zero of=/var/test bs=8k count=100000

下面对小机自身的磁盘进行IO测试写800M的数据只要12秒左右

msgcnt 2 vxfs: mesg 001: vx_nospace - /dev/vg00/lvol8 file system full (1 block extent)
I/O error 
47185+0 records in
47184+1 records out

real       11.7
user        0.0
sys         0.8

但是对EMC存储进行IO测试写800M的数据只要30多分还没有完成

rx6600-2:[/]#time dd if=/dev/zero of=/sx_data/test bs=8k count=100000
711856+0 records in
711855+0 records out

real    30:58.4
user        0.5
sys        13.0

这明显的是存储出了问题,后面得知管理人员早上10点多发现了存储有一个磁盘损坏了,存储做的raid 5,有热备盘.而且还有上百G的数据进行存储级的同步.与出现性能问题的时间一至。

到此问题原因找到了解决起来也就简单了.幸亏问题解决了,第二天有大领导来检查要不就…….哈哈

WLS 10.3.0 java.lang.IllegalArgumentException Registered more than one instance部署程序异常终止

单位上新上一应用在weblogic控制台进行应用程序更新发布出现
java.lang.IllegalArgumentException Registered more than one instance异常,然后weblogic就异常退出.

查看AdminServer.log日志可以看到如下信息:


####<2014-7-10 下午04时28分11秒 CST>     < [ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'>  <> <> <1404980891082>   
####<2014-7-10 下午04时28分21秒 CST>     < [ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'>  <> <> <1404980901145>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901148>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901148>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901149>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901173>  < [CompressingFilter/1.4.6] CompressingFilter is being destroyed...> 
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901176>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901177>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901177>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901178>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901178>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901190>  (RuntimeMBeanDelegate.java:255)
	at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:215)
	at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:193)
	at weblogic.work.WorkManagerRuntimeMBeanImpl.(WorkManagerRuntimeMBeanImpl.java:49)
	at weblogic.work.WorkManagerRuntimeMBeanImpl.getWorkManagerRuntime(WorkManagerRuntimeMBeanImpl.java:59)
	at weblogic.work.WorkManagerCollection.addWorkManagerRuntime(WorkManagerCollection.java:774)
	at weblogic.work.WorkManagerCollection.initialize(WorkManagerCollection.java:187)
	at weblogic.application.internal.flow.WorkManagerFlow.prepare(WorkManagerFlow.java:45)
	at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:1221)
	at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
	at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:367)
	at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:43)
	at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:154)
	at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:60)
	at weblogic.deploy.internal.targetserver.operations.RedeployOperation.createAndPrepareContainer(RedeployOperation.java:98)
	at weblogic.deploy.internal.targetserver.operations.RedeployOperation.doPrepare(RedeployOperation.java:122)
	at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:217)
	at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:747)
	at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1216)
	at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:250)
	at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:159)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
	at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
> 
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901192>  (RuntimeMBeanDelegate.java:255)
	at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:215)
	at weblogic.work.RequestClassRuntimeMBeanImpl.(RequestClassRuntimeMBeanImpl.java:32)
	at weblogic.work.WorkManagerRuntimeMBeanImpl.getRequestClassRuntime(WorkManagerRuntimeMBeanImpl.java:86)
	at weblogic.work.WorkManagerRuntimeMBeanImpl.getWorkManagerRuntime(WorkManagerRuntimeMBeanImpl.java:61)
	at weblogic.work.WorkManagerCollection.addWorkManagerRuntime(WorkManagerCollection.java:774)
	at weblogic.work.WorkManagerCollection.initialize(WorkManagerCollection.java:187)
	at weblogic.application.internal.flow.WorkManagerFlow.prepare(WorkManagerFlow.java:45)
	at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:1221)
	at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
	at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:367)
	at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:43)
	at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:154)
	at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:60)
	at weblogic.deploy.internal.targetserver.operations.RedeployOperation.createAndPrepareContainer(RedeployOperation.java:98)
	at weblogic.deploy.internal.targetserver.operations.RedeployOperation.doPrepare(RedeployOperation.java:122)
	at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:217)
	at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:747)
	at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1216)
	at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:250)
	at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:159)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
	at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)


从上面的错误信息可以看到

 redeploy operation for application, hninsiis

这说明是在更新应用程序重新加载hninsiis应用

java.lang.IllegalArgumentException: Registered more than one instance with the same objectName : com.bea:ServerRuntime=AdminServer,Name=default@hninsiis@null,WorkManagerRuntime=default,ApplicationRuntime=hninsiis,Type=RequestClassRuntime new:weblogic.work.RequestClassRuntimeMBeanImpl@1470f4af existing weblogic.work.RequestClassRuntimeMBeanImpl@3702476f

这个信息是说在加载应用程序时已经存在一个同名的应用程序,而实际上并不存在同名的应用程序

这时只能使用无所不能的MOS了,在MOS上找到了关于这个错误信息的文档


WLS 10.3.0: java.lang.IllegalArgumentException: Existing Timer Manager Has Different Work Manager (文档 ID 1097661.1)

In this Document
Symptoms
Cause
Solution
References

APPLIES TO:

Oracle Weblogic Server - Version 10.3 to 10.3
Information in this document applies to any platform.
***Checked for relevance on 2-May-2013***
SYMPTOMS

When trying to redeploy an application that uses a foreign JMS queue, the error below is shown. The server needs to be rebooted with every new deployment. Otherwise, the server fails consistently with the error: "Existing timer manager has different work manager."

###     < [ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1272310540224>  
CAUSE

This issue is caused by a flaw in the WLS code such that all MDB instances/pollers on the same WLS server of the same MDB deployment share the same timer manager for message polling. When one instance is disconnected due to JMS server migration, the timer manager is stopped, which in turn stops other MDB instances on the same server. This issue is addressed by unpublished defect 7669814.

SOLUTION

Patches are available for unpublished defect 7669814:

PATCH INFORMATION
WLS Version	Patch Number
10.3.0	Patch 7669814
Fixed in: 10.3.1

To apply one of these patches, click on the link for your WLS version and download the appropriate patch from My Oracle Support. You can also search in My Oracle Support for the patch number for your WLS version: for detailed instructions, please see Master Note: How to Locate and Download Patches for WebLogic Server Using My Oracle Support Document 1302053.1. For instructions on how to apply these patches to your system, please see How to Apply WebLogic Server (WLS) Patches Using Smart Update Document 876004.1. For other issues relating to Smart Update, please see Master Note on Troubleshooting Smart Update Issues Document 1230725.1.

Patches are specifically tied to a particular release and maintenance pack of WebLogic Server (WLS). A patch for WLS 10.3.3, for example, very likely would not work on WLS 10.3.5. In a few cases, patches are also specific to particular operating systems. If you think you are experiencing the problem outlined in this note, and you are running a WLS version which is eligible for error correction (see Document 950131.1 for more about the Oracle Error Correction Policy), but your WLS version is not included in the list of patches provided here, please contact support and ask for a version of the patch appropriate for you. Please reference this note as well as this bug number to help speed our service for you.

NOTE: Patches are applied per WLS installation and not per domain. That is, if you apply this patch on one WLS installation, then all of the servers from all the domains in that installation will have this patch. On the other hand, if you have a managed server in another machine in a domain (that is, set up with its own WLS installation), you need to install this patch on that other machine as well. Generally, patches can only be applied while the server is not running because WLS locks the needed files while it is running. If, however, you are able to apply a patch while WLS is running, you must restart WLS before the patch will take effect.


从上面的信息说已经在10.3.1这个版本已经修复这个bug了.但我们的weblogic版本是10.3.3.0

[root@sx-weblogic31 lib]# java -cp weblogic.jar weblogic.version

WebLogic Server 10.3.3.0  Fri Apr 9 00:05:28 PDT 2010 1321401 

Use 'weblogic.version -verbose' to get subsystem information

Use 'weblogic.utils.Versions' to get version information for all modules


其实给weblogic打补丁提供了两种方法

1. Using Smart Update

You can apply the patch using Smart Update with the following steps:

Download the patch from My Oracle Support (MOS). For more details, please refer to Master Note: How to Locate and Download Patches for WebLogic Server Using My Oracle Support Note 1302053.1.
Extract the contents from the zip file: you will have a jar file and patch-catalog_xxx.xml. A readme file may also be included.
Copy the files (for example, E5W8.jar and WGQJ.jar) and the patch-catalog_xxx.xml from the zip file to the target machine. You do not need the readme file. Copy the files to the appropriate cache_dir directory for the target system: for example, on Windows, %MIDDLEWARE_HOME%\utils\bsu\cache_dir, or on UNIX, $MIDDLEWARE_HOME/utils/bsu/cache_dir. The directory MW_HOME\utils\bsu\cache_diris created as the default patch download directory when you install Smart Update 3.3.0. (see http://docs.oracle.com/cd/E14759_01/doc.32/e14143/start.htm#i1071723)

NOTE: Always copy the patch-catalog_xxx.xml file from the downloaded patch to the cache_dir along with the patch itself. Do NOTrename this file. If you see an "unrecognized patch ID" error, please refer to Note 1186923.1 for details on how to resolve it.
Run Smart Update and apply the patches and/or patch sets to the target system. This can be done using the Smart Update GUI or the command-line interface (see http://download.oracle.com/docs/cd/E14759_01/doc.32/e14143/commands.htm#i1074489).
a. Smart Update in graphical (GUI) mode

Run the /utils/bsu/bsu script (bsu.sh for UNIX, bsu.cmd for Windows). This will start the Smart Update GUI.
Look for the patches you copied in the "Downloaded Patches" section at the bottom.
Select the "Apply" button for each patch you want to apply. This will validate the patch and apply it to the whole installation.
The following viewlet provides an example of using Smart Update in GUI mode:

 Video - Applying a Patch Using Smart Update in GUI Mode (1:15) Trouble seeing this video?

b. Command-line interface

This is the syntax for the command to view the downloaded patches as below:
./bsu.sh -prod_dir= -patch_download_dir= -status=downloaded -view -verbose
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patch_download_dir=/opt/bea/utils/bsu/cache_dir -status=downloaded -view -verbose
This is the syntax for the command to install a patch:
./bsu.sh -prod_dir= -patchlist= -verbose -install
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patchlist=E5W8 -verbose -install
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patchlist=E5W8,WGQJ -verbose -install
This is the syntax for the command to check if the patch is installed:
./bsu.sh -prod_dir= -patch_download_dir= -status=applied -verbose -view
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -status=applied -verbose -view
2. Applying the patch to the classpath manually

You can apply the patch to the system manually by extracting the actual patch and adding it to the classpath on the system:

Extract the actual patch jar file. It will be in the form .jar (for example: E5W8.jar). Inside this jar file is the actual patch jar file, which will be of the form CR326566_92mp3.jar. Extract the latter file for the following steps.
Add the extracted jar file as the first element of the classpath of the Admin server as well as the managed servers in the domain.
If you are starting servers using the WebLogic Server startup script, update the classpath in the startup script like this:
set CLASSPATH=\jars\CR326566_92mp3.jar;%CLASSPATH% (Windows)
CLASSPATH=/jars/CR326566_92mp3.jar:$CLASSPATH (UNIX)
where PATCH_DIR is the directory on the machine where you extracted/saved the patch file.
Similarly, if you are starting servers using Node Manager, add the patch jar to the beginning of the Class Path argument in the Server Start tab for the server(s).
NOTE: Applying the patch to the classpath manually (approach 2) is recommended only for releases prior to WLS 9.1. Smart Update should be used when it is available as it provides patch conflict and dependency checking.
REFERENCES

上面的第一种方法是要连接网络通过GUI界面或者./bsu.sh的命令行接口来进行但是这也是一种使用不了的方法因为很少有人把weblogic服务器置于连网状态.
所以还是使用第二种方法先下补丁Patch 7669814,然后将补丁包中的.jar文件复制到weblogic服务器主机上然后将.jar包添加到classpath环境变量中.

补丁Patch 7669814包中的文件为KG6I.jar将其复制到weblogic的lib目录下

[root@sx-weblogic32 lib]# ls -lrt
total 28
-rw-r----- 1 root root   702 Mar 29  2011 readme.txt
-rw-r--r-- 1 root root 23302 Jun 10  2011 KG6I.jar

下面就需要修改weblogic的启动脚本将KG6I.jar文件加载到classpath环境变量中然后再重新启动weblogic
在startWebLogic.sh中增加下面的记录
CLASSPATH=/bea11/user_projects/domains/mydomain/lib/KG6I.jar:$CLASSPATH

在重新启动weblogic之后再来更新应用程序

####<2014-7-11 上午08时54分05秒 CST>     < [ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040045244>   
####<2014-7-11 上午08时54分05秒 CST>     < [ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040045438>   
####<2014-7-11 上午08时54分05秒 CST>     < [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040045474>   
####<2014-7-11 上午08时54分05秒 CST>     < [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040045476>   
####<2014-7-11 上午08时54分05秒 CST>     < [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040045730>  < [CompressingFilter/1.4.6] CompressingFilter has initialized> 
####<2014-7-11 上午08时54分14秒 CST>     < [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040054094>   
####<2014-7-11 上午08时54分14秒 CST>     < [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040054094>   
####<2014-7-11 上午08时54分14秒 CST>     < [ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040054111>   

TNS-01190故障的处理

由于监听程序原来是使用的是端口1532.现在修改成1521,结果不能启动说监听已经启动了.于是停止监听报错
TNS-01190: The user is not authorized to execute the requested listener command

[oracle@jyrac1 admin]$ lsnrctl start

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 03-JUL-2014 11:23:07

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

TNS-01106: Listener using listener name LISTENER has already been started

[oracle@jyrac1 admin]$ lsnrctl stop

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 03-JUL-2014 11:23:24

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=jyrac1)(PORT=1521)))
TNS-01190: The user is not authorized to execute the requested listener command

查看监听状态:

[grid@jyrac1 ~]$ ps -ef | grep -i listener
grid      4180     1  0 11:28 ?        00:00:00 /grid/11.2.0/grid/bin/tnslsnr LISTENER -inherit
grid      4517  4138  0 11:56 pts/1    00:00:00 grep -i listener
[oracle@jyrac1 ~]$ lsnrctl status

LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 03-JUL-2014 11:24:25

Copyright (c) 1991, 2009, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=jyrac1)(PORT=1521)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date                03-JUL-2014 10:40:26
Uptime                    0 days 0 hr. 43 min. 59 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /grid/11.2.0/grid/network/admin/listener.ora
Listener Log File         /grid/11.2.0/grid/log/diag/tnslsnr/jyrac1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=jyrac1)(PORT=1521)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "jycs" has 1 instance(s).
  Instance "jycs", status READY, has 1 handler(s) for this service...
Service "jycs (ORACLE_HOME =/u01/app/oracle/11.2.0/db" has 1 instance(s).
  Instance "jycs", status UNKNOWN, has 1 handler(s) for this service...
Service "jycsXDB" has 1 instance(s).
  Instance "jycs", status READY, has 1 handler(s) for this service...
The command completed successfully

其中:Security ON: Local OS Authentication 此条提示信息表明监听处于Local OS Authentication认证模式
Oracle 10g版本以及之后的版本中推出了监听的本地操作系统认证安全特性.若监听程序是在当前用户下启动的,则当前用户具有
管理监听的所有权利,其他用户对监听的管理将受到限制

因为数据库是11.2.0.1而且使用了oracle restart特性且用户为grid.注册了listener服务且只对默认端口1521有效.之前是1532所以
oracle restart不会自动重启监听.由于将端口修改成了1521所以oracle restart自动重启了listener

[grid@jyrac1 ~]$ srvctl status listener
Listener LISTENER is enabled
Listener LISTENER is running on node(s): jyrac1

由于oracle restart 以grid用户自动启动了监听所以oracle用户不能重动由grid用户所启动的监听.

配置静态听sid_name大小写造成无法登录

配置静态监听时SID_NAME名字大小写造成登录失败.对于oracle数据库来说同样的名字不一样的大小写表示完全不同的数据库实例。一旦静态监听的实例名字与对应的数据库实例不一致时,便会出现无法连接数据库的问题。

由于原来的1521端口要给另一个实例使用,现在的这个实例要使用另外的端口客户的就使用静态监听在设置完重启监听后登录出错.

SQL> conn test/test@127
ERROR:
ORA-01034: ORACLE not available
ORA-27101: shared memory realm does not exist
Linux-x86_64 Error: 2: No such file or directory

查看oracle home目录和oracle_sid

[oracle@ggfwweb admin]$ echo $ORACLE_HOME
/u01/app/oracle/10gR2/db
[oracle@ggfwweb admin]$ echo $ORACLE_SID
hygeia
[oracle@ggfwweb admin]$ ps -ef | grep pmon
oracle   25830     1  0 09:39 ?        00:00:00 ora_pmon_hygeia
oracle   25950 25585  0 09:59 pts/1    00:00:00 grep pmon

查看监听文件文件

[oracle@ggfwweb admin]$ cd $ORACLE_HOME/network/admin
[oracle@ggfwweb admin]$ ls
listener.ora  listener.ora.bak  samples  shrept.lst  tnsnames.ora
[oracle@ggfwweb admin]$ cat listener.ora
# listener.ora Network Configuration File: /u01/app/oracle/10gR2/db/network/admin/listener.ora
# Generated by Oracle configuration tools.

SID_LIST_LISTENER =
  (SID_LIST =
    (SID_DESC =
      (SID_NAME = PLSExtProc)
      (ORACLE_HOME = /u01/app/oracle/10gR2/db)
      (PROGRAM = extproc)
  )
    (SID_DESC = 
      (GLOBAL_DBNAME = HYGEIA)
      (ORACLE_HOME = /u01/app/oracle/10gR2/db)
      (SID_NAME = HYGEIA)
    )
  )

LISTENER =
  (DESCRIPTION_LIST =
    (DESCRIPTION =
      (ADDRESS = (PROTOCOL = TCP)(HOST =10.142.11.108)(PORT = 1568))
      (ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC0))
    )
  )

可以看到监听文件中使用的是HYGEIA,而ORACLE_SID是hygeia

将SID_NAME=HYGEIA修改为SID_NAME=hygeia后重启监听

[oracle@ggfwweb admin]$ lsnrctl stop

LSNRCTL for Linux: Version 10.2.0.5.0 - Production on 03-JUL-2014 10:03:32

Copyright (c) 1991, 2010, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.142.11.108)(PORT=1568)))
The command completed successfully

[oracle@ggfwweb admin]$ lsnrctl start

LSNRCTL for Linux: Version 10.2.0.5.0 - Production on 03-JUL-2014 10:03:53

Copyright (c) 1991, 2010, Oracle.  All rights reserved.

Starting /u01/app/oracle/10gR2/db/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 10.2.0.5.0 - Production
System parameter file is /u01/app/oracle/10gR2/db/network/admin/listener.ora
Log messages written to /u01/app/oracle/10gR2/db/network/log/listener.log
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.142.11.108)(PORT=1568)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.142.11.108)(PORT=1568)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 10.2.0.5.0 - Production
Start Date                03-JUL-2014 10:03:53
Uptime                    0 days 0 hr. 0 min. 0 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/oracle/10gR2/db/network/admin/listener.ora
Listener Log File         /u01/app/oracle/10gR2/db/network/log/listener.log
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.142.11.108)(PORT=1568)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC0)))
Services Summary...
Service "PLSExtProc" has 1 instance(s).
  Instance "PLSExtProc", status UNKNOWN, has 1 handler(s) for this service...
Service "hygeia" has 1 instance(s).
  Instance "hygeia", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
[oracle@ggfwweb admin]$ sqlplus /nolog

SQL*Plus: Release 10.2.0.5.0 - Production on Thu Jul 3 10:04:12 2014

Copyright (c) 1982, 2010, Oracle.  All Rights Reserved.

SQL> conn / as sysdba
Connected.
SQL> show parameter local_listener

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
local_listener                       string
SQL> conn test/test@127
Connected.

在配置静态监听时要注意数据库实例名本身是区分大小写的,因此在配置静态监听配置SID_NAME时一定要注意大小写