WebLogic Server 11g and 12c Configure SSL

最近由于进行安全三级等保,对于web应用要启用https协议,由于修改应用代码需要一些时间,所以选择使用weblogic的ssl来暂时顶替,对Weblogic 11,12c配置SSL需要执行以下操作:
1.创建密钥存储库和证书
2.对Weblogic服务器配置SLL
3.测试通过SSL来访问weblogic

1.创建密钥存储库和证书

[root@ggfwapp1 base_domain]# keytool -genkey -alias server_cert -keyalg RSA -keysize 2048 -sigalg SHA256withRSA -dname "CN=`hostname`,OU=Support,O=Oracle,L=Reading,ST=Berkshire,C=CN" -keypass abcdef -keystore keystore.jks -storepass abcdef -validity 3600

[root@ggfwapp1 base_domain]# keytool -selfcert -v -alias server_cert -keypass abcdef -keystore keystore.jks -storepass abcdef -storetype jks -validity 3600
New certificate (self-signed):
[
[
  Version: V3
  Subject: CN=ggfwapp1, OU=Support, O=Oracle, L=Reading, ST=Berkshire, C=CN
  Signature Algorithm: SHA1withRSA, OID = 1.2.840.113549.1.1.5

  Key:  Sun RSA public key, 2048 bits
  modulus: 19602090623719098800636488696565132568971729399292278916627114486591858227333395742468863359051755318713390759458530012565088859559851142124513453375262901573573600161911606667875583835689988346028966122749514073743319097824833258333036879524621784635720260866218245014085664041715662535739270842819315383017303960463560367669088786552196892207222513740073834657274289777601277828585874837302630694982849059492221540794576755294558686273906935420995110081764654613750381617199683694501833519148327516410714923674297363912818905309716955901896590958012374365281839683438007996858276943741295216721069229600336393261499
  public exponent: 65537
  Validity: [From: Tue Dec 20 01:10:57 CST 2016,
               To: Thu Oct 29 01:10:57 CST 2026]
  Issuer: CN=ggfwapp1, OU=Support, O=Oracle, L=Reading, ST=Berkshire, C=CN
  SerialNumber: [    585814a1]

]
  Algorithm: [SHA1withRSA]
  Signature:
0000: 09 FD 21 12 54 58 E6 45   56 C2 B4 FE 79 31 0C EC  ..!.TX.EV...y1..
0010: 35 AF E5 14 B8 94 26 39   DB 8F C6 B7 6E E5 03 7B  5.....&9....n...
0020: CF 48 5F 67 9D E6 3E C8   EA 0E 2E A6 B0 DD F6 1C  .H_g..>.........
0030: 5B E2 13 93 3D CF F4 80   B1 37 0B EE 0A 4E 2C 89  [...=....7...N,.
0040: CB 24 31 88 0F 0A 47 C9   1F 26 59 33 6C A4 6D 8C  .$1...G..&Y3l.m.
0050: A8 4A 7A 93 F7 25 21 9F   0E 61 50 34 8C 63 CF 16  .Jz..%!..aP4.c..
0060: 00 D5 E5 8F AC 2C F0 66   5C 60 1F 37 52 24 85 45  .....,.f\`.7R$.E
0070: BD CF A3 25 02 DF 92 F9   7D 15 D1 48 D7 DD AC D0  ...%.......H....
0080: 5A E1 01 AA 5A BD 99 4B   D6 1A 06 CC BB 31 DC 11  Z...Z..K.....1..
0090: 44 39 38 EC B9 56 8B 59   A6 99 34 54 60 D1 F6 79  D98..V.Y..4T`..y
00A0: C1 B5 0A 56 6A 3A 77 8F   A0 6F 5C B8 D1 D9 F7 AF  ...Vj:w..o\.....
00B0: DE AB 8B 59 FE 76 8E 61   B5 83 F4 F1 F6 04 AC C8  ...Y.v.a........
00C0: 5A B3 FE E5 6E 4E F8 21   FC 3F 0C 95 06 50 24 5B  Z...nN.!.?...P$[
00D0: 12 5E 1C D2 11 D5 C0 71   14 FE A4 73 8E 4E 15 96  .^.....q...s.N..
00E0: D6 28 95 ED 4E 1E 30 6F   AF 26 B7 03 47 25 9E 6F  .(..N.0o.&..G%.o
00F0: EA 15 0A EB 40 F6 F6 D8   DB 32 DD 64 AD 0F F5 70  ....@....2.d...p

]
[Storing keystore.jks]
[root@ggfwapp1 base_domain]# keytool -export -v -alias server_cert -file "`hostname`-rootCA.der" -keystore keystore.jks -storepass abcdef
Certificate stored in file 
[root@ggfwapp1 base_domain]# keytool -import -v -trustcacerts -alias server_cert -file "`hostname`-rootCA.der"   -keystore trust.jks -storepass abcdef
Owner: CN=ggfwapp1, OU=Support, O=Oracle, L=Reading, ST=Berkshire, C=CN
Issuer: CN=ggfwapp1, OU=Support, O=Oracle, L=Reading, ST=Berkshire, C=CN
Serial number: 585814a1
Valid from: Tue Dec 20 01:10:57 CST 2016 until: Thu Oct 29 01:10:57 CST 2026
Certificate fingerprints:
         MD5:  97:3B:58:6C:D0:22:0F:C3:8C:6E:29:99:2D:44:DB:A3
         SHA1: 3B:BA:5B:56:4D:9B:8D:3B:E8:EF:3A:D7:79:D9:B8:9C:43:30:FF:03
         Signature algorithm name: SHA1withRSA
         Version: 3
Trust this certificate? [no]:  yes
Certificate was added to keystore
[Storing trust.jks]

CN代表服务器的IP地址,注意要记录-alias,-keypass与-storepass参数,因为之后的配置需要使用
Identity Keystore: “/wls12c/user_projects/domains/base_domain/keystore.jks”
Trust Keystore: “/wls12c/user_projects/domains/base_domain/trust.jks”
Alias: server_cert
Store Password: abcdef
Key Password: abcdef
Valid for: 3600 Days (Approx 10 Years)

2.对Weblogic服务器配置SLL
2.1登录weblogic控制台

2.2选择环境->服务器并点击要配置的服务器


2.3选择密钥库

2.4选择密钥库更改选择 定制标识和定制信任(Custom Identity and Custom Trust)

输入以下信息
定制标识密匙库: keystore.jks
定制标识密钥库类型: JKS
定制标识密钥库密码短语: abcdef
确认定制标识密钥库密码短语: abcdef
定制信任密钥库: trust.jks
定制信任密钥库类型: JKS
定制信任密钥库密码短语: abcdef
确认定制信任密钥库密码短语: abcdef

2.5选择SSL页面

私有密钥别名: server_cert
私有密钥密码短语: abcdef
确认私有密钥密码短语句: abcdef

2.6点击一般信息,勾选SSL监听端口

3.测试通过SSL来访问weblogic

Weblogic BEA-002616 java.io.IOException: Too many open files

某业务系统不能访问,weblogic控制台登录不了,查看系统资源发现CPU使用率100%

[root@app ~]# top
top - 11:38:40 up 17 days, 20:41,  1 user,  load average: 7.99, 7.82, 7.63
Tasks: 171 total,   1 running, 170 sleeping,   0 stopped,   0 zombie
Cpu(s):100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  16436780k total, 13225904k used,  3210876k free,   275464k buffers
Swap: 16779884k total,        0k used, 16779884k free,  2676416k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5419 root      25   0 10.5g 9.6g  43m S 791.8 61.0   2549:30 java
 7931 root      15   0 12740 1152  820 R  0.3  0.0   0:00.01 top
    1 root      15   0 10348  692  584 S  0.0  0.0   0:01.99 init
    2 root      RT  -5     0    0    0 S  0.0  0.0   0:00.08 migration/0
    3 root      34  19     0    0    0 S  0.0  0.0   0:00.13 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.15 migration/1
    5 root      34  19     0    0    0 S  0.0  0.0   0:00.30 ksoftirqd/1
    6 root      RT  -5     0    0    0 S  0.0  0.0   0:00.17 migration/2
    7 root      34  19     0    0    0 S  0.0  0.0   0:00.30 ksoftirqd/2
    8 root      RT  -5     0    0    0 S  0.0  0.0   0:00.15 migration/3
    9 root      34  19     0    0    0 S  0.0  0.0   0:00.28 ksoftirqd/3
   10 root      RT  -5     0    0    0 S  0.0  0.0   0:00.15 migration/4
   11 root      34  19     0    0    0 S  0.0  0.0   0:00.28 ksoftirqd/4
   12 root      RT  -5     0    0    0 S  0.0  0.0   0:00.14 migration/5
   13 root      34  19     0    0    0 S  0.0  0.0   0:00.27 ksoftirqd/5
   14 root      RT  -5     0    0    0 S  0.0  0.0   0:00.15 migration/6
   15 root      34  19     0    0    0 S  0.0  0.0   0:00.31 ksoftirqd/6
   16 root      RT  -5     0    0    0 S  0.0  0.0   0:00.70 migration/7
   17 root      34  19     0    0    0 S  0.0  0.0   0:00.21 ksoftirqd/7
   18 root      10  -5     0    0    0 S  0.0  0.0   7:31.65 events/0
   19 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 events/1
   20 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 events/2
   21 root      10  -5     0    0    0 S  0.0  0.0   0:00.02 events/3
   22 root      10  -5     0    0    0 S  0.0  0.0   0:00.01 events/4
   23 root      10  -5     0    0    0 S  0.0  0.0   0:00.02 events/5
   24 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 events/6
   25 root      10  -5     0    0    0 S  0.0  0.0   0:00.03 events/7
   26 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 khelper
  163 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kthread
  174 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0
  175 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/1
  176 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/2
  177 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/3
  178 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/4
  179 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/5
  180 root      10  -5     0    0    0 S  0.0  0.0   0:00.04 kblockd/6
  181 root      10  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/7

查看weblogic日志找到发下信息:

[root@app1 base_domain]# more weblogic.log
<Dec 15, 2016 9:09:19 AM CST> <Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 10.138.130.61:7001, failure count: 1, failing for 0 seconds, java.io.IOException: Too many open files>

问题原因是因为对服务器配置的允许打开文件的最大数量比weblogic所需要的数量小而造成的。
解决方法:
1.设置 ulimit -u 65535
2.修改/etc/security/limits.conf在文件末加上
soft nofile 65535
hard nofile 65535

参考:System Crashes With “java.net.SocketException: Too many open files” Error (Doc ID 1194575.1)

Weblogic BEA-141281 unable to get file lock, will retry 故障处理

今天兄弟单位的一台应用服务器需要从测试环境移交到机房并修改IP,在移交前关闭了应用服务器,在移交后启动weblogic时出现了问题,weblogic进程起来了,但是控制台进不了不能修改数据源设置(也就是修改jdbc连接串)。操作下如下:

手动执行启动脚本(但这里其实另一个同事已经执行过一次)

[root@cdydtest bin]# ./startWebLogic.sh
.
.
JAVA Memory arguments: -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=256m
.
WLS Start Mode=Development
.
CLASSPATH=/usr/bea/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/bea/jdk1.6.0_20/lib/tools.jar:/usr/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/usr/bea/wlserver_10.3/server/lib/weblogic.jar:/usr/bea/modules/features/weblogic.server.modules_10.3.6.0.jar:/usr/bea/wlserver_10.3/server/lib/webservices.jar:/usr/bea/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/usr/bea/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/usr/bea/wlserver_10.3/common/derby/lib/derbyclient.jar:/usr/bea/wlserver_10.3/server/lib/xqrl.jar
.
PATH=/usr/bea/wlserver_10.3/server/bin:/usr/bea/modules/org.apache.ant_1.7.1/bin:/usr/bea/jdk1.6.0_20/jre/bin:/usr/bea/jdk1.6.0_20/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
Starting WLS with line:
/usr/bea/jdk1.6.0_20/bin/java -client   -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy  -Xverify:none  -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server   -Dweblogic.management.discover=true  -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath  weblogic.Server

因为输出日志被重定向到了日志文件(47ggzj.log)中,所以这里没有显示完整的日志信息。

抓取java进程来判断是否weblogic已经启动

[root@cdydtest bin]# ps -ef | grep java
root     11933     1  0 14:23 ?        00:00:17 /usr/bea/jdk1.6.0_20/bin/java -client -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server

root     14675     1 20 15:10 pts/3    00:00:01 /usr/bea/jdk1.6.0_20/bin/java -client -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server
root     14695 14567  0 15:10 pts/3    00:00:00 grep java

从上面的信息看出现了两个weblogic进程(pid为11933,14675)在运行,这是因为两个同事都手动执行了一次启动脚本。但登录不了weblogic控制台,如是查看weblogic日志文件。

[root@cdydtest base_domain]# cat 47ggzj.log
<nov 19, 2015 3:10:34 PM CST> <info> <security> <bea -090905> <disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<nov 19, 2015 3:10:34 PM CST> <info> <security> <bea -090906> <changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<nov 19, 2015 3:10:35 PM CST> <info> <weblogicserver> <bea -000377> <starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 16.3-b01 from Sun Microsystems Inc.>
<nov 19, 2015 3:10:45 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:10:55 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:05 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:15 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:25 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:35 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:45 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:11:55 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:05 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:15 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:25 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:35 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:45 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:12:55 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>
<nov 19, 2015 3:13:06 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry …>

从上面消息可以看到不能获得文件锁,虽然weblogic进程已经在运行,但是不能执行任何操作。这里出现这个问题的原因是因为需要修改jdbc连接,因为数据库服务器的IP地址修改了,而原来在weblogic中配置的jdbc并没有修改,那么在启动weblogic时,就会一直尝试连接,在这时weblogic服务是没有成功启动的,也登录不了控制台,但尝试连接达到weblogic缺的次数后,就会放弃尝试连接而执行后续的启动操作,但这需要等待一定的时间,而这时业务人员说不能登录系统,一位同事如是登录weblogic控制不能登录,如是再次执行了一次启动脚本,所以出现了不能获得文件锁的问题。现在的处理方法是kill掉这两个weblogic进程,并删除被锁定的AdminServer.lok文件,再次执行weblogic启动脚本就能正常启动。

删除被锁定的AdminServer.lok文件

[root@cdydtest /]#cd /usr/bea/user_projects/domains/base_domain/servers/AdminServer/tmp

[root@cdydtest tmp]# ls
AdminServer.lok  WebServiceUtils.ser  _WL_internal  _WL_user
[root@cdydtest tmp]# rm AdminServer.lok
rm: remove regular empty file `AdminServer.lok'? y

手动执行启动脚本

[root@cdydtest bin]# ./startWebLogic.sh
.
.
JAVA Memory arguments: -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=256m
.
WLS Start Mode=Development
.
CLASSPATH=/usr/bea/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/bea/jdk1.6.0_20/lib/tools.jar:/usr/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/usr/bea/wlserver_10.3/server/lib/weblogic.jar:/usr/bea/modules/features/weblogic.server.modules_10.3.6.0.jar:/usr/bea/wlserver_10.3/server/lib/webservices.jar:/usr/bea/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/usr/bea/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/usr/bea/wlserver_10.3/common/derby/lib/derbyclient.jar:/usr/bea/wlserver_10.3/server/lib/xqrl.jar
.
PATH=/usr/bea/wlserver_10.3/server/bin:/usr/bea/modules/org.apache.ant_1.7.1/bin:/usr/bea/jdk1.6.0_20/jre/bin:/usr/bea/jdk1.6.0_20/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
Starting WLS with line:
/usr/bea/jdk1.6.0_20/bin/java -client   -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy  -Xverify:none  -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server   -Dweblogic.management.discover=true  -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath  weblogic.Server

查看日志信息

[root@cdydtest base_domain]# cat 47ggzj.log
2015-11-19 15:23:46 Initializing Insur_CHANGDE from init-parameters:
PowerSI Version v0.2.9(Build20140901)
starttime:2015-11-19 15:23:46
trace_busiconn:1
trace_dbconn:0
hostname:cdydtest
jdbclogger:/usr/bea/user_projects/domains/base_domain/applications/Insur_CHANGDE/WEB-INF/jdbclogger.properties
homedir:/usr/bea/user_projects/domains/base_domain/applications/Insur_CHANGDE
companyname:????
jdbclogger.maxBatchCount:10
jdbclogger.minRuntime:1000
jdbclogger.needCaller:true
service_centerid:
applicationname:Insur_CHANGDE
scheduler_flag:0
logger:/usr/bea/user_projects/domains/base_domain/applications/Insur_CHANGDE/WEB-INF/log4j.properties
log_level:0
instancename:cdydtest.Insur_CHANGDE

<nov 19, 2015 3:23:46 PM CST> <notice> <loggingservice> <bea -320400> <the log file /usr/bea/user_projects/domains/base_domain/servers/AdminServer/logs/base_domain.log will be rotated. Reopen the log file if tailing has stopped. This can happen on some platforms like Windows.>
<nov 19, 2015 3:23:46 PM CST> <notice> <loggingservice> <bea -320401> <the log file has been rotated to /usr/bea/user_projects/domains/base_domain/servers/AdminServer/logs/base_domain.log00027. Log messages will continue to be logged in /usr/bea/user_projects/domains/base_domain/servers/AdminServer/logs/base_domain.log.>
<nov 19, 2015 3:23:46 PM CST> <notice> <log Management> <bea -170027> <the Server has established connection with the Domain level Diagnostic Service successfully.>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000365> <server state changed to ADMIN>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000365> <server state changed to RESUMING>
<nov 19, 2015 3:23:46 PM CST> <notice> <server> <bea -002613> <channel "Default" is now listening on 10.138.130.251:7001 for protocols iiop, t3, ldap, snmp, http.>
<nov 19, 2015 3:23:46 PM CST> <notice> <server> <bea -002613> <channel "Default[1]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000331> <started WebLogic Admin Server "AdminServer" for domain "base_domain" running in Development Mode>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000365> <server state changed to RUNNING>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000360> <server started in RUNNING mode>

从上面信息可知weblogic成功启动。从这个故障的原因来看就是操作人员在处理问题时不够细心,在不能登录weblogic控制台时并没有检查当前已经启动weblogic服务,而就执行了启动脚本才产生的问题。处理问题时一定要搞清状况,了解必要的信息,弄清原因才能操作。

weblogic启动不能锁定AdminServer.lok的故障处理

weblogic的版本是10.3,操作系统是linux,安装weblogic创建了一个weblogic用户,由于使用root用户修改了一些性能参数后,使用了root用户启动了一次weblogic,而原来启动weblogic的脚本是使用了weblogic用户,在使用weblogic用户启动weblogic时出现了不能锁定AdminServer.lok文件的错误,具体错误信息如下所示:

[weblogic@localhost ~]$  cd /home/weblogic/Oracle/Middleware/user_projects/domains/sbkwsjb/bin
[weblogic@localhost bin]$ ./startWebLogic.sh
.
.
JAVA Memory arguments: -Xms1024m -Xmx2048m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=512m
.
WLS Start Mode=Development
.
CLASSPATH=/home/weblogic/Oracle/Middleware/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/weblogic/Oracle/Middleware/patch_ocp371/profiles/default/sys_manifes
t_classpath/weblogic_patch.jar:/usr/java64/jdk1.6.0_45/lib/tools.jar:/home/weblogic/Oracle/Middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/home/weblogic/Oracle/Middleware/wlserver_10.3/
server/lib/weblogic.jar:/home/weblogic/Oracle/Middleware/modules/features/weblogic.server.modules_10.3.6.0.jar:/home/weblogic/Oracle/Middleware/wlserver_10.3/server/lib/webservices.jar:/home
/weblogic/Oracle/Middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/weblogic/Oracle/Middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/home/weblogic/Oracle/Mi
ddleware/wlserver_10.3/common/derby/lib/derbyclient.jar:/home/weblogic/Oracle/Middleware/wlserver_10.3/server/lib/xqrl.jar:.:/usr/java64/jdk1.6.0_45/lib:/usr/java64/jdk1.6.0_45/lib/dt.jar:/u
sr/java64/jdk1.6.0_45/lib/tools.jar
.
PATH=/home/weblogic/Oracle/Middleware/wlserver_10.3/server/bin:/home/weblogic/Oracle/Middleware/modules/org.apache.ant_1.7.1/bin:/usr/java64/jdk1.6.0_45/jre/bin:/usr/java64/jdk1.6.0_45/bin:/
usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/java64/jdk1.6.0_45/bin:/home/weblogic/bin
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
Starting WLS with line:
/usr/java64/jdk1.6.0_45/bin/java -client   -Xms1024m -Xmx2048m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=512m -Dweblogic.Name=AdminServer -Djava.security.policy=/home/webl
ogic/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy  -Xverify:none  -da -Dplatform.home=/home/weblogic/Oracle/Middleware/wlserver_10.3 -Dwls.home=/home/weblogic/Oracle/Middleware
/wlserver_10.3/server -Dweblogic.home=/home/weblogic/Oracle/Middleware/wlserver_10.3/server   -Dweblogic.management.discover=true  -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToCon
sole= -Dweblogic.ext.dirs=/home/weblogic/Oracle/Middleware/patch_wls1036/profiles/default/sysext_manifest_classpath:/home/weblogic/Oracle/Middleware/patch_ocp371/profiles/default/sysext_mani
fest_classpath  weblogic.Server
bea -000362 server failed. Reason:

There are 1 nested errors:

weblogic.management.ManagementException: Unable to obtain lock on /home/weblogic/Oracle/Middleware/user_projects/domains/sbkwsjb/servers/AdminServer/tmp/AdminServer.lok. Server may already b
e running
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:206)
        at weblogic.management.internal.ServerLocks.getServerLock(ServerLocks.java:67)
        at weblogic.management.internal.DomainDirectoryService.start(DomainDirectoryService.java:74)
        at weblogic.t3.srvr.ServerServicesManager.startService(ServerServicesManager.java:461)
        at weblogic.t3.srvr.ServerServicesManager.startInStandbyState(ServerServicesManager.java:166)
        at weblogic.t3.srvr.T3Srvr.initializeStandby(T3Srvr.java:881)
        at weblogic.t3.srvr.T3Srvr.startup(T3Srvr.java:568)
        at weblogic.t3.srvr.T3Srvr.run(T3Srvr.java:469)
        at weblogic.Server.main(Server.java:71)


出现这种情况的原因有两种可能但不限于,一是执行kill -9杀死weblogic进程,一种是使用的不是安装weblogic的用户比如root用户来启动weblogic。这里出现这种情况之前,既使用了kill -9杀死weblogic进程,也使用root用户(而不是weblogic用户)来启动weblogic。

对于这种情况解决方法有以下三种
1.是修改AdminServer.lok文件的权限,让weblogic用户能锁定它.
2.删除AdminServer.lok文件并重新使用weblogic用户来启动weblogic服务.
3.杀死仍然运行的服务进程并重新weblogic服务.

这里我采用的是第一种解决方法:

[root@localhost bin]# cd /home/weblogic/Oracle/Middleware/user_projects/domains/sbkwsjb/servers/AdminServer/tmp/
[root@localhost tmp]# ls -lrt
total 12
drwxr----- 5 weblogic bea  4096 Dec 29 19:09 _WL_internal
drwxr----- 3 weblogic bea  4096 Mar 17 17:53 _WL_user
-rw-r----- 1 weblogic bea   264 Apr 21 17:11 WebServiceUtils.ser
-rw-r----- 1 root     root    0 Apr 21 17:24 AdminServer.lok
[root@localhost tmp]# chown weblogic:bea AdminServer.lok
[root@localhost tmp]# ls -lrt
total 12
drwxr----- 5 weblogic bea 4096 Dec 29 19:09 _WL_internal
drwxr----- 3 weblogic bea 4096 Mar 17 17:53 _WL_user
-rw-r----- 1 weblogic bea  264 Apr 21 17:11 WebServiceUtils.ser
-rw-r----- 1 weblogic bea    0 Apr 21 17:24 AdminServer.lok

第二种方法,删除AdminServer.lok文件并重新使用weblogic用户来启动weblogic服务的操作示例如下:
1. 在user_projects/domains/目录中移动或删除它被锁定的文件,例如:

$ find . -name "*DAT"
./servers/AdminServer/data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT
./servers/AdminServer/data/store/default/_WLS_ADMINSERVER000000.DAT

这里要注意的是Unix或Linux是大小敏感的,在执行查找时不要将”*DAT”写成”.dat”,如果写成”.dat”可能删除将?/security/SerializedSystemIni.dat文件删除。

2.删除EmbeddedLDAP.lok and AdminServer.lok 文件

[root@localhost tmp]rm  -rf /home/weblogic/Oracle/Middleware/user_projects/domains/sbkwsjb/servers/AdminServer/tmp/AdminServer.lok


[root@localhost tmp]rm  -rf
/home/weblogic/Oracle/Middleware/user_projects/domains/sbkwsjb/servers/AdminServer/data/ldap/ldapfiles/EmbeddedLDAP.lok

3.可以执行下面的脚本来找出进程正在监听的端口并通过kill -TERM将其杀掉。

#!/bin/ksh

line='---------------------------------------------'
pids=$(/usr/bin/ps -ef | sed 1d | awk '{print $2}')

if [ $# -eq 0 ]; then
   read ans?"Enter port you would like to know pid for: "
else
   ans=$1
fi

for f in $pids
do
   /usr/proc/bin/pfiles $f 2>/dev/null | /usr/xpg4/bin/grep -q "port: $ans"
   if [ $? -eq 0 ]; then
      echo $line
      echo "Port: $ans is being used by PID:\c"
      /usr/bin/ps -ef -o pid -o args | egrep -v "grep|pfiles" | grep $f
   fi
done
exit 0

4.重新启动weblogic服务

[weblogic@localhost bin]$nohup ./startWebLogic.sh &

第三种方法,杀死仍然运行的服务进程并重新weblogic服务的示例操作如下:
1. 找到Managed Server的:

ps -ef | grep ManagedServer

2.使用kill -9 杀掉Managed Server进程:

kill -9 

3. 重新启动weblogic服务:

nohup ./startWebLogic.sh &

要注意的是startWeblogic.sh脚本通常要使用nohup来启动,这样当用户退出时shell才能保持继续运行.

如果是Windows平台
1. AdminServer.lok 文件的缺存储目录是: C:\Oracle\Middleware\user_projects\domains\\servers\AdminServer\tmp\
2. 执行startWebLogic.cmd 命令来启动weblogic服务
注意,如果AdminServer.lok 不能删除可以在任务管理器中杀掉Weblogic服务的所有java.exe进程,再来删除AdminServer.lok file。

在解决不能锁定AdminServer.lok文件之后我们再来使用weblogic用户启动weblogic服务:

[weblogic@localhost bin]$ ./startWebLogic.sh
...省略...
an error occurred while initializing the Embedded LDAP Server. The exception thrown is java.lang.ClassCastException: com
.octetstring.vde.backend.BackendRoot cannot be cast to com.octetstring.vde.backend.standard.BackendStandard. This may indicate a problem with the data files for the Embedded LDAP Server. If
the problem is with the data files and it can not be corrected, backups of previous versions of the data files exist in /home/weblogic/Oracle/Middleware/user_projects/domains/sbkwsjb/servers
/AdminServer/data/ldap/backup
bea -000362 server failed. Reason:

There are 1 nested errors:

java.lang.ClassCastException: com.octetstring.vde.backend.BackendRoot cannot be cast to com.octetstring.vde.backend.standard.BackendStandard
        at weblogic.ldap.EmbeddedLDAP.start(EmbeddedLDAP.java:303)
        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
        at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

出现的错误是没有权限打开事务文件EmbeddedLDAP.tran,这还是权限原因:

/home/weblogic/Oracle/Middleware/user_projects/domains/sbkwsjb/servers/AdminServer/data/
ldap/ldapfiles/EmbeddedLDAP.tran (Permission denied)

修改EmbeddedLDAP.tran 文件的权限:

[root@localhost weblogic]# cd /home/weblogic/Oracle/Middleware/user_projects/domains/sbkwsjb/servers/AdminServer/data/ldap/ldapfiles/
[root@localhost ldapfiles]# ls -lrt
total 164
-rw-r----- 1 weblogic bea     864 Dec 30 09:01 EmbeddedLDAP.index
-rw-r----- 1 weblogic bea     204 Dec 30 09:01 EmbeddedLDAP.delete
-rw-r----- 1 weblogic bea  146088 Apr 21 16:16 EmbeddedLDAP.data
-rw-r----- 1 weblogic bea       8 Apr 21 17:24 EmbeddedLDAP.twpos
-rw-r----- 1 weblogic bea       8 Apr 21 17:24 EmbeddedLDAP.trpos
-rw-r----- 1 root     root      0 Apr 21 17:24 EmbeddedLDAP.tran
-rw-r----- 1 weblogic bea       0 Apr 21 19:26 EmbeddedLDAP.lok

从上面的信息可以看到EmbeddedLDAP.tran文件的权限是root用户所有,这是因为之前使用root用户启动过weblogic的原因,而现在是使用weblogic用户来启动weblogic

[root@localhost ldapfiles]# chown weblogic EmbeddedLDAP.tran
[root@localhost ldapfiles]# ls -lrt
total 164
-rw-r----- 1 weblogic bea     864 Dec 30 09:01 EmbeddedLDAP.index
-rw-r----- 1 weblogic bea     204 Dec 30 09:01 EmbeddedLDAP.delete
-rw-r----- 1 weblogic bea  146088 Apr 21 16:16 EmbeddedLDAP.data
-rw-r----- 1 weblogic bea       8 Apr 21 17:24 EmbeddedLDAP.twpos
-rw-r----- 1 weblogic bea       8 Apr 21 17:24 EmbeddedLDAP.trpos
-rw-r----- 1 weblogic root      0 Apr 21 17:24 EmbeddedLDAP.tran
-rw-r----- 1 weblogic bea       0 Apr 21 19:26 EmbeddedLDAP.lok
[root@localhost ldapfiles]# chown weblogic:bea EmbeddedLDAP.tran
[root@localhost ldapfiles]# ls -lrt
total 164
-rw-r----- 1 weblogic bea    864 Dec 30 09:01 EmbeddedLDAP.index
-rw-r----- 1 weblogic bea    204 Dec 30 09:01 EmbeddedLDAP.delete
-rw-r----- 1 weblogic bea 146088 Apr 21 16:16 EmbeddedLDAP.data
-rw-r----- 1 weblogic bea      8 Apr 21 17:24 EmbeddedLDAP.twpos
-rw-r----- 1 weblogic bea      8 Apr 21 17:24 EmbeddedLDAP.trpos
-rw-r----- 1 weblogic bea      0 Apr 21 17:24 EmbeddedLDAP.tran
-rw-r----- 1 weblogic bea      0 Apr 21 19:26 EmbeddedLDAP.lok

再次使用weblogic用户来启动weblogic

[weblogic@localhost bin]$ ./startWebLogic.sh
.
.
JAVA Memory arguments: -Xms1024m -Xmx2048m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=512m
.
WLS Start Mode=Development
.
CLASSPATH=/home/weblogic/Oracle/Middleware/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/home/weblogic/Oracle/Middleware/patch_ocp371/profiles/default/sys_manifes
t_classpath/weblogic_patch.jar:/usr/java64/jdk1.6.0_45/lib/tools.jar:/home/weblogic/Oracle/Middleware/wlserver_10.3/server/lib/weblogic_sp.jar:/home/weblogic/Oracle/Middleware/wlserver_10.3/
server/lib/weblogic.jar:/home/weblogic/Oracle/Middleware/modules/features/weblogic.server.modules_10.3.6.0.jar:/home/weblogic/Oracle/Middleware/wlserver_10.3/server/lib/webservices.jar:/home
/weblogic/Oracle/Middleware/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/home/weblogic/Oracle/Middleware/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/home/weblogic/Oracle/Mi
ddleware/wlserver_10.3/common/derby/lib/derbyclient.jar:/home/weblogic/Oracle/Middleware/wlserver_10.3/server/lib/xqrl.jar:.:/usr/java64/jdk1.6.0_45/lib:/usr/java64/jdk1.6.0_45/lib/dt.jar:/u
sr/java64/jdk1.6.0_45/lib/tools.jar
.
PATH=/home/weblogic/Oracle/Middleware/wlserver_10.3/server/bin:/home/weblogic/Oracle/Middleware/modules/org.apache.ant_1.7.1/bin:/usr/java64/jdk1.6.0_45/jre/bin:/usr/java64/jdk1.6.0_45/bin:/
usr/lib64/qt-3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/java64/jdk1.6.0_45/bin:/home/weblogic/bin
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
Starting WLS with line:
/usr/java64/jdk1.6.0_45/bin/java -client   -Xms1024m -Xmx2048m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=512m -Dweblogic.Name=AdminServer -Djava.security.policy=/home/webl
ogic/Oracle/Middleware/wlserver_10.3/server/lib/weblogic.policy  -Xverify:none  -da -Dplatform.home=/home/weblogic/Oracle/Middleware/wlserver_10.3 -Dwls.home=/home/weblogic/Oracle/Middleware
/wlserver_10.3/server -Dweblogic.home=/home/weblogic/Oracle/Middleware/wlserver_10.3/server   -Dweblogic.management.discover=true  -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToCon
sole= -Dweblogic.ext.dirs=/home/weblogic/Oracle/Middleware/patch_wls1036/profiles/default/sysext_manifest_classpath:/home/weblogic/Oracle/Middleware/patch_ocp371/profiles/default/sysext_mani
fest_classpath  weblogic.Server

至此问题已经解决。

总结:出现不能锁定AdminServer.lok文件的错误的原因是,因为使用kill -9 杀掉了weblogic服务进程,或者使用不是安装weblogic程序的用户来启动过weblogic所造成的。出现这种问题后,有三种解决方法:
1.是修改AdminServer.lok文件的权限,让weblogic用户能锁定它.
2.删除AdminServer.lok文件并重新使用weblogic用户来启动weblogic服务
3.杀死仍然运行的服务进程并重新weblogic服务.

weblogic 10.3.3 java.lang.IllegalArgumentException: Failed to properly unregister weblogic.work.RequestClassRuntimeMBeanImpl故障解决一例

weblogic 10.3.3异常终止服务了检查日志文件发现如下错误信息:

weblogic.management.DeploymentException: 
  at weblogic.application.internal.flow.ApplicationRuntimeMBeanFlow.unprepare(ApplicationRuntimeMBeanFlow.java:64)
  at weblogic.application.internal.BaseDeployment$1.previous(BaseDeployment.java:1233)
  at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:167)
  at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:159)
  at weblogic.application.internal.BaseDeployment.unprepare(BaseDeployment.java:495)
  at weblogic.application.internal.SingleModuleDeployment.unprepare(SingleModuleDeployment.java:43)
  at weblogic.application.internal.DeploymentStateChecker.unprepare(DeploymentStateChecker.java:205)
  at weblogic.deploy.internal.targetserver.AppContainerInvoker.unprepare(AppContainerInvoker.java:117)
  at weblogic.deploy.internal.targetserver.BasicDeployment.unprepare(BasicDeployment.java:287)
  at weblogic.management.deploy.internal.DeploymentAdapter$1.doUnprepare(DeploymentAdapter.java:81)
  at weblogic.management.deploy.internal.DeploymentAdapter.unprepare(DeploymentAdapter.java:224)
  at weblogic.management.deploy.internal.AppTransition$7.transitionApp(AppTransition.java:75)
  at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
  at weblogic.management.deploy.internal.ConfiguredDeployments.unprepare(ConfiguredDeployments.java:204)
  at weblogic.management.deploy.internal.ConfiguredDeployments.undeploy(ConfiguredDeployments.java:192)
  at weblogic.management.deploy.internal.DeploymentServerService.shutdownApps(DeploymentServerService.java:195)
  at weblogic.management.deploy.internal.DeploymentServerService.shutdownHelper(DeploymentServerService.java:127)
  at weblogic.application.ApplicationShutdownService.halt(ApplicationShutdownService.java:142)
  at weblogic.t3.srvr.ServerServicesManager.haltInternal(ServerServicesManager.java:504)
  at weblogic.t3.srvr.ServerServicesManager.halt(ServerServicesManager.java:336)
  at weblogic.t3.srvr.T3Srvr.shutdown(T3Srvr.java:1039)
  at weblogic.t3.srvr.T3Srvr.forceShutdown(T3Srvr.java:945)
  at weblogic.t3.srvr.T3Srvr$2.run(T3Srvr.java:958)

Caused By: java.lang.IllegalArgumentException: Failed to properly unregister weblogic.work.RequestClassRuntimeMBeanImpl@50b20090 for ObjectName com.bea:ServerRuntime=AdminServer,Name=default@plat_changde_test@null,WorkManagerRuntime=default,ApplicationRuntime=plat_changde_test,Type=RequestClassRuntime
  at weblogic.management.jmx.ObjectNameManagerBase.unregisterObject(ObjectNameManagerBase.java:219)
  at weblogic.management.jmx.ObjectNameManagerBase.unregisterObjectInstance(ObjectNameManagerBase.java:192)
  at weblogic.management.mbeanservers.internal.RuntimeMBeanAgent$1.unregisteredInternal(RuntimeMBeanAgent.java:124)
  at weblogic.management.mbeanservers.internal.RuntimeMBeanAgent$1.unregistered(RuntimeMBeanAgent.java:108)
  at weblogic.management.provider.core.RegistrationManagerBase.invokeRegistrationHandlers(RegistrationManagerBase.java:187)
  at weblogic.management.provider.core.RegistrationManagerBase.unregister(RegistrationManagerBase.java:126)
  at weblogic.management.runtime.RuntimeMBeanDelegate.unregister(RuntimeMBeanDelegate.java:287)
  at weblogic.management.runtime.RuntimeMBeanDelegate.unregisterChildren(RuntimeMBeanDelegate.java:350)
  at weblogic.management.runtime.RuntimeMBeanDelegate.unregister(RuntimeMBeanDelegate.java:274)
  at weblogic.management.runtime.RuntimeMBeanDelegate.unregisterChildren(RuntimeMBeanDelegate.java:350)
  at weblogic.management.runtime.RuntimeMBeanDelegate.unregister(RuntimeMBeanDelegate.java:274)
  at weblogic.j2ee.J2EEApplicationRuntimeMBeanImpl.unregister(J2EEApplicationRuntimeMBeanImpl.java:359)
  at weblogic.application.internal.flow.ApplicationRuntimeMBeanFlow.unprepare(ApplicationRuntimeMBeanFlow.java:62)
  at weblogic.application.internal.BaseDeployment$1.previous(BaseDeployment.java:1233)
  at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:167)
  at weblogic.application.utils.StateMachineDriver.previousState(StateMachineDriver.java:159)
  at weblogic.application.internal.BaseDeployment.unprepare(BaseDeployment.java:495)
  at weblogic.application.internal.SingleModuleDeployment.unprepare(SingleModuleDeployment.java:43)
  at weblogic.application.internal.DeploymentStateChecker.unprepare(DeploymentStateChecker.java:205)
  at weblogic.deploy.internal.targetserver.AppContainerInvoker.unprepare(AppContainerInvoker.java:117)
  at weblogic.deploy.internal.targetserver.BasicDeployment.unprepare(BasicDeployment.java:287)
  at weblogic.management.deploy.internal.DeploymentAdapter$1.doUnprepare(DeploymentAdapter.java:81)
  at weblogic.management.deploy.internal.DeploymentAdapter.unprepare(DeploymentAdapter.java:224)
  at weblogic.management.deploy.internal.AppTransition$7.transitionApp(AppTransition.java:75)
  at weblogic.management.deploy.internal.ConfiguredDeployments.transitionApps(ConfiguredDeployments.java:240)
  at weblogic.management.deploy.internal.ConfiguredDeployments.unprepare(ConfiguredDeployments.java:204)
  at weblogic.management.deploy.internal.ConfiguredDeployments.undeploy(ConfiguredDeployments.java:192)
  at weblogic.management.deploy.internal.DeploymentServerService.shutdownApps(DeploymentServerService.java:195)
  at weblogic.management.deploy.internal.DeploymentServerService.shutdownHelper(DeploymentServerService.java:127)
  at weblogic.application.ApplicationShutdownService.halt(ApplicationShutdownService.java:142)
  at weblogic.t3.srvr.ServerServicesManager.haltInternal(ServerServicesManager.java:504)
  at weblogic.t3.srvr.ServerServicesManager.halt(ServerServicesManager.java:336)
  at weblogic.t3.srvr.T3Srvr.shutdown(T3Srvr.java:1039)
  at weblogic.t3.srvr.T3Srvr.forceShutdown(T3Srvr.java:945)
  at weblogic.t3.srvr.T3Srvr$2.run(T3Srvr.java:958)

故障原因

Caused By: java.lang.IllegalArgumentException: Failed to properly unregister weblogic.work.RequestClassRuntimeMBeanImpl@50b20090 for ObjectName com.bea:ServerRuntime=AdminServer,Name=default@plat_changde_test@null,WorkManagerRuntime=default,ApplicationRuntime=plat_changde_test,Type=RequestClassRuntime

在MOS上找到一篇关于这个错误的文章说原因是

When you have JDBC connection pool name same as application name in config.xml we are running into the issue.

在config.xml文件中确实存在jb_zs,jb_test,plat_changde_test,plat_changde的应用名与连接池名相同将其修改为不一样后重新weblogic解决了此问题.

WLS 10.3.0 java.lang.IllegalArgumentException Registered more than one instance部署程序异常终止

单位上新上一应用在weblogic控制台进行应用程序更新发布出现
java.lang.IllegalArgumentException Registered more than one instance异常,然后weblogic就异常退出.

查看AdminServer.log日志可以看到如下信息:


####<2014-7-10 下午04时28分11秒 CST>     < [ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'>  <> <> <1404980891082>   
####<2014-7-10 下午04时28分21秒 CST>     < [ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'>  <> <> <1404980901145>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901148>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901148>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901149>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901173>  < [CompressingFilter/1.4.6] CompressingFilter is being destroyed...> 
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901176>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901177>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901177>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901178>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901178>   
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901190>  (RuntimeMBeanDelegate.java:255)
	at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:215)
	at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:193)
	at weblogic.work.WorkManagerRuntimeMBeanImpl.(WorkManagerRuntimeMBeanImpl.java:49)
	at weblogic.work.WorkManagerRuntimeMBeanImpl.getWorkManagerRuntime(WorkManagerRuntimeMBeanImpl.java:59)
	at weblogic.work.WorkManagerCollection.addWorkManagerRuntime(WorkManagerCollection.java:774)
	at weblogic.work.WorkManagerCollection.initialize(WorkManagerCollection.java:187)
	at weblogic.application.internal.flow.WorkManagerFlow.prepare(WorkManagerFlow.java:45)
	at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:1221)
	at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
	at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:367)
	at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:43)
	at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:154)
	at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:60)
	at weblogic.deploy.internal.targetserver.operations.RedeployOperation.createAndPrepareContainer(RedeployOperation.java:98)
	at weblogic.deploy.internal.targetserver.operations.RedeployOperation.doPrepare(RedeployOperation.java:122)
	at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:217)
	at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:747)
	at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1216)
	at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:250)
	at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:159)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
	at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)
> 
####<2014-7-10 下午04时28分21秒 CST>     < [STANDBY] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1404980901192>  (RuntimeMBeanDelegate.java:255)
	at weblogic.management.runtime.RuntimeMBeanDelegate.(RuntimeMBeanDelegate.java:215)
	at weblogic.work.RequestClassRuntimeMBeanImpl.(RequestClassRuntimeMBeanImpl.java:32)
	at weblogic.work.WorkManagerRuntimeMBeanImpl.getRequestClassRuntime(WorkManagerRuntimeMBeanImpl.java:86)
	at weblogic.work.WorkManagerRuntimeMBeanImpl.getWorkManagerRuntime(WorkManagerRuntimeMBeanImpl.java:61)
	at weblogic.work.WorkManagerCollection.addWorkManagerRuntime(WorkManagerCollection.java:774)
	at weblogic.work.WorkManagerCollection.initialize(WorkManagerCollection.java:187)
	at weblogic.application.internal.flow.WorkManagerFlow.prepare(WorkManagerFlow.java:45)
	at weblogic.application.internal.BaseDeployment$1.next(BaseDeployment.java:1221)
	at weblogic.application.utils.StateMachineDriver.nextState(StateMachineDriver.java:41)
	at weblogic.application.internal.BaseDeployment.prepare(BaseDeployment.java:367)
	at weblogic.application.internal.SingleModuleDeployment.prepare(SingleModuleDeployment.java:43)
	at weblogic.application.internal.DeploymentStateChecker.prepare(DeploymentStateChecker.java:154)
	at weblogic.deploy.internal.targetserver.AppContainerInvoker.prepare(AppContainerInvoker.java:60)
	at weblogic.deploy.internal.targetserver.operations.RedeployOperation.createAndPrepareContainer(RedeployOperation.java:98)
	at weblogic.deploy.internal.targetserver.operations.RedeployOperation.doPrepare(RedeployOperation.java:122)
	at weblogic.deploy.internal.targetserver.operations.AbstractOperation.prepare(AbstractOperation.java:217)
	at weblogic.deploy.internal.targetserver.DeploymentManager.handleDeploymentPrepare(DeploymentManager.java:747)
	at weblogic.deploy.internal.targetserver.DeploymentManager.prepareDeploymentList(DeploymentManager.java:1216)
	at weblogic.deploy.internal.targetserver.DeploymentManager.handlePrepare(DeploymentManager.java:250)
	at weblogic.deploy.internal.targetserver.DeploymentServiceDispatcher.prepare(DeploymentServiceDispatcher.java:159)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.doPrepareCallback(DeploymentReceiverCallbackDeliverer.java:171)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer.access$000(DeploymentReceiverCallbackDeliverer.java:13)
	at weblogic.deploy.service.internal.targetserver.DeploymentReceiverCallbackDeliverer$1.run(DeploymentReceiverCallbackDeliverer.java:46)
	at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:528)
	at weblogic.work.ExecuteThread.execute(ExecuteThread.java:201)
	at weblogic.work.ExecuteThread.run(ExecuteThread.java:173)


从上面的错误信息可以看到

 redeploy operation for application, hninsiis

这说明是在更新应用程序重新加载hninsiis应用

java.lang.IllegalArgumentException: Registered more than one instance with the same objectName : com.bea:ServerRuntime=AdminServer,Name=default@hninsiis@null,WorkManagerRuntime=default,ApplicationRuntime=hninsiis,Type=RequestClassRuntime new:weblogic.work.RequestClassRuntimeMBeanImpl@1470f4af existing weblogic.work.RequestClassRuntimeMBeanImpl@3702476f

这个信息是说在加载应用程序时已经存在一个同名的应用程序,而实际上并不存在同名的应用程序

这时只能使用无所不能的MOS了,在MOS上找到了关于这个错误信息的文档


WLS 10.3.0: java.lang.IllegalArgumentException: Existing Timer Manager Has Different Work Manager (文档 ID 1097661.1)

In this Document
Symptoms
Cause
Solution
References

APPLIES TO:

Oracle Weblogic Server - Version 10.3 to 10.3
Information in this document applies to any platform.
***Checked for relevance on 2-May-2013***
SYMPTOMS

When trying to redeploy an application that uses a foreign JMS queue, the error below is shown. The server needs to be rebooted with every new deployment. Otherwise, the server fails consistently with the error: "Existing timer manager has different work manager."

###     < [ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1272310540224>  
CAUSE

This issue is caused by a flaw in the WLS code such that all MDB instances/pollers on the same WLS server of the same MDB deployment share the same timer manager for message polling. When one instance is disconnected due to JMS server migration, the timer manager is stopped, which in turn stops other MDB instances on the same server. This issue is addressed by unpublished defect 7669814.

SOLUTION

Patches are available for unpublished defect 7669814:

PATCH INFORMATION
WLS Version	Patch Number
10.3.0	Patch 7669814
Fixed in: 10.3.1

To apply one of these patches, click on the link for your WLS version and download the appropriate patch from My Oracle Support. You can also search in My Oracle Support for the patch number for your WLS version: for detailed instructions, please see Master Note: How to Locate and Download Patches for WebLogic Server Using My Oracle Support Document 1302053.1. For instructions on how to apply these patches to your system, please see How to Apply WebLogic Server (WLS) Patches Using Smart Update Document 876004.1. For other issues relating to Smart Update, please see Master Note on Troubleshooting Smart Update Issues Document 1230725.1.

Patches are specifically tied to a particular release and maintenance pack of WebLogic Server (WLS). A patch for WLS 10.3.3, for example, very likely would not work on WLS 10.3.5. In a few cases, patches are also specific to particular operating systems. If you think you are experiencing the problem outlined in this note, and you are running a WLS version which is eligible for error correction (see Document 950131.1 for more about the Oracle Error Correction Policy), but your WLS version is not included in the list of patches provided here, please contact support and ask for a version of the patch appropriate for you. Please reference this note as well as this bug number to help speed our service for you.

NOTE: Patches are applied per WLS installation and not per domain. That is, if you apply this patch on one WLS installation, then all of the servers from all the domains in that installation will have this patch. On the other hand, if you have a managed server in another machine in a domain (that is, set up with its own WLS installation), you need to install this patch on that other machine as well. Generally, patches can only be applied while the server is not running because WLS locks the needed files while it is running. If, however, you are able to apply a patch while WLS is running, you must restart WLS before the patch will take effect.


从上面的信息说已经在10.3.1这个版本已经修复这个bug了.但我们的weblogic版本是10.3.3.0

[root@sx-weblogic31 lib]# java -cp weblogic.jar weblogic.version

WebLogic Server 10.3.3.0  Fri Apr 9 00:05:28 PDT 2010 1321401 

Use 'weblogic.version -verbose' to get subsystem information

Use 'weblogic.utils.Versions' to get version information for all modules


其实给weblogic打补丁提供了两种方法

1. Using Smart Update

You can apply the patch using Smart Update with the following steps:

Download the patch from My Oracle Support (MOS). For more details, please refer to Master Note: How to Locate and Download Patches for WebLogic Server Using My Oracle Support Note 1302053.1.
Extract the contents from the zip file: you will have a jar file and patch-catalog_xxx.xml. A readme file may also be included.
Copy the files (for example, E5W8.jar and WGQJ.jar) and the patch-catalog_xxx.xml from the zip file to the target machine. You do not need the readme file. Copy the files to the appropriate cache_dir directory for the target system: for example, on Windows, %MIDDLEWARE_HOME%\utils\bsu\cache_dir, or on UNIX, $MIDDLEWARE_HOME/utils/bsu/cache_dir. The directory MW_HOME\utils\bsu\cache_diris created as the default patch download directory when you install Smart Update 3.3.0. (see http://docs.oracle.com/cd/E14759_01/doc.32/e14143/start.htm#i1071723)

NOTE: Always copy the patch-catalog_xxx.xml file from the downloaded patch to the cache_dir along with the patch itself. Do NOTrename this file. If you see an "unrecognized patch ID" error, please refer to Note 1186923.1 for details on how to resolve it.
Run Smart Update and apply the patches and/or patch sets to the target system. This can be done using the Smart Update GUI or the command-line interface (see http://download.oracle.com/docs/cd/E14759_01/doc.32/e14143/commands.htm#i1074489).
a. Smart Update in graphical (GUI) mode

Run the /utils/bsu/bsu script (bsu.sh for UNIX, bsu.cmd for Windows). This will start the Smart Update GUI.
Look for the patches you copied in the "Downloaded Patches" section at the bottom.
Select the "Apply" button for each patch you want to apply. This will validate the patch and apply it to the whole installation.
The following viewlet provides an example of using Smart Update in GUI mode:

 Video - Applying a Patch Using Smart Update in GUI Mode (1:15) Trouble seeing this video?

b. Command-line interface

This is the syntax for the command to view the downloaded patches as below:
./bsu.sh -prod_dir= -patch_download_dir= -status=downloaded -view -verbose
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patch_download_dir=/opt/bea/utils/bsu/cache_dir -status=downloaded -view -verbose
This is the syntax for the command to install a patch:
./bsu.sh -prod_dir= -patchlist= -verbose -install
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patchlist=E5W8 -verbose -install
./bsu.sh -prod_dir=/opt/bea/weblogic92 -patchlist=E5W8,WGQJ -verbose -install
This is the syntax for the command to check if the patch is installed:
./bsu.sh -prod_dir= -patch_download_dir= -status=applied -verbose -view
For example:
./bsu.sh -prod_dir=/opt/bea/weblogic92 -status=applied -verbose -view
2. Applying the patch to the classpath manually

You can apply the patch to the system manually by extracting the actual patch and adding it to the classpath on the system:

Extract the actual patch jar file. It will be in the form .jar (for example: E5W8.jar). Inside this jar file is the actual patch jar file, which will be of the form CR326566_92mp3.jar. Extract the latter file for the following steps.
Add the extracted jar file as the first element of the classpath of the Admin server as well as the managed servers in the domain.
If you are starting servers using the WebLogic Server startup script, update the classpath in the startup script like this:
set CLASSPATH=\jars\CR326566_92mp3.jar;%CLASSPATH% (Windows)
CLASSPATH=/jars/CR326566_92mp3.jar:$CLASSPATH (UNIX)
where PATCH_DIR is the directory on the machine where you extracted/saved the patch file.
Similarly, if you are starting servers using Node Manager, add the patch jar to the beginning of the Class Path argument in the Server Start tab for the server(s).
NOTE: Applying the patch to the classpath manually (approach 2) is recommended only for releases prior to WLS 9.1. Smart Update should be used when it is available as it provides patch conflict and dependency checking.
REFERENCES

上面的第一种方法是要连接网络通过GUI界面或者./bsu.sh的命令行接口来进行但是这也是一种使用不了的方法因为很少有人把weblogic服务器置于连网状态.
所以还是使用第二种方法先下补丁Patch 7669814,然后将补丁包中的.jar文件复制到weblogic服务器主机上然后将.jar包添加到classpath环境变量中.

补丁Patch 7669814包中的文件为KG6I.jar将其复制到weblogic的lib目录下

[root@sx-weblogic32 lib]# ls -lrt
total 28
-rw-r----- 1 root root   702 Mar 29  2011 readme.txt
-rw-r--r-- 1 root root 23302 Jun 10  2011 KG6I.jar

下面就需要修改weblogic的启动脚本将KG6I.jar文件加载到classpath环境变量中然后再重新启动weblogic
在startWebLogic.sh中增加下面的记录
CLASSPATH=/bea11/user_projects/domains/mydomain/lib/KG6I.jar:$CLASSPATH

在重新启动weblogic之后再来更新应用程序

####<2014-7-11 上午08时54分05秒 CST>     < [ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040045244>   
####<2014-7-11 上午08时54分05秒 CST>     < [ACTIVE] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040045438>   
####<2014-7-11 上午08时54分05秒 CST>     < [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040045474>   
####<2014-7-11 上午08时54分05秒 CST>     < [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040045476>   
####<2014-7-11 上午08时54分05秒 CST>     < [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040045730>  < [CompressingFilter/1.4.6] CompressingFilter has initialized> 
####<2014-7-11 上午08时54分14秒 CST>     < [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040054094>   
####<2014-7-11 上午08时54分14秒 CST>     < [ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040054094>   
####<2014-7-11 上午08时54分14秒 CST>     < [ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> < > <> <> <1405040054111>