Weblogic BEA-141281 unable to get file lock, will retry 故障处理

今天兄弟单位的一台应用服务器需要从测试环境移交到机房并修改IP,在移交前关闭了应用服务器,在移交后启动weblogic时出现了问题,weblogic进程起来了,但是控制台进不了不能修改数据源设置(也就是修改jdbc连接串)。操作下如下:

手动执行启动脚本(但这里其实另一个同事已经执行过一次)

[root@cdydtest bin]# ./startWebLogic.sh
.
.
JAVA Memory arguments: -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=256m
.
WLS Start Mode=Development
.
CLASSPATH=/usr/bea/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/bea/jdk1.6.0_20/lib/tools.jar:/usr/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/usr/bea/wlserver_10.3/server/lib/weblogic.jar:/usr/bea/modules/features/weblogic.server.modules_10.3.6.0.jar:/usr/bea/wlserver_10.3/server/lib/webservices.jar:/usr/bea/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/usr/bea/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/usr/bea/wlserver_10.3/common/derby/lib/derbyclient.jar:/usr/bea/wlserver_10.3/server/lib/xqrl.jar
.
PATH=/usr/bea/wlserver_10.3/server/bin:/usr/bea/modules/org.apache.ant_1.7.1/bin:/usr/bea/jdk1.6.0_20/jre/bin:/usr/bea/jdk1.6.0_20/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
Starting WLS with line:
/usr/bea/jdk1.6.0_20/bin/java -client   -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy  -Xverify:none  -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server   -Dweblogic.management.discover=true  -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath  weblogic.Server

因为输出日志被重定向到了日志文件(47ggzj.log)中,所以这里没有显示完整的日志信息。

抓取java进程来判断是否weblogic已经启动

[root@cdydtest bin]# ps -ef | grep java
root     11933     1  0 14:23 ?        00:00:17 /usr/bea/jdk1.6.0_20/bin/java -client -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server

root     14675     1 20 15:10 pts/3    00:00:01 /usr/bea/jdk1.6.0_20/bin/java -client -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy -Xverify:none -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath weblogic.Server
root     14695 14567  0 15:10 pts/3    00:00:00 grep java

从上面的信息看出现了两个weblogic进程(pid为11933,14675)在运行,这是因为两个同事都手动执行了一次启动脚本。但登录不了weblogic控制台,如是查看weblogic日志文件。

[root@cdydtest base_domain]# cat 47ggzj.log
<nov 19, 2015 3:10:34 PM CST> <info> <security> <bea -090905> <disabling CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify -Dweblogic.security.allowCryptoJDefaultJCEVerification=true>
<nov 19, 2015 3:10:34 PM CST> <info> <security> <bea -090906> <changing the default Random Number Generator in RSA CryptoJ from ECDRBG to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true>
<nov 19, 2015 3:10:35 PM CST> <info> <weblogicserver> <bea -000377> <starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 16.3-b01 from Sun Microsystems Inc.>
<nov 19, 2015 3:10:45 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:10:55 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:11:05 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:11:15 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:11:25 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:11:35 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:11:45 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:11:55 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:12:05 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:12:15 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:12:25 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:12:35 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:12:45 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:12:55 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>
<nov 19, 2015 3:13:06 PM CST> <info> <management> <bea -141281> <unable to get file lock, will retry ***230;>

从上面消息可以看到不能获得文件锁,虽然weblogic进程已经在运行,但是不能执行任何操作。这里出现这个问题的原因是因为需要修改jdbc连接,因为数据库服务器的IP地址修改了,而原来在weblogic中配置的jdbc并没有修改,那么在启动weblogic时,就会一直尝试连接,在这时weblogic服务是没有成功启动的,也登录不了控制台,但尝试连接达到weblogic缺的次数后,就会放弃尝试连接而执行后续的启动操作,但这需要等待一定的时间,而这时业务人员说不能登录系统,一位同事如是登录weblogic控制不能登录,如是再次执行了一次启动脚本,所以出现了不能获得文件锁的问题。现在的处理方法是kill掉这两个weblogic进程,并删除被锁定的AdminServer.lok文件,再次执行weblogic启动脚本就能正常启动。

删除被锁定的AdminServer.lok文件

[root@cdydtest /]#cd /usr/bea/user_projects/domains/base_domain/servers/AdminServer/tmp

[root@cdydtest tmp]# ls
AdminServer.lok  WebServiceUtils.ser  _WL_internal  _WL_user
[root@cdydtest tmp]# rm AdminServer.lok
rm: remove regular empty file `AdminServer.lok'? y

手动执行启动脚本

[root@cdydtest bin]# ./startWebLogic.sh
.
.
JAVA Memory arguments: -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=256m
.
WLS Start Mode=Development
.
CLASSPATH=/usr/bea/patch_wls1036/profiles/default/sys_manifest_classpath/weblogic_patch.jar:/usr/bea/jdk1.6.0_20/lib/tools.jar:/usr/bea/wlserver_10.3/server/lib/weblogic_sp.jar:/usr/bea/wlserver_10.3/server/lib/weblogic.jar:/usr/bea/modules/features/weblogic.server.modules_10.3.6.0.jar:/usr/bea/wlserver_10.3/server/lib/webservices.jar:/usr/bea/modules/org.apache.ant_1.7.1/lib/ant-all.jar:/usr/bea/modules/net.sf.antcontrib_1.1.0.0_1-0b2/lib/ant-contrib.jar:/usr/bea/wlserver_10.3/common/derby/lib/derbyclient.jar:/usr/bea/wlserver_10.3/server/lib/xqrl.jar
.
PATH=/usr/bea/wlserver_10.3/server/bin:/usr/bea/modules/org.apache.ant_1.7.1/bin:/usr/bea/jdk1.6.0_20/jre/bin:/usr/bea/jdk1.6.0_20/bin:/usr/lib64/qt-3.3/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
.
***************************************************
*  To start WebLogic Server, use a username and   *
*  password assigned to an admin-level user.  For *
*  server administration, use the WebLogic Server *
*  console at http://hostname:port/console        *
***************************************************
starting weblogic with Java version:
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
Starting WLS with line:
/usr/bea/jdk1.6.0_20/bin/java -client   -Xms300m -Xmx300m -XX:CompileThreshold=8000 -XX:PermSize=256m  -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=/usr/bea/wlserver_10.3/server/lib/weblogic.policy  -Xverify:none  -da -Dplatform.home=/usr/bea/wlserver_10.3 -Dwls.home=/usr/bea/wlserver_10.3/server -Dweblogic.home=/usr/bea/wlserver_10.3/server   -Dweblogic.management.discover=true  -Dwlw.iterativeDev= -Dwlw.testConsole= -Dwlw.logErrorsToConsole= -Dweblogic.ext.dirs=/usr/bea/patch_wls1036/profiles/default/sysext_manifest_classpath  weblogic.Server

查看日志信息

[root@cdydtest base_domain]# cat 47ggzj.log
2015-11-19 15:23:46 Initializing Insur_CHANGDE from init-parameters:
PowerSI Version v0.2.9(Build20140901)
starttime:2015-11-19 15:23:46
trace_busiconn:1
trace_dbconn:0
hostname:cdydtest
jdbclogger:/usr/bea/user_projects/domains/base_domain/applications/Insur_CHANGDE/WEB-INF/jdbclogger.properties
homedir:/usr/bea/user_projects/domains/base_domain/applications/Insur_CHANGDE
companyname:????
jdbclogger.maxBatchCount:10
jdbclogger.minRuntime:1000
jdbclogger.needCaller:true
service_centerid:
applicationname:Insur_CHANGDE
scheduler_flag:0
logger:/usr/bea/user_projects/domains/base_domain/applications/Insur_CHANGDE/WEB-INF/log4j.properties
log_level:0
instancename:cdydtest.Insur_CHANGDE

<nov 19, 2015 3:23:46 PM CST> <notice> <loggingservice> <bea -320400> <the log file /usr/bea/user_projects/domains/base_domain/servers/AdminServer/logs/base_domain.log will be rotated. Reopen the log file if tailing has stopped. This can happen on some platforms like Windows.>
<nov 19, 2015 3:23:46 PM CST> <notice> <loggingservice> <bea -320401> <the log file has been rotated to /usr/bea/user_projects/domains/base_domain/servers/AdminServer/logs/base_domain.log00027. Log messages will continue to be logged in /usr/bea/user_projects/domains/base_domain/servers/AdminServer/logs/base_domain.log.>
<nov 19, 2015 3:23:46 PM CST> <notice> <log Management> <bea -170027> <the Server has established connection with the Domain level Diagnostic Service successfully.>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000365> <server state changed to ADMIN>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000365> <server state changed to RESUMING>
<nov 19, 2015 3:23:46 PM CST> <notice> <server> <bea -002613> <channel "Default" is now listening on 10.138.130.251:7001 for protocols iiop, t3, ldap, snmp, http.>
<nov 19, 2015 3:23:46 PM CST> <notice> <server> <bea -002613> <channel "Default[1]" is now listening on 127.0.0.1:7001 for protocols iiop, t3, ldap, snmp, http.>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000331> <started WebLogic Admin Server "AdminServer" for domain "base_domain" running in Development Mode>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000365> <server state changed to RUNNING>
<nov 19, 2015 3:23:46 PM CST> <notice> <weblogicserver> <bea -000360> <server started in RUNNING mode>

从上面信息可知weblogic成功启动。从这个故障的原因来看就是操作人员在处理问题时不够细心,在不能登录weblogic控制台时并没有检查当前已经启动weblogic服务,而就执行了启动脚本才产生的问题。处理问题时一定要搞清状况,了解必要的信息,弄清原因才能操作。

Proudly powered by WordPress | Indrajeet by Sus Hill.