RSS Feed

Home
About Us
Join Us
Contact Us
Disclaimer
Beginners Corner
General WebLogic
EJB
WLST
Ant
JMX
Debuggers
Proxy
Parser Issues
Work Managers
Web Services
Spring
Coherence
Hibernate
SOA
JVM
Security
JMS
Forums
Testimonials

Tag: Stuck Thread

8
Mar/11

Stuck Thread Alert using WebLogic SNMP Agent

by JaySenSharma under Alert, SNMP
Hi,

Jay SenSharma

Based on a requirement of a user named Shashi_sr on Oracle Forums we have created this post. Getting Alert is one of the most important requirement in
production environments. We can use many techinques to get alerts regarding the behaviour and scenario of WebLogic Server. WLST, JMX and WLDF are the most
common techniques to monitor our WebLogic server resources and the health check of these resources.

Here we are going to see a very simple example of creating a STUCK THREAD alert using SNMP Log Filters. So that as soon as any thread will be declared by
WebLogic as STUCK we will immediately get an SNMP Alert.

You can also do the same thing using WLST script have a look at the post Sending Email Alert For Stuck Threads With Thread Dumps

Step1). Login to AdminConsole and then nevigate to Diagnostics> SNMP

Step2). Now create an SNMP Agent like following:

Name: StuckThreadSNMPAgent
Enabled: true
Targeted Servers: AdminServer
UDP Port : 161
Community Prefix: public
Send Automatic Traps Enabled : true
 Community Based Access Enabled : true
Inform Retry Interval : 10000
Maximum Inform Retry Count : 1
Credential Cache Invalidation Interval : 3600000

LogFilter_StuckThread_1

Step3). Now create a Log Filter By clicking on

Diagnostics->SNMP>StuckThreadSNMPAgent>Configuration (Tab)>Log Filters

like following:
Name: StuckThreadLogFilter
Enabled Servers: AdminServer
Severity Level: Error
Subsystem Names: WebLogicServer
Message IDs : 000337
Message Substring : [STUCK] ExecuteThread
Like following:

LogFilter_StuckThread_2

Step4). Now Create a Trap Destination like following:

Diagnostics->SNMP>StuckThreadSNMPAgent>Configuration (Tab)>Trap Destinations

Name: StuckThreadTrapDest
Community: public
Host: localhost
Port : 165
as following:

LogFilter_StuckThread_3

Step5). Now restart your Server Just to make sure that every thing is OK.

Step6). Now open a command prompt and then run setWLSEnv.cmd/ setWLSEnv.sh then start the

java weblogic.diagnostics.snmp.cmdline.Manager SnmpTrapMonitor -p 165

 LogFilter_StuckThread_4

As soon as you will get any STUCK THREAD you will get an alert immediately as following:

LogFilter_StuckThread_5

.
.
Thank
Jay SenSharma
SNMP, Stuck Thread 12 Comments more...

21
Feb/11

Sending Email Alert for Threads Pool Health Using WLST

by Ravish Mody under Alert, WLST

Ravish Mody

This post is written considering to send an email alert message to Admins which would alert them about the Thread Pool Health State. All of us know if in Weblogic
Health State is as WARNING state then it means something is not going correct in the server and we have to check it. Thus this WLST script would surely help all the
Weblogic Administrators who are working in Production Environment.

In below script has a properties file in which you can give all the details about the domain as well as time interval and the number of times the ratio has to be checked.

Steps to Create an Email Alert for Threads Pool Health

Step1) Create a Directory somewhere in your file system like : C:\WLST

Step2) Write a Properties file domains.properties inside C:\WLST like following:

01 admin.url=t3://localhost:8001
02 admin.username=weblogic
 03 admin.password=weblogic
04
05 # Number of times the RATIO has to be checked
06 checkTimes_Number=25
07
08 # TIME INTERVAL between number of times the RATIO has to be checked (30000 milliseconds = 30 seconds)
09 checkInterval_in_Milliseconds=30000
10
11 ############ Accouding to the above values the checker will run for total 25 times in an interval of 30 seconds ea

Step-3) Create a WLST Script somewhere in your file system with some name like Alert_ThreadPoolHealth.py inside C:\WLST contents will be something like
following:

01 #############################################################################
02 #
03 # @author Copyright (c) 2010 - 2011 by Middleware Magic, All Rights Reserved.
04 #
05 #############################################################################
06
07 from java.io import FileInputStream
08 import java.lang
09 import os
10 import string
11
12 propInputStream = FileInputStream("domains.properties")
13 configProps = Properties()
14 configProps.load(propInputStream)
15
16 serverurl = configProps.get("server.url")
17 adminUser = configProps.get("admin.username")
18 adminPassword = configProps.get("admin.password")
19 checkTimes_Number = configProps.get("checkTimes_Number")
20 checkInterval_in_Milliseconds = configProps.get("checkInterval_in_Milliseconds")
21
22 i = 0
23 y = int(checkTimes_Number)
24
25 ############# This method would send the Alert Email #################
26 def sendMailString():
27 os.system('/bin/mailx -s "ALERT: Thread Pool Health is in WARNING !!! " ABCD@COMPANY.com < rw_file')
28 print '********* ALERT MAIL HAS BEEN SENT ***********'
29 print ''
30
31 ############# This method is checking the Thread Pool Health #################
32 def alertThreadPoolHealth(healthState):
33 state ="Checking HealthState: " + healthState
34 check = string.find(state,"HEALTH_WARN")
35 if check != -1:
36 print '!!!! ALERT !!!! Thread Pool Health is in WARNING State'
37 print ''
38 message = 'Please Check the Thread Pool is in WARNING State.'
39 cmd = "echo " + message +" > rw_file"
40 os.system(cmd)
41 sendMailString()
42 else:
43 print 'Everything is working fine till now'
44
45 connect(adminUser,adminPassword,serverurl)
46 serverRuntime()
47 cd('ThreadPoolRuntime/ThreadPoolRuntime')
48
49 while (i < y):
50 ls()
51 healthState=cmo.getHealthState();
52 alertThreadPoolHealth(str(healthState));
53 Thread.sleep(int(checkInterval_in_Milliseconds))
54 i = i + 1

Step-4) Open a command prompt and then run the setWLSEnv.cmd or setWLSEnv.sh to set the CLASSPATH and PATH variables. Better you do echo
%CLASSPATH% or echo $CLASSPATH to see whether the CLASSPATH is set properly or not. If you see an Empty Classpath even after running the
setWLSEnv.sh then please refer to the Note mentioned at Step3) in the Following post: http://middlewaremagic.com/weblogic/?page_id=1492

Step-5) Now run the WLST Script in the same command prompt like following:

1 java weblogic.WLST Alert_ThreadPoolHealth.py

You will see the following kind of results in the command prompt

01 java weblogic.WLST Alert_ThreadPoolHealth.py

 02
03 Initializing WebLogic Scripting Tool (WLST) ...
04 Welcome to WebLogic Server Administration Scripting Shell
05 Type help() for help on available commands
06
07 Connecting to t3://localhost:8001 with userid weblogic ...
08 Successfully connected to Admin Server 'AdminServer' that belongs to domain 'Domain_8001'.
09
10 Warning: An insecure protocol was used to connect to the
11 server. To ensure on-the-wire security, the SSL port or
12 Admin port should be used instead.
13
14 Location changed to serverRuntime tree. This is a read-only tree with ServerRuntimeMBean as the root.
15 For more help, use help(serverRuntime)
16 -r-- CompletedRequestCount 3714
17 -r-- ExecuteThreadIdleCount 0
18 -r-- ExecuteThreadTotalCount 6
19 -r-- ExecuteThreads weblogic.work.ExecuteThreadRuntime[weblogic.work.ExecuteThread
20 -r-- HealthState Component:threadpool,State:HEALTH_WARN,MBean:ThreadPoolRuntime
21 -r-- HoggingThreadCount 2
22 -r-- MinThreadsConstraintsCompleted 94
23 -r-- MinThreadsConstraintsPending 0
24 -r-- Name ThreadPoolRuntime
25 -r-- PendingUserRequestCount 0
26 -r-- QueueLength 0
27 -r-- SharedCapacityForWorkManagers 65536
28 -r-- StandbyThreadCount 3
29 -r-- Suspended false
30 -r-- Throughput 9.495252373813093
31 -r-- Type ThreadPoolRuntime
32 -r-x preDeregister Void :
33
34 !!!! ALERT !!!! Thread Pool Health is in WARNING State
35
36 ********* ALERT MAIL HAS BEEN SENT ***********

NOTE: This script is using mailx (i.e. but Windows box does not have mailx utility) so please do check if your mailx is configured properly
or else script would run properly but the mail would not be sent.

Alert Email

Regards,

Ravish Mody

Alert, Stuck Thread, Thread Pool, WLST 9 Comments more...

15
Feb/11

Sending Email Alert for Hogger Threads Count Using WLST

by Ravish Mody under Common Category, WLST
 Ravish Mody

This post is written considering to send an email alert message to Admins which would alert them about the hogger threads. Hogging threads can be called as a
candidates for stuck threads in other words, those threads that might get stuck are called hogging threads. These threads will be declared as stuck threads after
StuckThreadMaxTimeout seconds which by default value is 600secs.

Most of you guys might agree with me, that no one wants stuck threads in there production environment. Hence this script would surely help you guys to take
an relevant actions once you get an alert about the hogging threads have exceeded the given ration.
In below script we have used a properties file in which you can give all the details about the domain as well as hogger thread ration, time interval and the number of
times the ratio has to be checked.

Steps to Create an Email Alert for Hogger Threads Count

Step1) Create a Directory somewhere in your file system like : C:\WLST

Step2) Write a Properties file domains.properties inside C:\WLST like following:

01 admin.url=t3://localhost:7001
02 admin.username=weblogic
03 admin.password=weblogic
04
05 # This ExecuteThread_Vs_HoggerThreadRatio represtents the division of ExecuteThread/HoggerThreadRatio
06 ExecuteThread_Vs_HoggerThreadRatio=2
07
08 # Number of times the RATIO has to be checked
09 checkTimes_Number=25
10
11 # TIME INTERVAL between number of times the RATIO has to be checked (30000 milliseconds = 30 seconds)
12 checkInterval_in_Milliseconds=30000
13
14 ############ Accouding to the above values the checker will run for total 25 times in an interval of 30 seconds ea

Step-3) Create a WLST Script somewhere in your file system with some name like Alert_HoggerThreadCoung.py inside C:\WLST contents will be something
like following:

UPDATED [26-04-2011]: This script has been updated based on the comment > http://middlewaremagic.com/weblogic/?p=5423#comment-3760, thanks to
sathya.

01 #############################################################################
02 #
03 # @author Copyright (c) 2010 - 2011 by Middleware Magic, All Rights Reserved.
04 #
05 #############################################################################
06
07 from java.io import FileInputStream
08 import java.lang
09 import os
10
11 propInputStream = FileInputStream("domains.properties")
12 configProps = Properties()
13 configProps.load(propInputStream)
14
15 adminUrl = configProps.get("admin.url")
16 adminUser = configProps.get("admin.username")
17 adminPassword = configProps.get("admin.password")
18 executeThread_Vs_HoggerThreadRatio = configProps.get("ExecuteThread_Vs_HoggerThreadRatio")
19 checkTimes_Number = configProps.get("checkTimes_Number")
20 checkInterval_in_Milliseconds = configProps.get("checkInterval_in_Milliseconds")
21
22 i = 0
23 y = int(checkTimes_Number)
24
25 ############# This method would send the Alert Email #################
26 def sendMailString():
27 os.system('/bin/mailx -s "ALERT: Hogger Thread Count Exceeded the Limt !!! " abcd@company.com < rw_file')
28 print '********* ALERT MAIL HAS BEEN SENT ***********'
29 print ''
30
31 ############# This method is checking the Hogger Threads Ratio #################
32 def alertHoggerThreads(executeTTC , hoggerTC):
33 print 'Execute Threads : ', executeTTC
34 print 'Hogger Thread Count : ', hoggerTC
35 if hoggerTC != 0:
36 ratio=(executeTTC/hoggerTC)
37 print 'Ratio : ' , ratio
38 print ''
39 if (int(ratio) <= int(executeThread_Vs_HoggerThreadRatio)):
40 print ' !!!! ALERT !!!! Stuck Threads are on its way.....'
41 print ''
42 message = 'ExecuteThreads Count= ' + str(executeTTC) + ' HoggingThreads= '+ str(hoggerTC) +' Exec
43 cmd = "echo " + message +" > rw_file"
44 os.system(cmd)
45 sendMailString()
46 else:
47 print '++++++++++++++++++++++++++++++++++++'
48 print 'Everything is working fine till now'
49 print '++++++++++++++++++++++++++++++++++++'
50 else:
51 print '++++++++++++++++++++++++++++++++++++'
52 print 'Everything is working fine till now'
53 print '++++++++++++++++++++++++++++++++++++'
54
55 connect(adminUser,adminPassword,adminUrl)
56 serverRuntime()
57 cd('ThreadPoolRuntime/ThreadPoolRuntime')
58
59 while (i < y):
60 ls()
61 executeTTC=cmo.getExecuteThreadTotalCount();
62 hoggerTC=cmo.getHoggingThreadCount();
63 alertHoggerThreads(executeTTC , hoggerTC)
64 print 'Sleeping for ', int(checkInterval_in_Milliseconds) , ' ...'
65 print ''
66 Thread.sleep(int(checkInterval_in_Milliseconds))
67 i = i + 1

Step-4) Open a command prompt and then run the setWLSEnv.cmd or setWLSEnv.sh to set the CLASSPATH and PATH variables. Better you do echo
%CLASSPATH% or echo $CLASSPATH to see whether the CLASSPATH is set properly or not. If you see an Empty Classpath even after running the
setWLSEnv.sh then please refer to the Note mentioned at Step3) in the Following post: http://middlewaremagic.com/weblogic/?page_id=1492

Step-5) Now run the WLST Script in the same command prompt using the following command:

1 java weblogic.WLST Alert_HoggerThreadCoung.py

You will see the following kind of results in the command prompt

01 Initializing WebLogic Scripting Tool (WLST) ...

02 ]$ java weblogic.WLST Alert_HoggerThreadCoung.py
03
04 Initializing WebLogic Scripting Tool (WLST) ...
05
06 Welcome to WebLogic Server Administration Scripting Shell
07
08 Type help() for help on available commands
09
10 Connecting to t3://localhost:7001 with userid weblogic ...
11 Successfully connected to Admin Server 'AdminServer' that belongs to domain 'Domain_7001'.
12
13 Warning: An insecure protocol was used to connect to the
14 server. To ensure on-the-wire security, the SSL port or
15 Admin port should be used instead.
16
17 Location changed to serverRuntime tree. This is a read-only tree with ServerRuntimeMBean as the root.
18 For more help, use help(serverRuntime)
19
20 -r-- CompletedRequestCount 2606
21 -r-- ExecuteThreadIdleCount 1
22 -r-- ExecuteThreadTotalCount 6
23 -r-- ExecuteThreads weblogic.work.ExecuteThreadRuntime[weblogic.work.ExecuteThread
24 -r-- HealthState Component:threadpool,State:HEALTH_OK,MBean:ThreadPoolRuntime,R
25 -r-- HoggingThreadCount 0
26 -r-- MinThreadsConstraintsCompleted 104
27 -r-- MinThreadsConstraintsPending 0
28 -r-- Name ThreadPoolRuntime
29 -r-- PendingUserRequestCount 0
30 -r-- QueueLength 0
31 -r-- SharedCapacityForWorkManagers 65536
32 -r-- StandbyThreadCount 4
33 -r-- Suspended false
34 -r-- Throughput 5.0
35 -r-- Type ThreadPoolRuntime
36
37 -r-x preDeregister Void :
38
39 Execute Threads : 6
40 Hogger Thread Count : 0
41 ++++++++++++++++++++++++++++++++++++
 42 Everything is working fine till now
43 ++++++++++++++++++++++++++++++++++++
44 Sleeping for 30000 ...

NOTE: This script is using mailx (i.e. but Windows box does not have mailx utility) so please do check if your mailx is configured properly
or else script would run properly but the mail would not be sent.

Alert Email

Regards,

Ravish Mody

Alert, Hogger Thread, Stuck Thread, WLST 13 Comments more...

Testimonials

Ravish, Now the script is working. Thank you so much.

- Pavan

Middleware Magic JBoss

Integrate your Eclipse Juno IDE with Openshift February 24, 2013
Hi, In this article we will discuss how can we integrate our Eclipse IDE to OpenShift and create new applications.Read the Rest... []
How to load jars residing over NFS in JBossAS7 classpath ? January 25, 2013
Hi, In this article we will discuss how can we load jars residing over NFS in JBoss AS-7 classpath. InRead the Rest... []
How to achieve high availability on Openshift ? January 13, 2013
Hi, High availability is a very important feature for any kind of infrastructure. As the name suggests the services shouldRead the Rest... []

Receive FREE Updates

FREE Email updates of our new posts Enter your email address:
 Middleware Magic
Like

1,009 people like Middleware Magic.

Magic Account
Register
Log in
Entries RSS
Comments RSS
WordPress.org

Top Magic Users

swordfish (Magic-1245)
sumit (Magic-955)
sathya (Magic-800)
Kiran (Magic-725)
rakeshreddy12345 (Magic-575)
kiran reddy (Magic-560)
Mauricio (Magic-515)
vinobabu2001 (Magic-515)
Andrew Diengdoh (Magic-504)
manishy (Magic-455)

Magic Archives

Recent Tech. Discussion

Amrita on Cluster/NodeManager
parish on Spring and WebLogic: Messaging, WorkManagers and Timers
Ren van Wijk on Spring and WebLogic: Messaging, WorkManagers and Timers
dljohnson69 on WebLogic Server is in ADMIN State ?
parish on Spring and WebLogic: Messaging, WorkManagers and Timers
dljohnson69 on WebLogic Server is in ADMIN State ?
Ren van Wijk on WebLogic, Coherence and the G1 Collector
jerrinot on WebLogic, Coherence and the G1 Collector
Ajeet on Sending Email Alert for WebLogic Servers Current State
Ren van Wijk on WebLogic, Coherence and the G1 Collector
jerrinot on WebLogic, Coherence and the G1 Collector
sharad on Miscellaneous
jcatalano82 on Shared Library With Plan.xml
Ren van Wijk on Setting-up Web-Tier Components in a SOA Environment
Ren van Wijk on JVM Crash And Native OutOfMemory

Sitemeter Status
 ClusterMap 7-Nov-2011 till Date

ClusterMap 6-Nov-2010 till 7-Nov-2011

Tags

Alert ANT Cluster Coherence DataSource debugging Deploy EJB EJB3 Hibernate JMS JMX JVM JVM Tuning MDB
Monitoring Plan.xml Security Spring tuning weblogic WebService WLST WorkManager
Copyright 2010-2012 Middleware Magic. All rights reserved. | Disclaimer