Byteman JBoss scripting

Summary:
When your HA system crashes with JEE application running on JBoss you might want to try JBoss Byteman, an bytecode injection tool that lets you add logging without touching the application itself.

Main post:

The steps to debug an application using
1. Download byteman-1.0.3.CP02 (for JDK5) or byteman-1.3 (for JDK6)
2. Extract the zip into $BYTEMAN_HOME
3. Modify your JBoss startup configuration by adding the following 3 lines at the end of $JBOSS_HOME/jboss-as/bin/run.conf:
# Adding extra logging to standard output (BYTEMAN_HOME needs to be addapted)
BYTEMAN_HOME="/tmp/byteman-1.x.x"
JAVA_OPTS="$JAVA_OPTS -javaagent:$BYTEMAN_HOME/lib/byteman.jar=script:$BYTEMAN_HOME/script.txt"
4. For JDK5 add the file $BYTEMAN_HOME/script.txt :
RULE trace RepositoryClassLoader.loadClassImpl entry
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT ENTRY
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") ENTRY in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl exit
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT EXIT
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") EXIT in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl CALL wait
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT INVOKE Object.wait()
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") CALL wait in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl CALL notifyAll
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT INVOKE Object.notifyAll()
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") CALL wait in " + Thread.currentThread())
ENDRULE

5. For JDK6 add the file $BYTEMAN_HOME/script.txt :
RULE trace RepositoryClassLoader.loadClassImpl entry
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT ENTRY
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") ENTRY in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl exit
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT EXIT
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") EXIT in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl CALL wait
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT CALL wait ALL
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") CALL wait in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl CALL notifyAll
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT CALL notifyAll ALL
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") CALL wait in " + Thread.currentThread())
ENDRULE

6. After having starting your JBoss you should see in the standard out a lot of logging...
7. That should help you to find memory leaks, deadlocks ...

Howto write JBoss Java MBeans to expose to a JMX Client

Summary:

Sometime you will find yourself in the situation of wanting to monitor your application, but no monitoring capability is given. So you will have to write your own Java MBean that will expose information of your JBoss application to a JMX Client.

Main post:

JBoss documentation

JBoss Wiki

Examples: jboss-eap-4.3_CP08/jboss-as/docs/examples/jmx/logging-monitor

From the JBoss SVN you can get sample MBean classes:

/svn_jboss_org/trunk/varia/src/main/java/org/jboss/jmx/examples/configuration/

Have your MBean interface extend the org.jboss.system.Service and org.jboss.system.ServiceMBean interface.

org.jboss.system.Service and org.jboss.system.ServiceMBean can be found in:  

jboss-eap-4.3_CP08/jboss-as/server/all/deploy/management/console-mgr.sar/web-console.war/applet.jar

Add the classes

The ability to add rich descriptions to attribute and operations

The ability to expose notification information 

The ability to add persistence of attributes

The ability to add custom interceptors for security and remote access through a typed interface

You can see two MBean that are almost the same (up to the fact that one extends the org.jboss.system.ServiceMBean class)

1. 

Attributes: JndiName

Operations: start / stop

2. 

Extents the class org.jboss.system.ServiceMBean -- you can see that this class gives you out of the 

Attributes: Name / JndiName / StateString / State

Operations: destroy / start / stop /create / jbossInternalLifecycle

Notificaitons

Explanation on standard Java MBean like RuntimeMXBean, ThreadMXBean, MemoryPoolMXBean, ... can be found here: http://download.oracle.com/javase/1.5.0/docs/guide/management/overview.html

Jacorb to be found in:
eap43CP08/jboss-as/server/all/lib/jacorb.jar
org.jacorb.util.threadpool.ThreadPool.class (here we have to get the information on the thread pool counter.
In class ThreadPool within method createNewThread() the following line:
localThread.setName(this.namePrefix + this.threadCount++);

Jacorb example:
http://www.jpackage.org/browser/rpm.php?jppversion=6.0&id=7709

Set up environment:
jboss-log4j.xml