Byteman JBoss scripting

Summary:
When your HA system crashes with JEE application running on JBoss you might want to try JBoss Byteman, an bytecode injection tool that lets you add logging without touching the application itself.

Main post:

The steps to debug an application using
1. Download byteman-1.0.3.CP02 (for JDK5) or byteman-1.3 (for JDK6)
2. Extract the zip into $BYTEMAN_HOME
3. Modify your JBoss startup configuration by adding the following 3 lines at the end of $JBOSS_HOME/jboss-as/bin/run.conf:
# Adding extra logging to standard output (BYTEMAN_HOME needs to be addapted)
BYTEMAN_HOME="/tmp/byteman-1.x.x"
JAVA_OPTS="$JAVA_OPTS -javaagent:$BYTEMAN_HOME/lib/byteman.jar=script:$BYTEMAN_HOME/script.txt"
4. For JDK5 add the file $BYTEMAN_HOME/script.txt :
RULE trace RepositoryClassLoader.loadClassImpl entry
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT ENTRY
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") ENTRY in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl exit
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT EXIT
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") EXIT in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl CALL wait
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT INVOKE Object.wait()
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") CALL wait in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl CALL notifyAll
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT INVOKE Object.notifyAll()
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") CALL wait in " + Thread.currentThread())
ENDRULE

5. For JDK6 add the file $BYTEMAN_HOME/script.txt :
RULE trace RepositoryClassLoader.loadClassImpl entry
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT ENTRY
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") ENTRY in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl exit
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT EXIT
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") EXIT in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl CALL wait
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT CALL wait ALL
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") CALL wait in " + Thread.currentThread())
ENDRULE

RULE trace RepositoryClassLoader.loadClassImpl CALL notifyAll
CLASS RepositoryClassLoader
METHOD loadClassImpl
AT CALL notifyAll ALL
IF TRUE
DO traceln("*** " + $0 + ".loadClassImpl(" + $1 + ", " + $2 + ", " + $3
+ ") CALL wait in " + Thread.currentThread())
ENDRULE

6. After having starting your JBoss you should see in the standard out a lot of logging...
7. That should help you to find memory leaks, deadlocks ...