Howto use JBoss Operation Network (JON) for your provisioning

Post summary:
This post will illustrate how you can set up a Red Hat middleware provisioning environment using Linux (Fedora), KVM (virtualisation), JBoss EAP (application server) and JBoss Operations Network (supervision) and Apache httpd (mod_cluster).
A use case could be : JON detects that the an application deployed on EAP can not handle the charge any more and triggers the creation of a new virtual machine that contains a new JBoss EAP. Via a bundle the application is deployed in EAP who joins the cluster.

Main post :
Here I more or less dumped the technical steps to follow without explaining step by steps (as it's a long post use ctrl+F to be happy)

On host:

• sudo service sshd start

Connect to JON VM:

• cd /app/script
• ./initSshId.sh (removing the file /etc/.ssh/know_hosts might be necessary)
• Creation of a soft link for your script from the JON/bin directory to the script directory. As you can see in the next screen shot, this will allow JON to discover automatically those scripts (without soft link you would need a plugin)
  cd  /app/jon/jon-server/bin
  ln -s /app/script/virsh-start.sh virsh-start.sh

        

• Start services
  service postgresql restart
  service jon restart
  service jon-agent restart
  service httpd restart
  (Apache with mod_cluster installed: http://www.jboss.org/mod_cluster)

• Start JON
  cd /app/jon/jon-server/bin
  ./rhq-server.sh stop/start
• virsh-start.sh DemoLoad-1

On Host
sudo chkconfig iptables off
sudo service iptables stop
http://192.168.100.150:7080
usr: rhqadmin / rhqadmin

JON provisioning
consists of 2 things:
- bundle (files and spec of your application to be deployed)

- destination (group and directories)

Click on execute

To deploy a bundle 5.5. Deploying Bundles to a Platform

• Create group
  
• Click depoy

Correct the httpd plugin within JON

• By default the httpd shows an error (known issue http://jira.rhq-project.org/browse/RHQ-1177 )
• Set the URL to http://192.168.100.150:80/ as shown in the screen shot
  (make sure that you set 'http://' in front of the IP, otherwise you will get error:Unable to update this resource's connection properties. org.rhq.core.clientapi.agent.discovery.InvalidPluginConfigurationClientException: Unable to connect to managed resource of type [Apache HTTP Server] using the specified connection properties. Failed to start component for resource Resource[id=10917, type=Apache HTTP Server, key=/etc/httpd, name=/etc/httpd/conf/httpd.conf, parent=f13-jon, version=2.2.15]. at org.rhq.core.pc.inventory.InventoryManager.updatePluginConfiguration(InventoryManager.java:483)

• After this To correct it configure URL with value http://192.168.100.150 and the exclamation mark should disappear

            

Howto install a plugin

• Download jon-plugin-pack-eap-2.4.0.GA.zip
  cd /app/jon/jon-plugin-pack-eap-2.4.0.GA
  cp jopr-rhq-server-plugin-3.0.0.GA.jar /app/jon/jon-server/jbossas/server/default/deploy/rhq.ear/rhq-downloads/rhq-plugins/
• Restart JON
  service jon stop
  service jon start
• Now tell the JON agent to get all new plugins
  (in background the plugin-jars will be copied into /app/jon/rhq-agent/plugins/)

In the Overview choose 'RHQ Agent' --> Operations --> New --> 'Update All Plugins --> scroll down --> hit 'SCHEDULE'

Scroll down and add 'plugins' in the command line and hit 'Schedule'
more commands

• Adding your JBoss AS server to the supervised list

Naming Provider URL *
jnp://192.168.100.150:2099

JBoss Home Directory *
/app/jon/jon-server/jbossas

Server Home Dir *
/app/jon/jon-server/jbossas/server/default

Bind Address
192.168.100.150

Availability Check Period : 60 sec

After that you should see your JBoss AS Server on the left

Add DemoLoad-1 server

Naming Provider URL *

jnp://192.168.100.172:1099

JBoss Home Directory *

/app/jboss/jboss

Server Home Dir *

/app/jboss/jboss/server/default

Bind Address

192.168.100.172

-----------------

Configure JBoss: 
/etc/init.d/jboss is pointing to 

/app/jboss/jboss/bin/jboss_init_redhat.sh

modifiy :

JAVA_HOME=/app/java/java

JBOSS_HOME=${JBOSS_HOME:-"/app/jboss/jboss"}

JBOSS_BIND_ADDRESS=192.168.100.172

Configure RHQ agent: 
/etc/init.d/jon-agent is pointing to 
/app/rhq-agent/bin/rhq-agent-wrapper.sh
(nothing to modify)

Add JAVA HOME to /bin/rhq-agent-env.sh:
RHQ_AGENT_JAVA_HOME="/app/java/java"

Change startup script
sudo chkconfig --level 3 jon-agent off
(this makes sure that JON agent starts after the JON server!)

Configure mod_cluster:

Check if the following URLs are available:

http://192.168.100.150/mod_cluster-manager
http://192.168.100.150/load-demo/

Apache httpd configuration
1. Mod_cluster
2. SNMP

Modules: /etc/httpd/modules

mod_advertise.so
mod_jk.so
mod_manager.so
mod_proxy_cluster.so
mod_slotmem.so
libsnmpcommon.so
libsnmpsubagt.so
libsnmpmonagt.so

Configuration in /etc/httpd/conf.d
mod_cluster.conf

 Listen 192.168.100.150:6666
   ServerName 192.168.100.150
      Order deny,allow
      Deny from all
      Allow from 127.0.0.
      Allow from 192.168.100.
  
 KeepAliveTimeout 60
 MaxKeepAliveRequests 0
 ManagerBalancerName mycluster
 AdvertiseFrequency 5


   SetHandler mod_cluster-manager
   Order deny,allow
   Deny from all
   Allow from 127.0.0.1
   Allow from 192.168.100.


Howto setup scripts manage-virt.sh
On each virtual machine you have to modify the network file, so that the script manage-virt.sh can govern the different machines via the DHCP name:
/etc/sysconfig/network-scripts/ifcfg-eth0 (or 1)

Should look like this:
# Network Interface
DEVICE=eth0
HWADDR=52:54:00:1B:1B:20
ONBOOT=yes
DHCP_HOSTNAME=f13-jon

Howto setup PostgreSQL database


Change within the file: /var/lib/pgsql/data/postgresql.conf
listen_addresses='*'
http://www.postgresql.org/docs/9.0/static/auth-pg-hba-conf.html
/var/lib/pgsql/data/pg_hba.conf
host all all 0.0.0.0/0 trust
service postgresql restart

SQL commands can be executed via Web interface
http://192.168.100.150:7080/admin/test/sql.jsp

Troubleshooting

1. Troubleshooting JON Agent 1

Failed to start the agent
java.net.BindException: Cannot assign requested address

...

Agent will be restarted soon to see if the failure condition cleared up

./rhq-agent.sh -l

(reconfigure)

2. Troubleshooting JON Agent 2
[root@f13-v2 ~]# service jon-agent start
Starting RHQ Agent...
RHQ Agent (pid 1687) is NOT running
Failed to start - make sure the RHQ Agent is fully configured properly

TODO: add --- RHQ_AGENT_JAVA_HOME="/app/java/java" --- into rhq-agent-env.sh 

3. Troubleshooting WARN message


WARN  [ConfigurationManager.threadpool-1] (rhq.core.pc.configuration.ConfigurationCheckExecutor)- Unable to check for updated configuration
java.lang.RuntimeException: Configuration management is currently disabled for the ResourceType[id=0, category=Service, name=java.util.logging, plugin=RHQAgent] Resource with key [java.util.logging:type=Logging] - it can be enabled in the Resource's Inventory > Connection tab.

Choose in the inventory tab of the machine : yes / yes

4. Troubleshoot adding a new agent
This happens every time another agent connects to JON server --> bug, or other major updates within the system happen, pretty unstable

Alert triggering scripts
An interesting use case is that when one ressource is triggering an alert, another ressource can execute an operation.
After having set up an alert you can set up an operation on a alert :

References
http://rhq-project.org/display/RHQ/Bundles+Sprint+Demo
http://rhq-project.org/display/RHQ/Design-Bundle+Ant+Launcher
http://rhq-project.org/display/RHQ/Design-Bundle+Ant+Tasks

http://mazz.fedorapeople.org/downloads/sample-bundles/
Slide show:
http://mazz.fedorapeople.org/demos/provisioning_beta/prov-beta.htm
Java API of JON:
http://www.redhat.com/docs/en-US/JBoss_ON/2.4/api/domain/index.html?org/rhq/core/domain/content