Cluster Deployment of UltraESB
Currently I am working as a Software Engineer at
adroitlogic and its main product is UltraESB which is the fastest Enterprise Service Bus. UltraESB is a free and opensource product which is written in Java on top of Spring framework. It can be deployed and configured easily as a single server node or a cluster. Here you are provided with the major steps that must be followed in order to deploy UltraESB as a cluster. UltraESB uses zookeeper as the cluster management solution and so it is important to get a basic idea about zookeeper if you are not familiar with. Then I will turn into configuring and starting zookeeper and UltraESB in an Ubuntu environment.
Zookeeper
Apache Zookeeper is an opensource management solution for distributed systems which was developed as a sub project of hadoop. This provide valuable services such as distributed synchronization, distributed configuration, etc. In the cluster deployment of UltraESB, multiple ESB instances connect to same or different zookeeper servers. It is OK to have one zookeeper server to multiple ESB instances. For the reliability, many zookeeper servers can be used. Then each server knows where other nodes according to the configuration as we will discuss next. These servers provide aforementioned communicating with each other. At run time Zookeeper keeps an in memory file structure and data is written in nodes known as znodes. These nodes can be created programmetically and it is out of the scope of this discussion. Znodes can be created so that they are persisted after the zookeeper session is over. If znodes are created as ephemeral nodes, they are deleted at the end of the session.
In zookeeper there is a requirement that the number of zookeeper servers that are running must be greater than half the number of all servers in the cluster. This is known as quorum. For instance, if there are 4 zookeeper servers, there must be 4 or 3 active servers to meet this requirement. This implies when there are 4 zookeeper servers, then the maximum number of servers that can fail is 1. If 2 servers fail, then there are only 2 servers running, which is not greater than the half. If there are 3 servers then the maximum number that can fail is 1 which is same as in the case of 4 servers. So the reliability that can be achieved from 4 zookeeper servers and 3 servers is similar. Likewise the maximum number of servers that can fail in a 5 node or 6 node cluster is 2 where reliability is same. This gives a common pattern as below.
- The reliability that can be achieved from a N (N is even) zookeeper servers is similar to the reliability achieved from N-1 server.
So it is the recommendation that zookeeper servers must be deployed in odd numbers (1, 3, 5....).
Configuring and starting Zookeeper
Now we are moving to the main topics and for this you should be familiar with Ubuntu. In this deployment I use 3 zookeeper and 3 UltraESB instances. First you must have the UltraESB distribution with you. You can download it from
here or follow the steps below to build it from source code available at bitbucket.
source code using maven. If you build the source code, then the . Make sure sure have mercurial and maven installed. Open a terminal and create a directory where you need to download the source say 'zoo'.
$ mkdir zoo
$ cd zoo
$ hg clone https://bitbucket.org/adroitlogic/ultraesb
This will make a clone of source code under the directory 'ultraesb'. This will take couple of minutes and run the following commands when the download is complete.
$ cd ultraesb/
$ mvn clean install -Dmaven.test.skip=true
When the build process is successful, the distribution will be available in ~/zoo/modules/distribution/target under AGPL licence for example 'adroitlogic-ultraesb-2.0.0-SNAPSHOT-agpl.zip'. Extract it to 'zoo' and make three copies of with suitable names say UE1, UE2, UE3. For this execute the following commands.
$ cd modules/distribution/target
$ unzip adroitlogic-ultraesb-2.0.0-SNAPSHOT-agpl.zip -d ~/zoo/UE1
$ unzip adroitlogic-ultraesb-2.0.0-SNAPSHOT-agpl.zip -d ~/zoo/UE2
$ unzip adroitlogic-ultraesb-2.0.0-SNAPSHOT-agpl.zip -d ~/zoo/UE3
Since we are going to use 3 zookeeper nodes, we have to create 3 data directories where zookeeper store its data. You can create these in /var as in the example below.
$ sudo mkdir /var/zookeeper-1
$ sudo chown <your username> /var/zookeeper-1
In this manner create zookeeper-1, zookeeper-2, zookeeper-3 and provide permission for your account. (you can use a different set of names). Now you have to create a file named 'myid' in each of these directories and put 1, 2 and 3 as the content respectively. For example 'myid' file in zookeeper-1 directory contains 1.
Now open the zoo.cfg file in each UltraESB copy (eg:- ~/zoo/UE1/ultraesb-2.0.0-SNAPSHOT/conf )
and edit those in vi or any editor you like so that they contain following configurations.
UE1
tickTime=2000
dataDir=/var/zookeeper1
clientPort=2181
initLimit=5
syncLimit=2
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890
UE2
tickTime=2000
dataDir=/var/zookeeper2
clientPort=2182
initLimit=5
syncLimit=2
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890
UE3
tickTime=2000
dataDir=/var/zookeeper3
clientPort=2183
initLimit=5
syncLimit=2
server.1=localhost:2888:3888
server.2=localhost:2889:3889
server.3=localhost:2890:3890
localhost as the address and different ports have been used as we run these servers locally. Now zookeeper configuration is complete and servers can be started. For this go to bin directory of each UltraESB copy (eg: cd ~/zoo/UE1/ultraesb-2.0.0-SNAPSHOT/bin) and execute the following command.
If you executed this command for all three servers, you can verify that the server is running by trying to connect to each server using following command in the same terminal.
This runs zookeeper client that is shipped with UltraESB. When you run this command you will get an output like below.
Connecting to localhost:2181
Welcome to ZooKeeper!
JLine support is enabled
[zk: localhost:2181(CONNECTING) 0]
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0]
If you don't get the last line press Enter and you will get it if the connection is successful. Otherwise you will get this message.
[zk: localhost:2181(CONNECTING) 0]
In such a case revise the steps you followed carefully and make relevant corrections if any. In this way you can test all three servers using a this client.
Now it is time to configure the UltraESB instances so that they work in as a cluster. This is very easy as you have to do few changes in an XML file. This is ultra-root.xml which lies in 'conf' directory as zoo.cfg we discussed above (eg: ~/zoo/UE1/ultraesb-2.0.0-SNAPSHOT/conf ) .ultra-root.xml is the main Spring configuration file of UltraESB. I have modified it as following.
<?xml version="1.0" encoding="UTF-8"?>
<!--
***********************************************************************************************************************
NOTE: This is the default root configuration file of the UltraESB, and it includes most of the important
production deployment time configuration options as commented text. Do NOT try to understand this if you
are still new to the UltraESB
To Start learning about the UltraESB first refer to the sample configurations found in the samples/conf
directory which are much more simplified as tutorial examples. Also, look at the screen casts and other
documented articles found from http://adroitlogic.org
***********************************************************************************************************************
This is the default configuration file for the UltraESB and is picked up when the bin/ultraesb.sh or .bat scripts are
used, or the UltraESB is started as a service with the ultraesb-daemon.sh
To simplify management, the configuration is broken into two separate files, The ultra-root.xml (this file) is used to
define the transports, common more static configurations like clustering, caching, JMX management etc, and the
individual deployment units contain there own ultra-unit.xml file which is used to define the [more dynamic] proxy
services, sequences and endpoints to be hosted on the instance. When executing a sample configuration
(e.g. bin/ultraesb.sh -sample <n>), all configuration is held in one file for simplicity.
An UltraESB configuration is a Spring framework configuration file, and thus maybe broken into multiple fragments for
easier administration for a large configuration.
-->
<!--suppress SpringBeanNameConventionInspection, XmlUnusedNamespaceDeclaration -->
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:u="http://www.adroitlogic.org/ultraesb"
xmlns:s="http://www.springframework.org/schema/security"
xsi:schemaLocation="
http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.1.xsd
http://www.springframework.org/schema/security http://www.springframework.org/schema/security/spring-security-3.1.xsd
http://www.adroitlogic.org/ultraesb http://schemas.ultraesb.org/ultraesb-2.0.xsd">
<!--The fileCache bean is used by the HTTP/S transports to write messages received into optionally memory mapped
files for best performance. A FileCache instance facilitates the pooling and re-using of files for optimal
performance. See API documentation at http://api.adroitlogic.org for complete details on
configuration options available. Some of the configuration properties available includes:
memoryMapEnabled, memoryMapSize, warnThreshold, stopThreshold
Supports monitoring and management via JMX -->
<bean id="fileCache" class="org.adroitlogic.ultraesb.core.PooledMessageFileCache">
<constructor-arg value="tmp"/> <!--directory for temporary files (must be a local disk)-->
<constructor-arg value="200" type="int"/> <!--number of initial temporary files (~ average concurrancy x 2)-->
<!-- Advanced options See http://api.adroitlogic.org/ for more options
<constructor-arg value="true" type="boolean"/> memory map temp files
<constructor-arg value="4096" type="long"/> memory mapped size
-->
</bean>
<!-- The new RAMDisk based file cache for even better performance (since v1.5.0)
<bean id="fileCache" class="org.adroitlogic.ultraesb.core.RAMDiskFileCache">
<constructor-arg value="/tmp/ram"/> ram disk path
<constructor-arg value="200" type="int"/> initial size
<constructor-arg value="1024" type="int"/> warn threshold
<constructor-arg value="2048" type="int"/> stop threshold
<constructor-arg value="1024" type="int"/> max files on ram disk
<constructor-arg value="/tmp/overflow"/> disk to use on overflow
</bean-->
<!--The HTTP transport configuration. See API documentation at http://api.adroitlogic.org for complete details on
configuration options available.
e.g. properties such as:
bindAddress, defaultContextPath, replaceOriginServer, zeroCopyEnabled, requestFilters
Supports monitoring and management via JMX -->
<bean id="http-8280" class="org.adroitlogic.ultraesb.transport.http.HttpNIOListener">
<constructor-arg ref="fileCache"/>
<property name="port" value="8280"/>
<!-- Uncomment to set a custom context path e.g. /services/ for all proxy services
<property name="defaultContextPath" value="/services/"/>-->
<!--Advanced options See http://api.adroitlogic.org/ for more options
<property name="zeroCopyEnabled" value="false"/> use zero copy
<property name="bindAddress" value="192.168.1.3"/> a host name or IP address
<property name="noCompression" value="true"/> turn off response gzip compression, even if client requests
<property name="requestFilters"> optional request filters
<list> serve WSDLs and XSDs for a SOAP service
<bean class="org.adroitlogic.ultraesb.transport.http.ServiceResourceRequestFilter">
<property name="resourcePrefix" value="http://localhost:8280"/>
</bean> see sample 110 for basic and digest authentication
<bean class="org.adroitlogic.ultraesb.transport.http.auth.BasicAuthenticationFilter">
<property name="realmName" value="adroitlogic"/>
</bean>
</list>
</property>
<property name="enableConnectionDebug" value="true"/> turn on detailed connection debug info on errors
<property name="connectionDebugHeaders">
<set>
<value>all</value> specify HTTP headers to debug or 'none' or 'all'
</set>
</property>
<property name="properties"> advanced HttpCore NIO tuning properties
<map>
<entry key="http.socket.timeout" value="120000"/>
</map>
</property>
<property name="errorHandlers">
<map>
<entry key="404">
<bean class="org.adroitlogic.ultraesb.transport.http.AbstractErrorHandler - Implementation"/>
</entry>
</map>
</property>
-->
</bean>
<!--The HTTPS transport configuration. See API documentation at http://api.adroitlogic.org for complete details on
configuration options available.
e.g. properties inherited from the HTTP transport such as :
bindAddress, defaultContextPath, replaceOriginServer, zeroCopyEnabled, requestFilters
and properties specific to the HTTPS transport such as :
sslVerifyClient, nonProductionNoRemoteCertValidation
Supports monitoring and management via JMX -->
<bean id="https-8443" class="org.adroitlogic.ultraesb.transport.http.HttpsNIOListener">
<constructor-arg ref="fileCache"/>
<property name="sslVerifyClient" value="optional"/>
<property name="identityStorePath" value="conf/keys/identity.jks"/>
<property name="identityKeyPassword" value="password"/>
<property name="identityStorePassword" value="password"/>
<property name="trustStorePath" value="conf/keys/trust.jks"/>
<property name="trustStorePassword" value="password"/>
<property name="port" value="8443"/>
<!-- Uncomment to set a custom context path e.g. /services/ for all proxy services
<property name="defaultContextPath" value="/services/"/>-->
<!--Advanced options Refer HttpNIOListener (applies to this too) above, and als http://api.adroitlogic.org/
<property name="nonProductionNoRemoteCertValidation" value="true"/> use to turn off SSL validations for testing
-->
</bean>
<!--The HTTP transport sender configuration. See API documentation at http://api.adroitlogic.org for complete details
on configuration options available.
e.g. properties such as:
proxyHost, proxyPort, proxyBypassList, replaceUserAgent, unzipResponseEntities, zeroCopyEnabled
Supports monitoring and management via JMX -->
<bean id="http-sender" class="org.adroitlogic.ultraesb.transport.http.HttpNIOSender">
<constructor-arg ref="fileCache"/>
<!--Advanced options See http://api.adroitlogic.org/ for more options
<property name="zeroCopyEnabled" value="false"/> use zero copy
<property name="proxyHost" value="proxy.server.com"/> the proxy server to use for outgoing connections
<property name="proxyPort" value="3128"/> the proxy server port
(set a "proxyBypassList" to bypass proxy for some hosts)
<property name="unzipResponseEntities" value="false"/> do not unzip compressed responses received (e.g. no out mediation)
<property name="enableConnectionDebug" value="true"/> turn on detailed connection debug info on errors
<property name="keepalive" value="30000"/> the maximum time to keep a socket open for later reuse
<property name="keepAliveSafetyThreshold" value="5000"/>the threshold to not re-use a keepalive socket when its about to timeout
<property name="connectionDebugHeaders">
<set>
<value>all</value> specify HTTP headers to debug or 'none' or 'all'
</set>
</property>
<property name="properties"> advanced HttpCore NIO tuning properties
<map>
<entry key="http.socket.timeout" value="120000"/>
<entry key="http.connection.timeout" value="10000"/>
</map>
</property>
-->
</bean>
<!--The HTTPS transport sender configuration. See API documentation at http://api.adroitlogic.org for complete
details on configuration options available.
e.g. properties inherited from the HTTP transport such as :
proxyHost, proxyPort, proxyBypassList, replaceUserAgent, unzipResponseEntities, zeroCopyEnabled
and properties specific to the HTTPS transport such as :
hostnameVerifier, nonProductionNoRemoteCertValidation
Supports monitoring and management via JMX -->
<bean id="https-sender" class="org.adroitlogic.ultraesb.transport.http.HttpsNIOSender">
<constructor-arg ref="fileCache"/>
<!-- Advanced options see HttpNIOSender (applies to this too) above and http://api.adroitlogic.org/
<property name="hostnameVerifier" value="Default"/> Strict & DefaultAndLocalHost are options too
<property name="nonProductionNoRemoteCertValidation" value="true"/> disable SSL validation for testing
-->
</bean>
<!--
Uncomment to use Jasypt to secure passwords used on this configuration (e.g. keystore, DB passwords etc)
e.g. specify the HTTPS identity store password as <property name="identityKeyPassword" value="${secure_password}"/>
where the default conf/encypted.properties file contains the below encrypted password
secure_password=ENC(Rtk9+TzSec70ikJLwXlT9Y0tiwNCybE0)
-->
<!--<bean id="propertyConfigurer" class="org.jasypt.spring.properties.EncryptablePropertyPlaceholderConfigurer">
<constructor-arg>
<bean class="org.jasypt.encryption.pbe.StandardPBEStringEncryptor">
<property name="config">-->
<!-- Use this to specify the Jasypt password as an Environment variable (e.g. named APP_PASSWORD)-->
<!--<bean class="org.jasypt.encryption.pbe.config.EnvironmentStringPBEConfig">
<property name="algorithm" value="PBEWithMD5AndDES"/>
<property name="passwordEnvName" value="APP_PASSWORD"/>
</bean>-->
<!--A simple hard coded master password (e.g. "jasypt") for Jasypt-->
<!--<bean class="org.jasypt.encryption.pbe.config.SimpleStringPBEConfig">
<property name="algorithm" value="PBEWithMD5AndDES"/>
<property name="password" value="jasypt"/>
</bean>
</property>
</bean>
</constructor-arg>
<property name="locations">
<list>
<value>encrypted.properties</value>
</list>
</property>
</bean>-->
<!--
To encrypt the passwords using a rather simple mechanism (the same used in JBoss AS) use the following bean. To
improve the password with a salt, you could set the salted attribute to true, forcing a unique string on each run
-->
<!--<bean class="org.adroitlogic.ultraesb.util.encrypt.SecurePropertyManager" init-method="init">
<property name="salted" value="true"/>
<property name="location">
<value>encrypted.properties</value>
</property>
</bean>-->
<!-- Clustering configuration of the UltraESB server -->
<bean id="cluster-manager" class="org.adroitlogic.ultraesb.clustering.ClusterManager">
<!-- Change the value to true of this constructor argument to enable cluster controls -->
<!-- then change the following 4 properties according to your zookeeper configuration -->
<constructor-arg value="true" type="boolean"/>
<!-- Connection String of the zookeeper, which could have a comma separated ${host}:${port}
pairs. All of those should be running zookeeper instances. In the development environment
it is OK to have only one ZooKeeper instance while it is recommended to have a ZooKeeper
quorum -->
<property name="zkConnectString" value="localhost:2181,localhost:2182,localhost:2183"/>
<!-- session timeout value used in seconds when creating the ZooKeeper session from
the UltraESB to the ZooKeeper server -->
<property name="zkSessionTimeout" value="30"/>
<!-- Clustering domain the UltraESB instance is in. You can use the same ZooKeeper instances
with same connection string and different domains to isolate 2 clusters running on the same
ZooKeeper instances -->
<property name="domain" value="default"/>
<!-- Name of the node tobe used for storing node details in the cluster in ZooKeeper, it is
a must to have a unique nodeName for the UltraESB servers on a single clustering domain -->
<property name="nodeName" value="node1"/>
<!-- Timeout value in seconds to detect a cluster manager startup failure. In other words
the server startup thread will wait for the cluster manager to start maximum this amount of
seconds. The value of this property should be a positive integer for the timeout to be
effective with the following 2 exceptions;
1) setting a -1 or any minus value will result in the server startup to infinitely wait
for the cluster manager to start
2) setting this to 0 will result in the startup thread to not to wait for the cluster
manager startup completion. In this case the server startup will not worry about the
cluster manager state.
Default value of this property is 30 and it is highly recommended to have a finite non zero
value in production environments
-->
<property name="startupTimeout" value="30"/>
<!-- Up time reporter timer interval. This guarantees the session end time is reported to
an accuracy of this time interval -->
<property name="upTimeReportInterval" value="60"/>
<!-- The failover processing configuration. Uncomment this and configure the failover-processor bean
to activate failover of nodes in a clustered deployment -->
<!--<property name="failoverProcessor" ref="failover-processor"/>-->
</bean>
<!-- Uncomment, link to the cluster manager with "failoverProcessor" property and configure to activate
clustered node failover functionality to transfer the failed node pinned services automatically to failover node -->
<!--<bean id="failover-processor" class="org.adroitlogic.ultraesb.clustering.FailoverProcessor">-->
<!-- The matrix defined supporting failover configuration, all nodes in the cluster should have identical
failover node matrix. The entry keys specify the names of the nodes and the value is the corresponding
failover node set for that node as a comma separated list of server names -->
<!--<property name="failoverNodeMatrix">
<map>
<entry key="node1" value="node2,node3"/>
<entry key="node2" value="node1,node3"/>
<entry key="node3" value="node1,node2"/>
</map>
</property>-->
<!-- If configured the delay that the server waits to see whether the failed node re-joins the cluster
before starting to act as it. This defaults to zero meaning that the server will immediately act as
the failed node -->
<!--<property name="secondsBeforeFailover" value="60"/>-->
<!-- Whether the missing nodes at startup, configured this node as there failover node,
be treated as failed by this node, and start to act as them at startup -->
<!--<property name="failoverMissingNodesAtStartup" value="true"/>-->
<!-- If the The "failoverMissingNodesAtStartup" is true, the delay that gives some time for the
missing nodes to appear on the cluster -->
<!--<property name="secondsBeforeFailoverAtStartup" value="300"/>-->
<!--</bean>-->
<!-- Caching/State replication configuration of the UltraESB server -->
<!--bean id="cache-manager" class="org.adroitlogic.ultraesb.cache.ehcache.EhCacheManager">
<!- EH Cache configuration file ->
<property name="ehCacheConfig" value="conf/ehcache.xml"/>
</bean-->
<!-- Environment configuration. You could use any named environment of which the name is given as the constructor
argument. There are pre-defined environments for "dev", "unit_test", "sample", "test", "stage", "prod". Please
read the documentation on runtime environments to understand the default behavior of each environments and to
tune the environments -->
<bean id="environment" class="org.adroitlogic.ultraesb.core.Environment">
<constructor-arg value="dev" type="java.lang.String"/>
</bean>
<!--This bean definition is a required part of any UltraESB configuration and is for internal use only -->
<bean id="ultra-config" class="org.adroitlogic.ultraesb.core.ConfigurationImpl">
<!-- Advanced properties to tune
<property name="defaultWorkManager">
<bean class="org.adroitlogic.ultraesb.core.work.SimpleQueueWorkManager" id="default-wm">
<constructor-arg ref="ultra-config"/>
<property name="primaryCoreThreads" value="20"/>
<property name="primaryMaxThreads" value="300"/>
<property name="primaryKeepaliveSeconds" value="5"/>
<property name="primaryQueueSize" value="0"/>
<property name="secondaryCoreThreads" value="10"/>
<property name="secondaryMaxThreads" value="10"/>
<property name="secondaryKeepaliveSeconds" value="5"/>
<property name="secondaryQueueSize" value="-1"/>
</bean>
</property>-->
<!--<property name="configurationWatchers">
<bean class="java.util.ArrayList">
<constructor-arg>
<list>
<bean name="detailedMXBeanRegistrar" class="org.adroitlogic.ultraesb.core.mgt.DetailedMXBeanRegistrar"/>
</list>
</constructor-arg>
</bean>
</property>-->
<!--<property name="parserCount" value="2048"/>
<property name="serializerCount" value="2048"/>
<property name="xPathCount" value="2048"/>
<property name="transformerCount" value="2048"/>
<property name="secureProcessingEnabled" value="true"/>
<property name="cipherCacheMax" value="2048"/>
<property name="xmlCipherCacheMax" value="2048"/>
<property name="keyGeneratorCacheMax" value="2048"/>-->
<!-- enable custom Spring beans to be re-loaded (Experimental Feature)
<property name="dynamicSpringBeansEnabled" value="true"/>-->
<!--<property name="extensionPackages">
<list>
<value>com.mycompany</value>
</list>
</property>-->
<property name="environment" ref="environment"/>
</bean>
<!--This is a standard Spring import for another configuration fragment file to be imported as part of the root
configuration. None of the elements of a root configuration can be unloaded/reloaded dynamically-->
<import resource="ultra-custom.xml"/>
<!-- Metrics related configurations -->
<import resource="monitoring/ultra-metrics.xml"/>
<!--JMX management and monitoring configuration
Uncomment and configure JMX access as required for remote JMX connections.
This is a standard Spring framework bean - thus refer to the Spring documentation for specifics and examples-->
<!--<bean id="serverConnector" class="org.springframework.jmx.support.ConnectorServerFactoryBean" depends-on="registry">
<property name="objectName" value="connector:name=iiop"/>
<!- Remember to edit bin/ultraesb.sh or conf/wrapper.conf to specify the -Djava.rmi.server.hostname=<your-ip-address> property for JMX ->
<property name="serviceUrl" value="service:jmx:rmi://localhost:9994/jndi/rmi://localhost:1099/ultra"/>
<property name="threaded" value="true"/>
<property name="daemon" value="true"/>
<property name="environment">
<map>
<entry key="jmx.remote.x.password.file" value="conf/management/jmxremote.password"/>
<entry key="jmx.remote.x.access.file" value="conf/management/jmxremote.access"/>
</map>
</property>
</bean>
<bean id="registry" class="org.springframework.remoting.rmi.RmiRegistryFactoryBean">
<property name="port" value="1099"/>
</bean>-->
<!-- Zabbix agent configuration for management and monitoring via JMX -->
<bean id="zabbix-agent" class="org.adroitlogic.ultraesb.core.mgt.zabbix.ZabbixAgent" init-method="init" destroy-method="destroy">
<!-- Uncomment and change for configuring the zabbix agent request handler listen port -->
<!--<property name="port" value="11819"/>-->
<!--<property name="bindAddress" value="127.0.0.1"/>-->
</bean>
</beans>
I have done following two changes for the original file.
- Enabled clustering by changing false to true in line - 330 above.
- Specified quorum url of the cluster as 'localhost:2181,localhost:2182,localhost:2183'
You can use the above configuration for UE1. Make these two changes to the other ultra-root.xml as well. In addition to these do the following changes to UE2 and UE3 respectively.
- In line 72, change the port number to 8281 and 8282
- In line 127, change the port number to 8444 and 8445
- In line 249, change node name to node2 and node3
- Uncomment line 394 and change 11819 to 11820 and 11821 or remove lines 392 to 396.
That is all for the configuration and now you can start the ESB instances. For this change to a bin directory (eg: cd ~/zoo/UE1/ultraesb-2.0.0-SNAPSHOT/bin). Then execute the following command.
$ sh ultraesb.sh
If you did things correctly you will get a console output similar to following
Starting AdroitLogic UltraESB ...
Using JAVA_HOME : /opt/java
Using ULTRA_HOME: /home/samila/zoo/UE1/ultraesb-2.0.0-SNAPSHOT
Reading configuration from : /home/samila/zoo/UE1/ultraesb-2.0.0-SNAPSHOT/conf/ | Current directory is : /home/samila/zoo/UE1/ultraesb-2.0.0-SNAPSHOT/.
2013-07-29 22:57:54,684 [-] [main] INFO ServerManager ServerManager initializing..
2013-07-29 22:57:55,175 [-] [main] INFO UltraServer UltraESB JAR file version/s : [2.0.0-SNAPSHOT-eca88e22360c]
2013-07-29 22:57:56,567 [-] [main] INFO PooledMessageFileCache Directory : /home/samila/zoo/UE1/ultraesb-2.0.0-SNAPSHOT/tmp/samila_localhost locked for file cache
2013-07-29 22:57:56,568 [-] [main] WARN PooledMessageFileCache System ulimit of : 1024 file descriptors does not seem adequate for production use
2013-07-29 22:57:57,072 [-] [ZabbixAgent] INFO ZabbixAgent Agent started on port : 11819
2013-07-29 22:57:57,120 [-] [main] INFO ConfigurationImpl Starting AdroitLogic (http://www.adroitlogic.org) - UltraESB/2.0.0-SNAPSHOT (GA) - node1
2013-07-29 22:57:57,120 [-] [main] INFO ConfigurationImpl ****************************************************************************
2013-07-29 22:57:57,121 [-] [main] INFO ConfigurationImpl *** Copyright (C) 2010-13 AdroitLogic Private Ltd. All Rights Reserved ***
2013-07-29 22:57:57,121 [-] [main] INFO ConfigurationImpl *** ***
2013-07-29 22:57:57,121 [-] [main] INFO ConfigurationImpl *** Licensed under the GNU Affero General Public License ***
2013-07-29 22:57:57,121 [-] [main] INFO ConfigurationImpl *** See LICENSE-AGPL.TXT or http://www.gnu.org/licenses/agpl-3.0.html ***
2013-07-29 22:57:57,121 [-] [main] INFO ConfigurationImpl ****************************************************************************
2013-07-29 22:57:58,535 [-] [main] ERROR NotificationAction Notification manager has not been configured, alert notification will not be sent
2013-07-29 22:57:58,540 [-] [main] ERROR NotificationAction Notification manager has not been configured, alert notification will not be sent
2013-07-29 22:57:58,576 [-] [main] INFO ConfigurationImpl Pre-initialization of the engine..
2013-07-29 22:57:59,005 [-] [main] INFO XMLFeatures Initializing the XML feature
2013-07-29 22:57:59,166 [-] [main] INFO TransformationUtils Initializing Transformation feature
2013-07-29 22:57:59,213 [-] [main] INFO MediationImpl Initializing the Mediation feature
2013-07-29 22:57:59,530 [-] [main] INFO JSONUtils Initializing the JSON feature
2013-07-29 22:57:59,550 [-] [main] INFO FastInfosetUtils Initializing the Fast-Infoset feature
2013-07-29 22:57:59,566 [-] [main] INFO CryptoSupport Initializing the Security and Crypto features
2013-07-29 22:57:59,635 [-] [main] INFO ServerManager Starting server ...
2013-07-29 22:57:59,774 [-] [main] INFO ServerManager Linux - amd64 [3.2.0-37-generic] / Processors : 4
2013-07-29 22:57:59,775 [-] [main] INFO ServerManager Total physical memory : 3,956,160K (149,744K free) Heap max : 1,004,928K
2013-07-29 22:57:59,775 [-] [main] INFO ServerManager Java HotSpot(TM) 64-Bit Server VM, Sun Microsystems Inc. [19.0-b09]
2013-07-29 22:57:59,812 [-] [main] INFO ServerManager Starting cluster manager..
2013-07-29 22:57:59,812 [-] [main] INFO ClusterManager Asynchronous starting signal received by the cluster manager
2013-07-29 22:57:59,892 [-] [main] INFO ClusterManager Waiting maximum 29 seconds for the cluster manager to start
2013-07-29 22:58:00,008 [-] [main-EventThread] INFO ClusterManager Zookeeper connection established State:CONNECTED Timeout:30000 sessionid:0x2402b312c530002 local:/127.0.0.1:48695 remoteserver:localhost/127.0.0.1:2182 lastZxid:0 xid:1 sent:1 recv:1 queuedpkts:0 pendingresp:0 queuedevents:0
2013-07-29 22:58:00,892 [-] [main] INFO ClusterManager Waiting maximum 28 seconds for the cluster manager to start
2013-07-29 22:58:00,907 [-] [main-EventThread] INFO ClusterManager Clustering command processors initialized, the command version set to : 0
2013-07-29 22:58:00,929 [-] [main-EventThread] INFO ClusterManager Cluster manager has been successfully started
2013-07-29 22:58:00,931 [-] [main] INFO HttpsNIOSender Https transport sender : https-sender using default identity keystore
2013-07-29 22:58:00,932 [-] [main] INFO HttpsNIOSender Https transport sender : https-sender using the default trust keystore
2013-07-29 22:58:01,245 [-] [HttpsNIOSender-https-sender] INFO HttpsNIOSender Starting NIO Sender : https-sender ...
2013-07-29 22:58:01,248 [-] [HttpNIOSender-http-sender] INFO HttpNIOSender Starting NIO Sender : http-sender ...
2013-07-29 22:58:01,301 [-] [main] INFO fileCache Memory mapping is enabled for : 8192 bytes of each file
2013-07-29 22:58:01,302 [-] [main] INFO fileCache Initialized cache of : 200 files at : /home/samila/zoo/UE1/ultraesb-2.0.0-SNAPSHOT/tmp/samila_localhost
2013-07-29 22:58:01,330 [-] [main] INFO SimpleQueueWorkManager Started Work Manager : default
2013-07-29 22:58:01,330 [-] [main] INFO ServerManager Initializing transport listeners
2013-07-29 22:58:01,336 [-] [main] INFO ServerManager Starting Proxy Services, Endpoints and Sequences
2013-07-29 22:58:01,376 [-] [main] INFO Address Started Address : address of endpoint : echo-service
2013-07-29 22:58:01,376 [-] [main] INFO echo-service Started endpoint : echo-service
2013-07-29 22:58:01,377 [-] [main] INFO Address Started Address : address of endpoint : mediation.response
2013-07-29 22:58:01,377 [-] [main] INFO Endpoint Started endpoint : mediation.response
2013-07-29 22:58:02,555 [-] [main] INFO error-handler Sequence : error-handler started
2013-07-29 22:58:02,627 [-] [main] INFO health-check-inSequence Sequence : health-check-inSequence started
2013-07-29 22:58:02,628 [-] [main] INFO health-check Proxy service : health-check started
2013-07-29 22:58:02,628 [-] [main] INFO ServerManager UltraESB root deployment unit started successfully
2013-07-29 22:58:02,656 [-] [main] INFO DeploymentUnitBuilder Using the hot-swap class loading of libraries for deployment unit default
2013-07-29 22:58:02,807 [-] [main] INFO DeploymentUnitBuilder Successfully created the deployment unit : default
2013-07-29 22:58:02,886 [-] [main] INFO echo-back-inSequence Sequence : echo-back-inSequence started
2013-07-29 22:58:02,886 [-] [main] INFO echo-back Proxy service : echo-back started
2013-07-29 22:58:02,965 [-] [main] INFO echo-proxy-outSequence Sequence : echo-proxy-outSequence started
2013-07-29 22:58:02,967 [-] [main] INFO Address Started Address : address of endpoint : echo-proxy-inDestination
2013-07-29 22:58:02,967 [-] [main] INFO echo-proxy-inDestination Started endpoint : echo-proxy-inDestination
2013-07-29 22:58:02,969 [-] [main] INFO Address Started Address : address of endpoint : echo-proxy-outDestination
2013-07-29 22:58:02,969 [-] [main] INFO echo-proxy-outDestination Started endpoint : echo-proxy-outDestination
2013-07-29 22:58:02,969 [-] [main] INFO echo-proxy Proxy service : echo-proxy started
2013-07-29 22:58:02,969 [-] [main] INFO ConfigurationImpl Successfully added the deployment unit : default
2013-07-29 22:58:02,969 [-] [main] INFO ServerManager Starting transport listeners
2013-07-29 22:58:03,016 [-] [HttpNIOListener-http-8280] INFO HttpNIOListener Starting NIO Listener : http-8280 on port : 8280 ...
2013-07-29 22:58:03,061 [-] [main] INFO HttpsNIOListener Identity keystore loaded from : conf/keys/identity.jks
2013-07-29 22:58:03,101 [-] [main] INFO HttpsNIOListener Trust keystore loaded from : conf/keys/trust.jks
2013-07-29 22:58:03,102 [-] [main] INFO ConfigurationImpl UltraESB/2.0.0-SNAPSHOT (GA) - node1 started with root configuration..
2013-07-29 22:58:03,118 [-] [HttpsNIOListener-https-8443] INFO HttpsNIOListener Starting NIO Listener : https-8443 on port : 8443 ...
You can test the correctness by the following commands.
$ cd ~/zoo/UE1/ultraesb-2.0.0-SNAPSHOT/bin
$ sh zkCli.sh
This will connect to localhost:2181. Now run the following command.
ls /ultraesb/default/nodes/active
This should produce the following output
[node2, node3, node1]
These are the basic steps that you should follow in order to deploy the UltraESB in a cluster.
Thanks.