1 Reply Latest reply on Nov 22, 2009 10:57 AM by babazs

    INSTRUCTIONS FOR JBOSS 2 NODE HORIZONTAL CLUSTER INSTALL AND

    asookazian

      #########################################################################################

      SUBJECT: INSTRUCTIONS FOR JBOSS 2 NODE HORIZONTAL CLUSTER INSTALL AND CONFIG ON WINDOWS OS 32-bit
      AUTHOR: A SOOKAZIAN
      DATE: 9/19/2008
      VERSION: 1.0.0

      INTRO: This document describes how to install and setup a 2 node horizontal cluster JBoss EAP4.3.0 servers.
      Use the production configuration (see jboss-eap\jboss-as\server\production directory).
      The production config has some customizations/optimizations done for production environment and is cluster-enabled.
      Make sure that all software versions and OS are 32-bit.

      NOTE: There is a pre-configured and tested node folder here: T:\IT\JBoss\clustering setup\clustered\node.
      You may simply skip some of the configuration steps below by copying the node folder into jboss-eap\jboss-as\server on each server.
      It's mandatory to specify a unique Node-ID for each JBoss Messaging (JBM) instance in your cluster. You have to set this on your messaging-service.xml file.
      See JBoss case # 217243 for details.

      There is no EJB3 entity class caching configuration as per http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/html-single/Server_Configuration_Guide/index.html#Clustered_Entity_EJBs-Entity_Bean_in_EJB_3.0
      using Hibernate 2nd level cache

      RESOURCES: JBoss Enterprise Application Platform 4.3.0.GA installation guide
      http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/html-single/Installation_Guide/index.html

      Release Notes CP02
      http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/pdf/readme/readme.pdf

      Release Notes CP02 FP01
      http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02_fp01/html-single/readme/index.html

      PHX JBoss installation Ant script
      http://subversion.cox.com/Repositories/jbossconfigurationfiles/trunk/CoxAppEnvironment

      Use Alternative Databases with JBoss AS
      http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/html-single/Server_Configuration_Guide/index.html#alternative_DBs

      Using JBoss Native to run JBoss AS as a Windows Service
      http://wiki.jboss.org/wiki/JBossNativeWindows

      Post Installation Security Configuration
      http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/html/Installation_Guide/Post-Installation.html

      I want to have two clusters on the same network and I don't want them to see each other
      http://wiki.jboss.org/wiki/TwoClustersSameNetwork

      Clustering with UDP or TCP - What is best?
      http://www.jboss.com/index.html?module=bb&op=viewtopic&t=135242


      JIRA ISSUES: run.conf ignored by run.bat under Windows1:08 PM 9/19/2008
      https://jira.jboss.org/jira/browse/JBAS-5363

      WARNINGS: "Nodes improperly communicating with each other is one of the most common issues users have with JBoss AS clustering."
      source: http://www.jboss.org/file-access/default/members/jbossas/freezone/docs/Clustering_Guide/beta422/html/ch07s07s08.html

      #########################################################################################

      1. install Windows 2003 5.2, x86 server (or equivalent 32-bit version)

      2. install JDK
      - run exe inside T:\IT\JBoss\install\shared setup\jrockit-R27.5.0-jdk1.5.0_14-windows-ia32.zip to install JRockit JDK

      3. extract JBoss EAP 4.3.0.GA_CP02
      - unzip T:\IT\JBoss\install\shared setup\jboss-eap-4.3.0.GA_CP02.zip to C:\

      4. extract JBoss EAP 4.3.0.GA_CP02_FP01
      - unzip T:\IT\JBoss\install\shared setup\jboss-eap-4.3.0.GA_CP02.zip to C:\jboss-eap-4.3 (overwrite all existing files)
      - check these notes as required: http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02_fp01/html-single/readme/index.html#Installation_Notes

      5. copy C:\jboss-eap-4.3\jboss-as\server\production folder to C:\temp, rename to node, move back to C:\jboss-eap-4.3\jboss-as\server\

      5. copy MSSQL JDBC driver
      - copy T:\IT\JBoss\install\shared setup\sqljdbc.jar to jboss-eap-4.3\jboss-as\node\lib

      6. configure environment variables
      - Create an environment variable that points to the JDK installation directory (C:\Program Files\Java\jrockit-R27.5.0-jdk1.5.0_14) and call it JAVA_HOME.
      - Create an environment variable that points to the JBoss installation directory (C:\jboss-eap-4.3\jboss-as) and call it JBOSS_HOME.

      7. verify JDK installation
      - verify installation by starting cmd line and excecuting java -version

      8. adjust default memory settings
      - in jboss-eap-4.3\jboss-as\bin\run.bat, delete the following lines:

      rem JVM memory allocation pool parameters. Modify as appropriate.
      set JAVA_OPTS=%JAVA_OPTS% -Xms128m -Xmx512m

      rem With Sun JVMs reduce the RMI GCs to once per hour
      set JAVA_OPTS=%JAVA_OPTS% -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000

      9. smoke test startup of JBoss
      - go to cmd line and exec: run.bat -b [servername] -c node in the bin folder of JBoss extraction
      - check jboss-eap-4.3\jboss-as\server\node\log\server.log to see if there are any exceptions and that JBoss started successfully
      - you should see a line at the end of the C:\jboss-eap-4.3\jboss-as\server\node\log\server.log similar to this:
      15:50:42,289 INFO [Server] JBoss (MX MicroKernel) [4.3.0.GA_CP01 (build: SVNTag=JBPAPP_4_3_0_GA_CP01 date=200804211657)] Started in 23s:985ms
      - Now open http://[servername]:8080 in your web browser. You should see the jmx console and web console links, among some others.

      10. modify log4j settings
      - to minimize disk I/O, set log4j threshold to WARN level for server.log in jboss-as\server\node\conf\jboss-log4j.xml as follows:

      - see T:\IT\JBoss\install\shared setup\jboss-log4j.xml as an example

      11. install JBoss Native windows service
      - extract all contents (with overwrite) of T:\IT\JBoss\install\shared setup\jboss-native-2.0.4-windows-all-ssl.zip into C:\jboss-eap-4.3\jboss-as
      - configure C:\jboss-eap-4.3\jboss-as\bin\service.bat such that service binds with server name of the current machine (e.g., replace -b corg0dv202 with -b [servername])
      - NOTE: use IP address instead of servername if the server may be multi-homed in the future (multiple ethernet cards for multiple IP addresses to support vertical scaling)
      - start cmd line from C:\jboss-eap-4.3\jboss-as\bin and exec:
      service.bat install
      - go to services window and refresh to check to see if new JBoss AS service is there now
      - follow instructions here: http://wiki.jboss.org/wiki/JBossNativeWindows
      - NOTE: JVM tuning is configured in the service.bat file as follows: -Xms:1g -Xmx:1g -XgcPrio:pausetime

      12. smoke test startup of service by restarting windows and navigating to http://[servername]:8080

      13. perform Hypersonic "surgery" to replace all HSQLDB occurrences with MSSQL
      - NOTE: this entire step can be skipped if you copy the node folder from T:\IT\JBoss\install\clustered into jboss-eap-4.3\jboss-as\server into C:\jboss-eap-4.3\jboss-as\server
      - follow instructions here: T:\IT\JBoss\install\shared setup\Replace hsqldb.doc
      - coordinate with DBA to create jboss db, login/user/pswd, etc. (there is currently a jboss db on DV02 that may be used for testing purposes)
      - coordinate with DBA to create boBookingTest db (for testing purposes only, it's required for the Seam booking application; there is currently a boBookingTest db on DV02 that may be used for testing purposes)

      step-by-step:
      - delete jboss-as\server\node\deploy\hsqldb-ds.xml
      - delete jboss-as\server\node\farm\cluster-examples-service.xml
      - delete jboss-as\server\node\deploy\jboss-messaging.sar\clustered-hsqldb-persistence-service.xml
      - copy T:\IT\JBoss\install\shared setup\mssql-ds.xml to node\deploy folder
      - make sure the connection-url points to the correct sql server, instance (if any) and db
      - copy T:\IT\JBoss\install\shared setup\mssql-persistence-service.xml to jboss-as/server/node/deploy/jboss-messaging.sar/
      - change the jboss-as/server/node/conf/standardjbosscmp-jdbc.xml file so that the fk-constraint property is true
      - configure the database dialect in the jboss-as/server/node/deploy/ejb3.deployer/META-INF/persistence.properties file.
      un-comment the hibernate.dialect property and change its value to org.hibernate.dialect.SQLServerDialect.
      - Change Other JBoss AS Services to Use the External Database (the more flexible way)
      - replace all instances of 'DefaultDS' with 'MSSQLDS' in the following files:
      jboss-as/server/node/conf/login-config.xml
      jboss-as/server/node/conf/standardjbosscmp-jdbc.xml
      jboss-as/server/node/deploy/ejb-deployer.xml
      jboss-as/server/node/deploy/schedule-manager-service.xml
      jboss-as/server/node/deploy/uuid-key-generator.sar/META-INF/jboss-service.xml
      jboss-as/server/node/deploy/snmp-adaptor.sar/attributes.xml
      jboss-as/server/node/deploy/juddi-service.sar/META-INF/jboss-service.xml
      jboss-as/server/node/deploy/juddi-service.sar/juddi.war/WEB-INF/jboss-web.xml

      14. Configure Tomcat (you must complete this manual step even if you use the template node folder here: T:\IT\JBoss\install\clustered\node)
      To complete the configuration, we also need to name each node accordingly to the names specified in workers.properties.
      Edit JBOSS_HOME/server/node/deploy/jbossweb-deployer.sar/server.xml
      Locate the element and add an attribute jvmRoute (where nodeX=node1 for one instance and node2 for the other, they must not be the same):



      The jvmRoute attribute must match the name specified in workers.properties for each node.

      Finally, we need to tell Tomcat to add the jvmRoute value to its session cookies so that mod_jk can route incoming requests.
      Edit JBOSS_HOME/server/node/deploy/jbossweb-deployer.sar/META-INF/jboss-service.xml
      Locate the element with a name of UseJK, and set its value to "true":

      true

      15. perform Post Installation Security Configuration
      - follow these instructions: http://www.redhat.com/docs/en-US/JBoss_Enterprise_Application_Platform/4.3.0.cp02/html/Installation_Guide/Post-Installation.html
      - Set up Accounts for jmx-console and the invokers by modifying: $JBOSS_HOME/server/$CONFIG/conf/props/jmx-console-users.properties
      - Set up Accounts for web-console users by modifying: $JBOSS_HOME/server/$CONFIG/deploy/management/console-mgr.sar/web-console.war/WEB-INF/classes/web-console-users.properties
      - Set SuckerPassword for JBoss Messaging: $JBOSS_HOME/server/$CONFIG/deploy/jboss-messaging.sar/messaging-service.xml

      16. DNS name creation
      - create DNS name (as required) for server
      - navigate to http://[myDNS]:8080/seam-booking and execute a test case

      17. configure IIS and mod_jk to forward requests to JBoss server
      - copy T:\IT\JBoss\install\clustered\isapi_redirect-1.2.26 folder to C:\
      - NOTE: make sure you add configuration(s) for any deployed app(s) to isapi_redirect-1.2.26\conf\uriworkermap.properties file, otherwise, mod_jk will not forward
      requests to JBoss for that servlet context
      - follow these directions: http://wiki.jboss.org/wiki/UsingMod_jk1.2WithJBossAndIIS
      - you will need to install a ISAPI filter (name it ISAPI_REDIRECT) for the Default Web Site pointing to C:\isapi_redirect-1.2.26\bin\isapi_redirect.dll.
      After you do this, make sure there is a green up arrow in the status column for this filter on the ISAPI filters tab.
      isapi_redirect.dll is the mod_jk redirection filter which will forward all request for servlet contexts specified in C:\isapi_redirect-1.2.26\conf\uriworkermap.properties
      isapi_redirect.dll is handling the Tomcat/JBoss load balancing and failover via the workers.properties file.
      There is a log here: C:\isapi_redirect-1.2.26\log\isapi_redirect.log for trouble-shooting redirection failures, etc.
      - restart World Wide Web Publishing service in services window.

      18. test IIS/mod_jk/JBoss connector configuration
      - make sure WWW publishing service and JBoss service are both running
      - navigate to http://[nameOfIISserver]/web-console
      - if you're prompted for credentials, enter 'admin' for username and password

      19. deploy and test seam booking app
      - copy T:\IT\JBoss\install\shared setup\jboss-seam-booking.ear and T:\IT\JBoss\install\shared setup\jboss-seam-booking-ds.xml into jboss-eap-4.3\jboss-as\replace\deploy
      - start Jboss via service if not already started (otherwise, JBoss will hot-deploy the app)
      - navigate to http://[servername]:8080/seam-booking and execute a test case
      - this will test the Hypersonic "surgery" and CRUD transactions to the db for this app

      20. repeat all above steps for nth server (nth addition to horizontal cluster); warning: see troubleshooting section below, item #3.

      /************************************************************************************************************************************************************/

      Troubleshooting

      1) JGroups Troubleshooting: Nodes do not form a cluster
      http://www.jboss.org/file-access/default/members/jbossas/freezone/docs/Clustering_Guide/beta422/html/ch07s07s11.html

      2) Isolating JGroups Channels: Nodes improperly communicating with each other is one of the most common issues users have with JBoss AS clustering.
      http://www.jboss.org/file-access/default/members/jbossas/freezone/docs/Clustering_Guide/beta422/html/ch07s07s08.html

      3) java.lang.IllegalArgumentException: Cannot start post office since there is already a post office in the cluster with the same node id (0). Are you sure you have given each node a unique node id during installation?
      it's mandatory to specify a unique Node-ID for each JBoss Messaging (JBM) instance in your cluster. You have to set this on your messaging-service.xml file as follows :

      ...
      ${jboss.messaging.ServerPeerID:2}
      ...

      The messaging-service.xml is located at :
      $JBOSS_HOME/server/$YOUR_SERVER_CONFIG/deploy/jboss-messaging.sar/

      for more info, refer to JBoss sppt case ID 217243

      4) Failed to execute: CREATE TABLE JBM_POSTOFFICE (POSTOFFICE_NAME VARCHAR(255), NODE_ID INTEGER,QUEUE_NAME VARCHAR(255), CONDITION VARCHAR(1023), SELECTOR VARCHAR(1023), CHANNEL_ID BIGINT, CLUSTERED CHAR(1), ALL_NODES CHAR(1), PRIMARY KEY(POSTOFFICE_NAME, NODE_ID, QUEUE_NAME))
      java.sql.SQLException: Table already exists: JBM_POSTOFFICE in statement [CREATE TABLE JBM_POSTOFFICE]
      at org.hsqldb.jdbc.Util.sqlException(Unknown Source)

      This is not an ERROR but a DEBUG message which you can safely ignore.[2]

      [2] http://wiki.jboss.org/wiki/WhyDoIGetquotCouldNotCreateTableWithSQLquotDEBUGMessagesAtStartup

      5) When executing the hotel booking use case, if you see no records (i.e., hotels) on the main.seam page, check the boBookingTest db on DV02 to see if any records
      are actually in the Hotel table. If no records exist, then run this script from SQL management studio: T:\IT\JBoss\install\shared setup\booking seam app sql server setup script.sql

      6) 2008-09-23 12:51:59,939 DEBUG [org.jboss.remoting.transport.socket.SocketServerInvoker] SocketServerInvoker[10.46.175.13:4457] caught exception in run()
      java.net.SocketException: socket closed
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
      at java.net.ServerSocket.implAccept(ServerSocket.java:450)
      at java.net.ServerSocket.accept(ServerSocket.java:421)
      at org.jboss.remoting.transport.socket.SocketServerInvoker.run(SocketServerInvoker.java:520)
      at java.lang.Thread.run(Thread.java:595)

      JBoss sppt: this can safely be ignored. As indicated by its "level," this is debugging information and is related to JBoss Messaging (which uses port 4457).

      7) 10:31:10,643 INFO [CacheListener] Possible concurrency problem: Replicated version id 5 matches in-memory version for session ml5CzYIuE0X62k2vNReP4g**

      If you see this, check your JBOSS_HOME/server/node/deploy/jbossweb-deploy.sar/server.xml jvmRoute settings. (http://www.techienuggets.com/Comments?tx=29768)