1 2 3 Previous Next 35 Replies Latest reply: Aug 25, 2012 10:06 AM by rang s Go to original post RSS
  • 15. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    rang s Newbie

    ok thank you for the prompt reply Tomaz !

  • 16. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    Johan Vermeij Newbie

    Maybe this is not the right thread but I try and try without result.

     

    Using 7.1.0:

    I can access the cluster using Apache httpd when I comment out ServerAdvertise (when I uncomment ServerAdvertise I don't find the cluster)

    <IfModule manager_module>

      Listen 192.168.2.2:6666

      ManagerBalancerName other-server-group

      <VirtualHost 192.168.2.2:6666>

        <Location />

         Order deny,allow

         Deny from all

         Allow from all

        </Location>

     

        ErrorLog "logs/chatter_log"

        LogLevel debug

        KeepAliveTimeout 100

        MaxKeepAliveRequests 0

    #   ServerAdvertise on http://127.0.0.1:6666

        AdvertiseFrequency 30

        #AdvertiseSecurityKey secret

        #AdvertiseGroup 224.0.1.105:23364

        EnableMCPMReceive

     

      </VirtualHost>

    </IfModule>

    </IfModule>

    NameVirtualHost 192.168.2.3:8880

    <VirtualHost 192.168.2.3:8880>

        ServerAdmin johanvermeij@gmail.com

        ServerName 192.168.2.3

        ErrorLog "logs/cluster_log"

        LogLevel debug

     

        ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On

        ProxyPassReverse / balancer://other-server-group

        ProxyPreserveHost On

     

        <Location />

            Order deny,allow

            Allow from All

        </Location>

     

        <Location /mod_cluster_manager>

            SetHandler mod_cluster-manager

            Order deny,allow

            Deny from all

            Allow from all

        </Location>

    </VirtualHost>

     

    That works

     

    I now would like to use 7.1.1 clustering.

     

    When I use the same httpd.conf as above under 7.1.1 all I ever get is (I have tried everything I could think of for 2 days straight):

    [Tue Aug 21 21:24:17 2012] [debug] mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED

    [Tue Aug 21 21:24:17 2012] [error] proxy: CLUSTER: (balancer://other-server-group). All workers are in error state

  • 17. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    Jean-Frederic Clere Master

    Does /mod_cluster_manager show the nodes?

  • 18. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    rang s Newbie

    Let me know which version are you trying this ? Did you try on 7.1.1/7.1.2. ?

  • 19. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    Johan Vermeij Newbie

    I am now at the point where I can run master and slave instances but I cannot access the cluster (do transactions from a terminal) when I start up a slave.

    As soon as I stop the slave I can do transactions again.

     

    This is the output from mod_cluster_manager with the master running

    mod_cluster/1.2.0.Final

    Auto Refresh show DUMP output show INFO output

    Node undefined (ajp://192.168.2.3:8259):

    Enable Contexts Disable Contexts

    Balancer: other-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: OK,Elected: 1,Read: 9065,Transferred: 149,Connected: 0,Load: 100

    Virtual Host 1:

    Contexts:

    /internetbanking, Status: ENABLED Request: 0 Disable /beans, Status: ENABLED Request: 0 Disable /wssimple, Status: ENABLED Request: 0 Disable 

    Aliases:

    localhost default-host example.com 

    When I bring up a slave I get this intermittently (httpd is very busy stopping and starting workers)

    mod_cluster/1.2.0.Final

    Auto Refresh show DUMP output show INFO output

    Node REMOVED (ajp://192.168.2.3:8259):

    Enable Contexts Disable Contexts

    Balancer: other-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: NOTOK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 100

     

    modcluster settings in domain.xml are as follows (I have added the proxy-list and advertise-security-key this morning).

     

    <subsystem xmlns="urn:jboss:domain:modcluster:1.0">

                    <mod-cluster-config advertise-socket="modcluster" proxy-list="192.168.2.3:6666" advertise-security-key="secret">

                        <dynamic-load-provider>

                            <load-metric type="busyness"/>

                        </dynamic-load-provider>

                    </mod-cluster-config>

                </subsystem>

     

     

    As soon as I start a slave I get this error on both master and slave:

    [Server:server-three-Linux] 12:50:48,703 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy 192.168.2.3/192.168.2.3:6666, configuration will be reset

     

    I post my current httpd.conf settings as well (I copied again from 1.7.0 and this time things started working).

     

      Listen 192.168.2.3:6666

      ManagerBalancerName other-server-group

      <VirtualHost 192.168.2.3:6666>

        <Location />

         Order deny,allow

         Deny from all

         Allow from all

        </Location>

     

        ServerName vantagebanking.com

        ErrorLog "logs/chatter_log"

        LogLevel debug

        KeepAliveTimeout 100

        MaxKeepAliveRequests 0

        ServerAdvertise on http://127.0.0.1:6666

        AdvertiseFrequency 30

        AdvertiseSecurityKey secret

        AdvertiseGroup 224.0.1.105:23364

        EnableMCPMReceive

     

      </VirtualHost>

    </IfModule>

     

    NameVirtualHost 192.168.2.3:8000

    <VirtualHost 192.168.2.3:8000>

        ServerAdmin johanvermeij@gmail.com

        ServerName vantagebanking.com

        ErrorLog "logs/cluster_log"

        LogLevel debug

     

        ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On

        ProxyPassReverse / balancer://other-server-group

        ProxyPreserveHost On

     

        <Location />

            Order deny,allow

            Allow from All

        </Location>

     

        <Location /mod_cluster_manager>

            SetHandler mod_cluster-manager

            Order deny,allow

            Deny from all

            Allow from all

        </Location>

     

    /VirtualHost>

     

    Sorry, all a bit messy, for me there is a lot of trial and error in this.

     

    I am amazed by JBoss clustering and I am determined to make it work

  • 20. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    Johan Vermeij Newbie

    I use 7.1.1

     

    Is 7.1.2 out already ?

  • 21. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    rang s Newbie

    Did you make the node names unique (specifying unique names for master and slave through host.xml) . If everything is fine, the mod_cluster_manager should show both the nodes in the paeg.

  • 22. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    rang s Newbie

    7.1.2 is a tag for EAP release, it has to be built by downloading the JBoss source code. However for mod_cluster to work 7.1.1 should be fine.

  • 23. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    Johan Vermeij Newbie

    That is what I am looking for, I read that an instance-id can be added, also through the web console.

    But I can't find where.

    I suppose the instance-id can be set in this section of host.xml

    Can you give me a hint ?

    <servers>
        <server name="server-one" group="main-server-group">
            <!-- Remote JPDA debugging for a specific server
            <jvm name="default">
              <jvm-options>
                <option value="-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"/>
              </jvm-options>
           </jvm>
           -->
        </server>
        <server name="server-two" group="main-server-group" auto-start="false">
            <!-- server-two avoids port conflicts by incrementing the ports in
                 the default socket-group declared in the server-group -->
            <socket-bindings port-offset="150"/>
        </server>
        <server name="server-three-Linux" group="other-server-group" auto-start="true">
            <!-- server-three avoids port conflicts by incrementing the ports in
                 the default socket-group declared in the server-group -->
            <socket-bindings port-offset="250"/>
        </server>
    </servers>
  • 24. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    Jean-Frederic Clere Master

    the undefined is due to a bug... JBPAPP-8451?

  • 25. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    rang s Newbie

    As mentioned in Jean-Frederic's post above try adding the workaround as you use 7.1.1

     

    Try adding this in domain.xml 

     

       <subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">

     

    Also, hope you have set the names for master and slave as per https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto

     

    something like , <host name="slave" xmlns="urn:jboss:domain:1.2">

     

    I hope this should solve your issue. Also go through the comments in  https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto . It has solution and workarounds for most of the issues for Cluster setup.

  • 26. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    Johan Vermeij Newbie

    Thank you very much, it all works fine

     

    I am already running a jmeter test against a cluster with a HP Proliant, an iMac and a Linux PC

     

    It is fast

  • 27. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    Johan Vermeij Newbie

    When I am running a stress test against the above cluster under 7.1.1 (domain mode) I get this error very often.

     

    [Server:server-three-slave-HP] 18:46:25,182 ERROR [com.vantage.capital.service.xml.XMLTranServiceBean] (ajp--192.168.2.12-8259-22) DDF0001,CashTransaction:EXCP:org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate

     

    In fact, I don't need jmeter for this.

    When I open 2 terminals and I do transactions quickly by hitting enter fast I get the same error.

     

    Looking at the JBoss logs, netstat, Apache logs everything looks fine

  • 28. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    rang s Newbie

    I am not too sure what kind of stress testing are you performing. If you can elaborate a bit, it would be better to understand the problem.

     

    From your logs, I see that you are using Hibernate and perhaps the Infinispan cache for Hibernate. Are those caches up when you are performing a test ?  What kind of caches have you configured (any remote caches?)?

  • 29. Re: mod_cluster 1.2 + JBossAS 7.1 problem
    Johan Vermeij Newbie

    I am testing a financial application.

    I prepare jmeter jmx files using a sample soap message captured from a terminal.

    I use the sample to propagate many other also simulating different users, bank branches, accounts etc.

     

    I point the jmeter injector at Apache http on port 8000 (see httpd.conf below) using mod_cluster (latest)

    I use one or more laptops as jmeter injectors

     

    httpd then load balances this to a cluster which consist of a HP Proliant mini server (Centos 6 64 bit, 192.168.2.12), a PC (Centos 6 64 bit, 192.168.2.3) and an iMac (MacOs, 192.168.2.4)

    I use an Oracle 11g database on the HP, httpd runs on the PC.

     

    The stress test runs very fast when only the master is up (PC is the master)

     

    As soon as I bring up a slave (and also when I bring up master and 2 slaves from the start) I get this error:

    [Server:server-three-slave-HP] 19:28:03,650 ERROR [com.vantage.capital.service.xml.XMLTranServiceBean] (ajp--192.168.2.12-8259-146) DDF0002,CashWithdrawal:EXCP:org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate

     

    The above timeout happens on the PC and the HP.

    It slows the test down to about 1TPS (by comparison I can do almost 100TPS against the HP in standalone mode)

     

    The iMac has a different error:

    [Server:server-three-slave-iMac] 19:30:22,593 WARNING [org.jgroups.protocols.UDP] (pool-23-thread-1) null: no physical address for 3cf4cecf-6754-af54-1c50-f2ba733aac93, dropping message

    [Server:server-three-slave-iMac] 19:30:25,595 WARNING [org.jgroups.protocols.pbcast.GMS] (pool-23-thread-1) JOIN(slave1:server-three-slave-iMac/hibernate) sent to 3cf4cecf-6754-af54-1c50-f2ba733aac93 timed out (after 3000 ms), retrying

     

    I have gone through the startup logs and everything looks fine there, there are no errors or warnings.

     

    I don't expect you to fix this for me Rang, maybe you can give me a hint.

    We have been working on this for almost a week now and we are almost there.

    Problem is also that Googling these kinds of conditions is very "circular" and a lot of the discussions pertain to older JBoss releases.

     

    I say it again, what you guys have done is incredible, no commercial software can come near you.

     

    From httpd.conf

    <IfModule manager_module>

      Listen 192.168.2.3:6666

      AllowDisplay On

      ManagerBalancerName other-server-group

      <VirtualHost 192.168.2.3:6666>

        <Location />

         Order deny,allow

         Deny from all

         Allow from all

        </Location>

     

        ServerName vantagebanking.com

        ErrorLog "logs/chatter_log"

        LogLevel debug

        KeepAliveTimeout 100

        MaxKeepAliveRequests 0

    #   ServerAdvertise on http://192.168.2.3:6666

        AdvertiseFrequency 30

    #   AdvertiseSecurityKey secret

    #   AdvertiseGroup 224.0.1.105:23364

        EnableMCPMReceive

     

      </VirtualHost>

    </IfModule>

     

    NameVirtualHost 192.168.2.3:8000

    <VirtualHost 192.168.2.3:8000>

        ServerAdmin johanvermeij@gmail.com

        ServerName vantagebanking.com

        ErrorLog "logs/cluster_log"

        LogLevel debug

        AllowDisplay On

        ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On

        ProxyPassReverse / balancer://other-server-group

        ProxyPreserveHost On

     

        <Location />

            Order deny,allow

            Allow from All

        </Location>

     

        <Location /mod_cluster_manager>

            SetHandler mod_cluster-manager

            Order deny,allow

            Deny from all

            Allow from all

        </Location>

     

    </VirtualHost>