-
15. Re: mod_cluster 1.2 + JBossAS 7.1 problem
ranga033 Mar 29, 2012 9:12 AM (in response to ctomc)ok thank you for the prompt reply Tomaz !
-
16. Re: mod_cluster 1.2 + JBossAS 7.1 problem
johanvermeij Aug 21, 2012 10:47 AM (in response to ctomc)Maybe this is not the right thread but I try and try without result.
Using 7.1.0:
I can access the cluster using Apache httpd when I comment out ServerAdvertise (when I uncomment ServerAdvertise I don't find the cluster)
<IfModule manager_module>
Listen 192.168.2.2:6666
ManagerBalancerName other-server-group
<VirtualHost 192.168.2.2:6666>
<Location />
Order deny,allow
Deny from all
Allow from all
</Location>
ErrorLog "logs/chatter_log"
LogLevel debug
KeepAliveTimeout 100
MaxKeepAliveRequests 0
# ServerAdvertise on http://127.0.0.1:6666
AdvertiseFrequency 30
#AdvertiseSecurityKey secret
#AdvertiseGroup 224.0.1.105:23364
EnableMCPMReceive
</VirtualHost>
</IfModule>
</IfModule>
NameVirtualHost 192.168.2.3:8880
<VirtualHost 192.168.2.3:8880>
ServerAdmin johanvermeij@gmail.com
ServerName 192.168.2.3
ErrorLog "logs/cluster_log"
LogLevel debug
ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On
ProxyPassReverse / balancer://other-server-group
ProxyPreserveHost On
<Location />
Order deny,allow
Allow from All
</Location>
<Location /mod_cluster_manager>
SetHandler mod_cluster-manager
Order deny,allow
Deny from all
Allow from all
</Location>
</VirtualHost>
That works
I now would like to use 7.1.1 clustering.
When I use the same httpd.conf as above under 7.1.1 all I ever get is (I have tried everything I could think of for 2 days straight):
[Tue Aug 21 21:24:17 2012] [debug] mod_proxy_cluster.c(1543): proxy: byrequests balancer FAILED
[Tue Aug 21 21:24:17 2012] [error] proxy: CLUSTER: (balancer://other-server-group). All workers are in error state
-
17. Re: mod_cluster 1.2 + JBossAS 7.1 problem
jfclere Aug 22, 2012 1:50 AM (in response to johanvermeij)Does /mod_cluster_manager show the nodes?
-
18. Re: mod_cluster 1.2 + JBossAS 7.1 problem
ranga033 Aug 22, 2012 2:25 AM (in response to johanvermeij)Let me know which version are you trying this ? Did you try on 7.1.1/7.1.2. ?
-
19. Re: mod_cluster 1.2 + JBossAS 7.1 problem
johanvermeij Aug 22, 2012 2:19 AM (in response to jfclere)I am now at the point where I can run master and slave instances but I cannot access the cluster (do transactions from a terminal) when I start up a slave.
As soon as I stop the slave I can do transactions again.
This is the output from mod_cluster_manager with the master running
mod_cluster/1.2.0.Final
Auto Refresh show DUMP output show INFO output
Node undefined (ajp://192.168.2.3:8259):
Enable Contexts Disable Contexts
Balancer: other-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: OK,Elected: 1,Read: 9065,Transferred: 149,Connected: 0,Load: 100
Virtual Host 1:
Contexts:
/internetbanking, Status: ENABLED Request: 0 Disable /beans, Status: ENABLED Request: 0 Disable /wssimple, Status: ENABLED Request: 0 Disable
Aliases:
localhost default-host example.com
When I bring up a slave I get this intermittently (httpd is very busy stopping and starting workers)
mod_cluster/1.2.0.Final
Auto Refresh show DUMP output show INFO output
Node REMOVED (ajp://192.168.2.3:8259):
Enable Contexts Disable Contexts
Balancer: other-server-group,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: NOTOK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: 100
modcluster settings in domain.xml are as follows (I have added the proxy-list and advertise-security-key this morning).
<subsystem xmlns="urn:jboss:domain:modcluster:1.0">
<mod-cluster-config advertise-socket="modcluster" proxy-list="192.168.2.3:6666" advertise-security-key="secret">
<dynamic-load-provider>
<load-metric type="busyness"/>
</dynamic-load-provider>
</mod-cluster-config>
</subsystem>
As soon as I start a slave I get this error on both master and slave:
[Server:server-three-Linux] 12:50:48,703 ERROR [org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler] (ContainerBackgroundProcessor[StandardEngine[jboss.web]]) Error [MEM: MEM: Can't read node: {4}] sending command STATUS to proxy 192.168.2.3/192.168.2.3:6666, configuration will be reset
I post my current httpd.conf settings as well (I copied again from 1.7.0 and this time things started working).
Listen 192.168.2.3:6666
ManagerBalancerName other-server-group
<VirtualHost 192.168.2.3:6666>
<Location />
Order deny,allow
Deny from all
Allow from all
</Location>
ServerName vantagebanking.com
ErrorLog "logs/chatter_log"
LogLevel debug
KeepAliveTimeout 100
MaxKeepAliveRequests 0
ServerAdvertise on http://127.0.0.1:6666
AdvertiseFrequency 30
AdvertiseSecurityKey secret
AdvertiseGroup 224.0.1.105:23364
EnableMCPMReceive
</VirtualHost>
</IfModule>
NameVirtualHost 192.168.2.3:8000
<VirtualHost 192.168.2.3:8000>
ServerAdmin johanvermeij@gmail.com
ServerName vantagebanking.com
ErrorLog "logs/cluster_log"
LogLevel debug
ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On
ProxyPassReverse / balancer://other-server-group
ProxyPreserveHost On
<Location />
Order deny,allow
Allow from All
</Location>
<Location /mod_cluster_manager>
SetHandler mod_cluster-manager
Order deny,allow
Deny from all
Allow from all
</Location>
/VirtualHost>
Sorry, all a bit messy, for me there is a lot of trial and error in this.
I am amazed by JBoss clustering and I am determined to make it work
-
20. Re: mod_cluster 1.2 + JBossAS 7.1 problem
johanvermeij Aug 22, 2012 2:30 AM (in response to ranga033)I use 7.1.1
Is 7.1.2 out already ?
-
21. Re: mod_cluster 1.2 + JBossAS 7.1 problem
ranga033 Aug 22, 2012 2:30 AM (in response to johanvermeij)Did you make the node names unique (specifying unique names for master and slave through host.xml) . If everything is fine, the mod_cluster_manager should show both the nodes in the paeg.
-
22. Re: mod_cluster 1.2 + JBossAS 7.1 problem
ranga033 Aug 22, 2012 2:31 AM (in response to johanvermeij)7.1.2 is a tag for EAP release, it has to be built by downloading the JBoss source code. However for mod_cluster to work 7.1.1 should be fine.
-
23. Re: mod_cluster 1.2 + JBossAS 7.1 problem
johanvermeij Aug 22, 2012 2:45 AM (in response to ranga033)That is what I am looking for, I read that an instance-id can be added, also through the web console.
But I can't find where.
I suppose the instance-id can be set in this section of host.xml
Can you give me a hint ?
<servers> <server name="server-one" group="main-server-group"> <!-- Remote JPDA debugging for a specific server <jvm name="default"> <jvm-options> <option value="-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n"/> </jvm-options> </jvm> --> </server> <server name="server-two" group="main-server-group" auto-start="false"> <!-- server-two avoids port conflicts by incrementing the ports in the default socket-group declared in the server-group --> <socket-bindings port-offset="150"/> </server> <server name="server-three-Linux" group="other-server-group" auto-start="true"> <!-- server-three avoids port conflicts by incrementing the ports in the default socket-group declared in the server-group --> <socket-bindings port-offset="250"/> </server> </servers> -
24. Re: mod_cluster 1.2 + JBossAS 7.1 problem
jfclere Aug 22, 2012 2:45 AM (in response to johanvermeij)the undefined is due to a bug... JBPAPP-8451?
-
25. Re: mod_cluster 1.2 + JBossAS 7.1 problem
ranga033 Aug 22, 2012 2:54 AM (in response to johanvermeij)As mentioned in Jean-Frederic's post above try adding the workaround as you use 7.1.1
Try adding this in domain.xml
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" instance-id="${jboss.node.name}" native="false">
Also, hope you have set the names for master and slave as per https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto
something like , <host name="slave" xmlns="urn:jboss:domain:1.2">
I hope this should solve your issue. Also go through the comments in https://docs.jboss.org/author/display/AS71/AS7+Cluster+Howto . It has solution and workarounds for most of the issues for Cluster setup.
-
26. Re: mod_cluster 1.2 + JBossAS 7.1 problem
johanvermeij Aug 22, 2012 5:51 AM (in response to ranga033)Thank you very much, it all works fine
I am already running a jmeter test against a cluster with a HP Proliant, an iMac and a Linux PC
It is fast
-
27. Re: mod_cluster 1.2 + JBossAS 7.1 problem
johanvermeij Aug 23, 2012 7:52 AM (in response to johanvermeij)When I am running a stress test against the above cluster under 7.1.1 (domain mode) I get this error very often.
[Server:server-three-slave-HP] 18:46:25,182 ERROR [com.vantage.capital.service.xml.XMLTranServiceBean] (ajp--192.168.2.12-8259-22) DDF0001,CashTransaction:EXCP:org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate
In fact, I don't need jmeter for this.
When I open 2 terminals and I do transactions quickly by hitting enter fast I get the same error.
Looking at the JBoss logs, netstat, Apache logs everything looks fine
-
28. Re: mod_cluster 1.2 + JBossAS 7.1 problem
ranga033 Aug 23, 2012 8:01 AM (in response to johanvermeij)I am not too sure what kind of stress testing are you performing. If you can elaborate a bit, it would be better to understand the problem.
From your logs, I see that you are using Hibernate and perhaps the Infinispan cache for Hibernate. Are those caches up when you are performing a test ? What kind of caches have you configured (any remote caches?)?
-
29. Re: mod_cluster 1.2 + JBossAS 7.1 problem
johanvermeij Aug 23, 2012 8:49 AM (in response to ranga033)I am testing a financial application.
I prepare jmeter jmx files using a sample soap message captured from a terminal.
I use the sample to propagate many other also simulating different users, bank branches, accounts etc.
I point the jmeter injector at Apache http on port 8000 (see httpd.conf below) using mod_cluster (latest)
I use one or more laptops as jmeter injectors
httpd then load balances this to a cluster which consist of a HP Proliant mini server (Centos 6 64 bit, 192.168.2.12), a PC (Centos 6 64 bit, 192.168.2.3) and an iMac (MacOs, 192.168.2.4)
I use an Oracle 11g database on the HP, httpd runs on the PC.
The stress test runs very fast when only the master is up (PC is the master)
As soon as I bring up a slave (and also when I bring up master and 2 slaves from the start) I get this error:
[Server:server-three-slave-HP] 19:28:03,650 ERROR [com.vantage.capital.service.xml.XMLTranServiceBean] (ajp--192.168.2.12-8259-146) DDF0002,CashWithdrawal:EXCP:org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.hibernate.cache.CacheException: org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate; org.infinispan.CacheException: org.jgroups.TimeoutException: timeout sending message to master:server-three-Linux/hibernate
The above timeout happens on the PC and the HP.
It slows the test down to about 1TPS (by comparison I can do almost 100TPS against the HP in standalone mode)
The iMac has a different error:
[Server:server-three-slave-iMac] 19:30:22,593 WARNING [org.jgroups.protocols.UDP] (pool-23-thread-1) null: no physical address for 3cf4cecf-6754-af54-1c50-f2ba733aac93, dropping message
[Server:server-three-slave-iMac] 19:30:25,595 WARNING [org.jgroups.protocols.pbcast.GMS] (pool-23-thread-1) JOIN(slave1:server-three-slave-iMac/hibernate) sent to 3cf4cecf-6754-af54-1c50-f2ba733aac93 timed out (after 3000 ms), retrying
I have gone through the startup logs and everything looks fine there, there are no errors or warnings.
I don't expect you to fix this for me Rang, maybe you can give me a hint.
We have been working on this for almost a week now and we are almost there.
Problem is also that Googling these kinds of conditions is very "circular" and a lot of the discussions pertain to older JBoss releases.
I say it again, what you guys have done is incredible, no commercial software can come near you.
From httpd.conf
<IfModule manager_module>
Listen 192.168.2.3:6666
AllowDisplay On
ManagerBalancerName other-server-group
<VirtualHost 192.168.2.3:6666>
<Location />
Order deny,allow
Deny from all
Allow from all
</Location>
ServerName vantagebanking.com
ErrorLog "logs/chatter_log"
LogLevel debug
KeepAliveTimeout 100
MaxKeepAliveRequests 0
# ServerAdvertise on http://192.168.2.3:6666
AdvertiseFrequency 30
# AdvertiseSecurityKey secret
# AdvertiseGroup 224.0.1.105:23364
EnableMCPMReceive
</VirtualHost>
</IfModule>
NameVirtualHost 192.168.2.3:8000
<VirtualHost 192.168.2.3:8000>
ServerAdmin johanvermeij@gmail.com
ServerName vantagebanking.com
ErrorLog "logs/cluster_log"
LogLevel debug
AllowDisplay On
ProxyPass '/beans/XMLTranServiceBean/XMLTranServiceBean' balancer://other-server-group stickysession=JSESSIONID|jsessionid nofailover=On
ProxyPassReverse / balancer://other-server-group
ProxyPreserveHost On
<Location />
Order deny,allow
Allow from All
</Location>
<Location /mod_cluster_manager>
SetHandler mod_cluster-manager
Order deny,allow
Deny from all
Allow from all
</Location>
</VirtualHost>