-
15. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)
jfclere Feb 22, 2010 2:36 AM (in response to naveenj)"192.168.100.150:8082
java.net.ConnectException: Connection refused"Firewall issue?
-
16. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)
rhusar Feb 22, 2010 5:05 AM (in response to naveenj)Isnt this a problem with port mismatch? Your virtual host listens on 80 and the status message from mod_cluster is being sent to port 8082. -
17. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)
naveenj Feb 23, 2010 1:03 AM (in response to rhusar)Radoslav and Jean,
That was because I changed the Virtual Host Port to 8082. When I change it back to 80, no errors are displayed at JBoss, but apache has the following error:
/var/log/apache2/error.log
[Mon Feb 22 22:47:54 2010] [error] [client 192.168.100.150] client denied by server configuration: proxy:balancer://mycluster/
[Mon Feb 22 22:47:59 2010] [error] [client 192.168.100.150] client denied by server configuration: proxy:balancer://mycluster/
[Mon Feb 22 22:48:00 2010] [error] [client 192.168.100.150] client denied by server configuration: proxy:balancer://mycluster/I managed to solve the error.I had to change Deny from all to Deny from none in proxy.conf as given below.
<Proxy *>
AddDefaultCharset off
Order deny,allow
Deny from none
#Allow from .example.com
</Proxy>But I noticed my apache log contains other errors as below. Any idea why?
/var/log/apache2/error.log
[Mon Feb 22 23:27:40 2010] [error] ajp_cping_cpong: apr_socket_recv failed
[Mon Feb 22 23:28:59 2010] [error] proxy: dialog to 192.168.100.150:8009 (192.168.100.150) failed
[Mon Feb 22 23:28:59 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Mon Feb 22 23:29:01 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:12 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:22 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:32 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:42 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:52 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Mon Feb 22 23:29:55 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:11 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Tue Feb 23 00:04:11 2010] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Feb 23 00:04:11 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.100.150:8009 (192.168.100.150)
[Tue Feb 23 00:04:11 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:14 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 00:04:16 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:18 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:21 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:24 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 00:04:25 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 00:04:34 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 00:04:44 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 00:04:54 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 00:05:04 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)Note: http://192.168.100.150/ redirects to JBoss, but sometimes, it goes down "Server temporarily Unavailable" - maybe for few seconds and again its back.
-
18. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)
jfclere Feb 23, 2010 2:25 AM (in response to naveenj)"Tue Feb 23 00:04:11 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Tue Feb 23 00:04:11 2010] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Feb 23 00:04:11 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.100.150:8009 (192.168.100.150)"Your application on AS side is too slow to answer. Try to use nodeTimeout (http://www.jboss.org/mod_cluster/java/properties.html) in the bean properties.
-
19. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)
naveenj Feb 23, 2010 10:16 PM (in response to jfclere)I added the following in mod-cluster-jboss-beans.xml
<property name="nodeTimeout">30</property>
This removed the the error as given below
Tue Feb 23 00:04:11 2010] [error] (70007)The timeout specified has expired: ajp_ilink_receive() can't receive header
[Tue Feb 23 00:04:11 2010] [error] ajp_read_header: ajp_ilink_receive failed
[Tue Feb 23 00:04:11 2010] [error] (120006)APR does not understand this error code: proxy: read response failed from 192.168.100.150:8009 (192.168.100.150)but the following error still remains.
/var/log/apache2/error.log - When I start the first node
[Tue Feb 23 16:21:24 2010] [error] [client 192.168.100.150] File does not exist: /htdocs
[Tue Feb 23 16:22:24 2010] [error] [client 192.168.100.150] File does not exist: /htdocs
[Tue Feb 23 16:23:24 2010] [error] [client 192.168.100.150] File does not exist: /htdocs
[Tue Feb 23 16:24:24 2010] [error] [client 192.168.100.150] File does not exist: /htdocs
[Tue Feb 23 16:25:00 2010] [error] ajp_cping_cpong: apr_socket_recv failed
[Tue Feb 23 16:25:20 2010] [error] ajp_cping_cpong: apr_socket_recv failedAfter say about 3 min, the apache error log is appended with the following.
[Tue Feb 23 16:28:10 2010] [error] proxy: dialog to 192.168.100.150:8009 (192.168.100.150) failed
[Tue Feb 23 16:28:10 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:10 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 16:28:11 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:16 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:17 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:17 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:17 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:19 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:19 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:20 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 16:28:28 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:29 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
[Tue Feb 23 16:28:30 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 16:28:40 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 16:28:50 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)
[Tue Feb 23 16:29:00 2010] [error] proxy: ajp: disabled connection for (192.168.100.150)When I add another node to the cluster, the same set of errors appear in the log for that IP.
However, I am able to redirect apache to JBoss, but these errors keep getting appended.
I noticed a relevant post: http://community.jboss.org/thread/68842
-
20. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)
jfclere Feb 24, 2010 2:17 AM (in response to naveenj)"After say about 3 min"
With or without load?
-
21. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)
naveenj Feb 25, 2010 1:49 AM (in response to jfclere)Without Load, I just added 1 extra node and for every node I add, the errors repeat once.
But, its weird. After sometime I noticed that the errors totally disappeared and apache error log was clean.So far so good.
Why does this happen and is it something I should be concerned about when going live?
Thanks.
-
22. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)
jfclere Feb 25, 2010 6:00 AM (in response to naveenj)Could you please set the log level to debug and post the error_log?
-
23. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)
naveenj Feb 25, 2010 6:00 PM (in response to jfclere)Ok, done. Set it to debug and I attached the apache error log output.Note: I also noticed another issue. One my my nodes shutdown abruptly, hence JBoss did not shutdown. gracefullyWhen I checked http://192.168.100.150/mod_cluster-manager, it still shows 2 nodes. I cleared cache etc, no luck. I had to restart apache for it to be cleared.Thanks in advance. -
24. Re: Redirect traffic to JBoss AS (Equivalent of JkMount of mod_jk)
jfclere Feb 26, 2010 2:58 AM (in response to naveenj)There are 2 things:
1 -
+++
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy.c(993): Running scheme balancer handler (attempt 0)
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_ajp.c(644): proxy: AJP: serving URL ajp://192.168.100.150:8009/
[Thu Feb 25 16:49:24 2010] [debug] proxy_util.c(1999): proxy: AJP: has acquired connection for (192.168.100.150)
[Thu Feb 25 16:49:24 2010] [debug] proxy_util.c(2055): proxy: connecting ajp://192.168.100.150:8009/ to 192.168.100.150:8009
[Thu Feb 25 16:49:24 2010] [debug] proxy_util.c(2153): proxy: connected / to 192.168.100.150:8009
[Thu Feb 25 16:49:24 2010] [debug] ajp_utils.c(31): Into ajp_handle_cping_cpong
[Thu Feb 25 16:49:24 2010] [debug] ajp_utils.c(102): ajp_handle_cping_cpong: Done
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(224): Into ajp_marshal_into_msgb
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[0] [User-Agent] = [check_http/v1944 (nagios-plugins 1.4.11)]
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[1] [Connection] = [close]
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(290): ajp_marshal_into_msgb: Header[2] [Host] = [192.168.100.150:80]
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(450): ajp_marshal_into_msgb: Done
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_ajp.c(265): proxy: APR_BUCKET_IS_EOS
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_ajp.c(270): proxy: data to read (max 8186 at 4)
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_ajp.c(285): proxy: got 0 bytes of data
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(687): ajp_read_header: ajp_ilink_received 09
[Thu Feb 25 16:49:24 2010] [debug] ajp_header.c(697): ajp_parse_type: got 09
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_ajp.c(540): proxy: Processing of request failed backend: 1, output: 0
[Thu Feb 25 16:49:24 2010] [error] proxy: dialog to 192.168.100.150:8009 (192.168.100.150) failed
[Thu Feb 25 16:49:24 2010] [debug] proxy_util.c(2017): proxy: AJP: has released connection for (192.168.100.150)
[Thu Feb 25 16:49:24 2010] [debug] mod_proxy_cluster.c(1118): proxy: byrequests balancer FAILED
[Thu Feb 25 16:49:24 2010] [error] proxy: CLUSTER: (balancer://mycluster). All workers are in error state
+++That is weird the request gets a CPONG instead the headers.
2 - STATUS should retry the worker even if there was an error before. (JIRA MODCLUSTER-133).