1 2 Previous Next 16 Replies Latest reply on Sep 20, 2016 3:32 AM by deaconfrost2k

    mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error

    ftost

      Hello:

       

      I'm writing to you because I'm really desesperated with mod_cluster and apache 2.4. I have had a working setup with apache 2.2, but there's something different in apache 2.4 ->mod_cluster and wildfly that I can't fix.  I have followed the tutorials but nothing. I'm also using redhat 7.

       

      I'm getting these errors at apache's log:

       

      [Mon Aug 17 13:13:05.710893 2015] [proxy:error] [pid 11333:tid 140564964628224] (111)Connection refused: AH00957: ajp: attempt to connect to 192.168.216.3:27233 (192.168.216.3) failed

      [Mon Aug 17 13:13:05.710931 2015] [proxy:error] [pid 11333:tid 140564964628224] AH00959: ap_proxy_connect_backend disabling worker for (192.168.216.3) for 60s

      [Mon Aug 17 13:13:11.436188 2015] [proxy:error] [pid 11333:tid 140564884354816] (111)Connection refused: AH00957: ajp: attempt to connect to 192.168.216.3:27233 (192.168.216.3) failed

      [Mon Aug 17 13:13:11.436230 2015] [proxy:error] [pid 11333:tid 140564884354816] AH00959: ap_proxy_connect_backend disabling worker for (192.168.216.3) for 60s

      [Mon Aug 17 13:13:15.711481 2015] [proxy:error] [pid 11333:tid 140564964628224] AH00940: ajp: disabled connection for (192.168.216.3)

      [Mon Aug 17 13:13:21.436769 2015] [proxy:error] [pid 11333:tid 140564884354816] AH00940: ajp: disabled connection for (192.168.216.3)

      [Mon Aug 17 13:13:25.711999 2015] [proxy:error] [pid 11333:tid 140564964628224] AH00940: ajp: disabled connection for (192.168.216.3)

      [Mon Aug 17 13:13:31.437417 2015] [proxy:error] [pid 11333:tid 140564884354816] AH00940: ajp: disabled connection for (192.168.216.3)

      [Mon Aug 17 13:13:35.712539 2015] [proxy:error] [pid 11333:tid 140564964628224] AH00940: ajp: disabled connection for (192.168.216.3)

       

       

       

      When I try to look at mod_cluster's page, I get a NOTOK, as you can see:

       

      [jboss@localhost bin]$ curl http://192.168.216.3:6666/mod_cluster_manager

      <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">

      <html><head>

      <title>Mod_cluster Status</title>

      </head><body>

      <h1>mod_cluster/1.3.1.Final</h1><a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&refresh=10">Auto Refresh</a> <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=DUMP&Range=ALL">show DUMP output</a> <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=INFO&Range=ALL">show INFO output</a>

      <h1> Node REMOVED (ajp://192.168.216.3:18009): </h1>

      <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=ENABLE-APP&Range=NODE&JVMRoute=REMOVED">Enable Contexts</a> <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=DISABLE-APP&Range=NODE&JVMRoute=REMOVED">Disable Contexts</a> <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=STOP-APP&Range=NODE&JVMRoute=REMOVED">Stop Contexts</a><br/>

      Balancer: mycluster,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: NOTOK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: -1

      </pre><h1> Node master:vnint01 (ajp://192.168.216.3:8009): </h1>

      <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=ENABLE-APP&Range=NODE&JVMRoute=master:vnint01">Enable Contexts</a> <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=DISABLE-APP&Range=NODE&JVMRoute=master:vnint01">Disable Contexts</a> <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=STOP-APP&Range=NODE&JVMRoute=master:vnint01">Stop Contexts</a><br/>

      Balancer: vncluster,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: NOTOK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: -1

      <h2> Virtual Host 1:</h2><h3>Contexts:</h3><pre>/cluster-demo, Status: ENABLED Request: 0 <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=DISABLE-APP&Range=CONTEXT&JVMRoute=master:vnint01&Alias=localhost&Context=/cluster-demo">Disable</a> <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=STOP-APP&Range=CONTEXT&JVMRoute=master:vnint01&Alias=localhost&Context=/cluster-demo">Stop</a>

      </pre><h3>Aliases:</h3><pre>localhost

      default-host

      </pre><h1> Node slave:vnint02 (ajp://192.168.216.3:18009): </h1>

      <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=ENABLE-APP&Range=NODE&JVMRoute=slave:vnint02">Enable Contexts</a> <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=DISABLE-APP&Range=NODE&JVMRoute=slave:vnint02">Disable Contexts</a> <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=STOP-APP&Range=NODE&JVMRoute=slave:vnint02">Stop Contexts</a><br/>

      Balancer: vncluster,LBGroup: ,Flushpackets: Off,Flushwait: 10000,Ping: 10000000,Smax: 26,Ttl: 60000000,Status: NOTOK,Elected: 0,Read: 0,Transferred: 0,Connected: 0,Load: -1

      <h2> Virtual Host 1:</h2><h3>Contexts:</h3><pre>/cluster-demo, Status: ENABLED Request: 0 <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=DISABLE-APP&Range=CONTEXT&JVMRoute=slave:vnint02&Alias=localhost&Context=/cluster-demo">Disable</a> <a href="/mod_cluster_manager?nonce=103175c6-14e8-45e2-afa8-f5ad32a6864a&Cmd=STOP-APP&Range=CONTEXT&JVMRoute=slave:vnint02&Alias=localhost&Context=/cluster-demo">Stop</a>

      </pre><h3>Aliases:</h3><pre>localhost

      default-host

      </pre></body></html>

       

       

       

      The firewall is stopped and both apache and wildfly are on the same machine:

       

      [root@localhost logs]# service firewalld status

      Redirecting to /bin/systemctl status  firewalld.service

      firewalld.service - firewalld - dynamic firewall daemon

         Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled)

         Active: inactive (dead) since Fri 2015-08-14 13:41:50 CEST; 2 days ago

       

       

       

      This is my mod_cluster config:

       

       

      <IfModule manager_module>

        Listen 192.168.216.3:6666

        ManagerBalancerName vncluster

        <VirtualHost 192.168.216.3:6666>

          <Location />

           Allow from all

           Require all granted

           #Require ip 192.168.216.3

          </Location>

       

       

          KeepAliveTimeout 300

          MaxKeepAliveRequests 0

          #ServerAdvertise on http://@IP@:6666

          AdvertiseFrequency 5

          #AdvertiseSecurityKey secret

          #AdvertiseGroup @ADVIP@:23364

          EnableMCPMReceive

       

       

          <Location /mod_cluster_manager>

             SetHandler mod_cluster-manager

             Allow from all

             Require all granted

             #Require ip 192.168.216.3

          </Location>

       

       

        </VirtualHost>

      </IfModule>

       

       

      Edit: I have also attached my configuration files.

       

      Please, can you help me? Thanks in advance.

        • 1. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
          mbabacek

          I'll take a look...

          • 2. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
            rhusar

            In general, upgrading from 2.2 to 2.4 is explain in this article

            http://httpd.apache.org/docs/2.4/upgrading.html

            (thus for instance "Allow from all" is wrong in your configuration)

             

            But I assume the problem is elsewhere. Looking at the logs:

            attempt to connect to 192.168.216.3:27233

            27233 is this the right port? AJP connector usually uses port 8009. Try switching from AJP to HTTP connector to see if that works.

            • 3. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
              ftost

              Thanks Radoslav.

               

              It has no sense that port for me or I can't seem to find the problem.

               

               

              From the domain.xml:

               

              <mod-cluster-config advertise-socket="modcluster" advertise="false" connector="ajp"/>

              ....(tried http, but it dit not work either)

               

              And later in that config:

              <socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>

              • 4. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                mbabacek
                • O.K, If it is Wildfly 9, you should use attribute proxies, see my answer on the topic of having advertising disabled.
                • furthermore, each host (i.e. Wildfly instance) has to have instance-id="name_of_the_worker" set. I'm not sure whether you won't end up with multiple "slave" node names with your current setup
                • define <dynamic-load-provider> as it is supposed to be by default
                • verify that you have LoadModule cluster_slotmem_module modules/mod_cluster_slotmem.so loaded, because the name of the shared object changed between m_c 1.2.x and 1.3.x
                • and the most important thing of all:
                  • share your httpd configuration files, especially those of mod_proxy and mod_cluster; the mod_cluster snippet you provided does not show which modules you are loading.
                  • share httpd error_log on LogLevel Debug; the short snippet is cut out of its context
                  • tell us what is in the Wildfly log; are there Exceptions coming out of the modcluster subsystem about contacting the balancer?

                 

                Cheer up, it most definitely works fine; we will find the conf error in your setup

                • 5. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                  ftost

                  First thanks for all the help

                   

                  - I'm using wildfly 8.2.1 final.

                  - dynamic-load-provided tag, did not seem to let my instances to wake up again. Anyhow I understand that is not important.

                  -  Yes. I'm using mod_cluster 1.3.1 final. Is fine. When this tag is wrong, Apache does not start up.

                   

                  -I'm attaching the files you requested with log_level Debug. Wildfly does not seem to have anything wrong, just some warnings that don't seem to be important.

                   

                  [jboss@localhost log]$ grep WARN server.log

                  2015-08-17 16:01:32,268 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 33) JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.

                  2015-08-17 16:01:32,739 WARN  [jacorb.config] (MSC service thread 1-4) jacorb.home unset! Will use '.'

                  2015-08-17 16:01:32,740 WARN  [jacorb.config] (MSC service thread 1-4) File ./jacorb.properties for configuration jacorb not found

                  2015-08-17 16:01:32,934 WARN  [jacorb.config] (MSC service thread 1-4) jacorb.home unset! Will use '.'

                  2015-08-17 16:01:32,934 WARN  [jacorb.config] (MSC service thread 1-4) File ./jacorb.properties for configuration jacorb not found

                  2015-08-17 16:01:32,953 WARN  [jacorb.config] (MSC service thread 1-4) no properties found for configuration jacorb

                   

                   

                  About httpd.conf, it does also have all mod_cluster's configurations (everything is on the same file)

                   

                  Thanks

                  • 6. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                    mbabacek

                    I'm sorry Fran, the error_log is insufficient. I need to see the CONFIG message to find out where 27233 comes from. Please:

                    • stop the Apache
                    • stop the Wildfly instance
                    • delete all logs in logs/
                    • delete all httpd cache files in cache/
                    • start Apache
                    • start Wildfly
                    • give it ~30 seconds
                    • stop Wildfly
                    • stop Apache
                    • archive and send the error_log
                    • 7. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                      ftost

                      I have attached the new log

                       

                      Thanks a lot

                      • 8. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                        rhusar

                        So the ports in the CONFIG message are 8009 and 18009 -- which is in sync with the privided configuration. There is no mention of port 27233 which seems like it was part of some experiment. I am suspecting misconfiguration on a different level such as selinux, routing, firewalld/iptables started again after a restart, etc. Unless Michal sees something specific.

                        • 9. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                          mbabacek

                          Hmm, I tried to mimic your environment, but I cannot reproduce this thing in your log:

                          [client 192.168.216.3:34365] AH00944: connecting ajp://192.168.216.3:27233/ to 192.168.216.3:27233

                          [client 192.168.216.3:34365] AH00947: connected / to 192.168.216.3:27233

                          (111)Connection refused: AH00957: ajp: attempt to connect to 192.168.216.3:27233 (192.168.216.3) failed

                           

                          furthermore, I even unloaded shm module and loaded lbmethod heartbeat so as to get the same error on startup as you have:

                               [lbmethod_heartbeat:notice] [pid 28385:tid 140592057104448] AH02281: Failed to lookup provider 'shm' for 'slotmem'. Maybe you need to load mod_slotmem_shm?

                          It looks like mod_proxy remembers a mapping to worker on IP:PORT 27233 and that is weird. Could you check whether it is always the same port? What does lsof | grep 27233 say during your test?

                           

                          What looks like this in your log:

                          proxy_util.c(2020): AH00942: ajp: has acquired connection for (192.168.216.3) 
                          proxy_util.c(2072): [client 192.168.216.3:34365] AH00944: connecting ajp://192.168.216.3:27233/ to 192.168.216.3:27233 
                          proxy_util.c(2194): [client 192.168.216.3:34365] AH00947: connected / to 192.168.216.3:27233 
                          (111)Connection refused: AH00957: ajp: attempt to connect to 192.168.216.3:27233 (192.168.216.3) failed 
                          
                          proxy_util.c(2020): AH00942: ajp: has acquired connection for (192.168.216.3) 
                          proxy_util.c(2072): [client 192.168.216.3:34367] AH00944: connecting ajp://192.168.216.3:27233/ to 192.168.216.3:27233 
                          proxy_util.c(2194): [client 192.168.216.3:34367] AH00947: connected / to 192.168.216.3:27233
                           (111)Connection refused: AH00957: ajp: attempt to connect to 192.168.216.3:27233 (192.168.216.3) failed 
                          

                          is always the following with mine:

                          [proxy:debug] [pid 16706] proxy_util.c(2020): AH00942: ajp: has acquired connection for (rhel7gax86-64) 
                          [proxy:debug] [pid 16706] proxy_util.c(2072): [client 127.0.0.1:50027] AH00944: connecting ajp://rhel7gax86-64:8009/ to rhel7gax86-64:8009 
                          [proxy:debug] [pid 16706] proxy_util.c(2194): [client 127.0.0.1:50027] AH00947: connected / to rhel7gax86-64:8009
                          [:debug] [pid 16706] mod_proxy_cluster.c(1413): proxy_cluster_try_pingpong: connected to backend 
                          -- 
                          [proxy:debug] [pid 28390:tid 140591534900992] proxy_util.c(2020): AH00942: ajp: has acquired connection for (192.168.122.172)
                          [proxy:debug] [pid 28390:tid 140591534900992] proxy_util.c(2072): [client 192.168.122.172:44673] AH00944: connecting ajp://192.168.122.172:9109/ to 192.168.122.172:9109
                          [proxy:debug] [pid 28390:tid 140591534900992] proxy_util.c(2194): [client 192.168.122.172:44673] AH00947: connected / to 192.168.122.172:9109
                          [:debug] [pid 28390:tid 140591534900992] mod_proxy_cluster.c(1413): proxy_cluster_try_pingpong: connected to backend
                          

                          Did you erase /var/cache/mod_cluster (or wherever you have the *.shm) files after Apache restart?  Is the error you experience intermittent or always deterministically reproducible?

                          • 10. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                            ftost

                            Thanks Michal

                             

                            I have tried loading that module and erasing the cache, and nothing new seems to happen. I still get the NOTOK inside mod_cluster's page.

                             

                            lsof | grep 27233 has no output

                            Neither does netstat -anp | grep LISTEN | grep 27233

                            • 11. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                              mbabacek

                              With this exact message (port number) each restart or the port changes?

                              (111)Connection refused: AH00957: ajp: attempt to connect to 192.168.216.3:27233 (192.168.216.3) failed

                              • 12. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                                ftost

                                Thanks Radoslav, Michal.

                                 

                                Yes, that port seems to be allways the same. I can tell you that I have had the similar problem on a RHEL 6, and after downgrading to apache 2.2 and mod_cluster 1.2.6, everything is working fine in that machine.

                                 

                                Now, I'm on RHEL 7 and apache 2.4 seems to be mandatory, so I'm stuck with mod_cluster 1.3.1 and this problem.

                                 

                                I'm using the imported configs that I know that work in the RHEL6, in the new with RHEL7 and the changes I've made for apache 2.4


                                I really don't know whatelse to do. I've tried even with wildfly 9 and different apache 2.4 versions and nothing.


                                Thanks...

                                • 13. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                                  mbabacek

                                  Dear Fran, the configuration file you provided was tampered with either before or after your test.

                                  It is invalid not only with respect to Allow from directives, but there is apparently some portion of it missing because config elements aren't properly closed: Expected </Directory> but saw </IfModule>.

                                   

                                  Help us help you and do not provide configuration files that weren't used during the test runtime and therefore do not match the logs.

                                   

                                  Lemme suggest these next steps:

                                  • backup, if it's not your sandbox/testing environment
                                  • yum remove httpd, remove all files in /etc/httpd, /var/log/httpd, /var/cache/httpd and any possible stray files in /usr/lib64/httpd/modules
                                  • yum install httpd, i.e. vanilla httpd 2.4 from RHEL7 repos
                                  • configure bare minimum, i.e. IP addresses, ServerName, Listen...
                                  • In /etc/httpd/conf.modules.d/00-proxy.conf  comment: #LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
                                  • use attached /etc/httpd/conf.d/mod_cluster.conf (change IP address for your own...) and your mod_cluster binaries OR see the RPM paragraph at the bottom
                                  • either temporarily disable SeLinux and Firewalld or use semanage port -a -t http_port_t -p tcp 6666, semanage port -a -t http_port_t -p tcp 8009, semanage port -a -t http_port_t -p udp 23364 ....etc and firewall-cmd --add-port=23364/udp ..and so on in order to configure the system properly
                                  • run your tests

                                   

                                  Regarding the mod_cluster installation: You could also rebuild mod_cluster 1.3.1.Final RPM from Fedora on your RHEL7 box and use it for installation:

                                  • you need build essentials (gcc, make...), rpmbuild, httpd-devel, autotools, apr-devel, apr-util-devel and a few things it might shout at you as missing deps
                                  • grab https://karm.fedorapeople.org/mod_cluster/mod_cluster-1.3.1-1.fc22.src.rpm and rpm -i install it  (Don't worry about its origin, it's from my Fedora profile, so for testing it's good enough )
                                  • edit your ~/rpmbuild/SPECS/mod_cluster.spec and set %define with_java 0
                                  • in ~/rpmbuild/SPECS/ run: rpmbuild -ba mod_cluster.spec
                                  • yum install ~/rpmbuild/RPMS/x86_64/mod_cluster-1.3.1-1.el7.x86_64.rpm

                                  These are community bits, provided as is, usual disclaimer applies, read ~/rpmbuild/BUILD/mod_cluster-1.3.1.Final/JBossORG-EULA.txt.

                                  • 14. Re: mod_cluster 1.3.1, apache 2.4 and wildfly 8/9 error
                                    ftost

                                    Thanks a lot for all your help

                                     

                                    I have downloaded apache 2.2 with mod_cluster 1.2.6 from mod_cluster's site and is working perfectly on rhel7 (mi only configuration was to set the IP after uncompression).

                                    The same did not work with their 2.4 apache. Now at least, I have a working config, but I don't understand what is going on in this new version. It happened in rhel6 and another machine too.

                                     

                                    Thanks

                                    1 2 Previous Next